ECO: Quantized Training without Full-Precision Master Weights

Benchmark Model Rank Results
common-sense-reasoning-on-arc-easyDeepSeek-MoE-16B (INT4 w/o MW ECO + RTN)–Accuracy: 71.59
question-answering-on-piqaDeepSeek-MoE-16B (INT4 w/o MW ECO + SR)–Accuracy: 81.5
sentence-completion-on-hellaswagDeepSeek-MoE-16B (INT4 w/o MW ECO + RTN)–Accuracy: 78.88