LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

Benchmark Model Rank Results
multiple-choice-question-answering-mcqa-on-arc-cLatentLoop (7B / 1B active)#5Accuracy: 51.71
language-modelling-on-lambadaLatentLoop (7B / 1B active)#52Accuracy: 45.92
question-answering-on-piqaLatentLoop (7B / 1B active)#96Accuracy: 76.33
common-sense-reasoning-on-winograndeLatentLoop (7B / 1B active)#99Accuracy: 63.93
sentence-completion-on-hellaswagLatentLoop (7B / 1B active)#116Accuracy: 68.44