What Makes Recurrence Effective in Looped Language Models?

Benchmark Model Rank Results
common-sense-reasoning-on-winograndeBaseLoop (Llama3.1-1B, 3x4, Muon)–Accuracy: 52.88
question-answering-on-piqaBaseLoop (Llama3.1-1B, 3x4, Muon)–Accuracy: 65.18
reading-comprehension-on-raceBaseLoop (Llama3.1-1B, 3x4, Muon)–Accuracy: 30.62
sentence-completion-on-hellaswagBaseLoop (Llama3.1-1B, 3x4, Muon)–Accuracy: 31.48
zero-shot-question-answering-on-openbookqaBaseLoop (Llama3.1-1B, 3x4, Muon)–Accuracy: 20.8