Parcae: Scaling Laws For Stable Looped Language Models

Benchmark Model Rank Results
question-answering-on-copa-zero-shotParcae-1.3B (T=8, 0-shot)#5Accuracy: 68.666667
zero-shot-question-answering-on-openbookqaParcae-770M (T=8, 0-shot)#15Accuracy: 40.666669
language-modelling-on-lambadaParcae-770M (T=8, 0-shot)#55Accuracy: 43.22401
language-modelling-on-lambadaParcae (1.3B)#70Perplexity: 14.71
common-sense-reasoning-on-arc-easyParcae-1.3B (T=8, 10-shot)#60Accuracy: 68.981483
common-sense-reasoning-on-arc-easyParcae (350M)#81Accuracy: 48.78
question-answering-on-boolqParcae-1.3B (T=8, 10-shot)#83Accuracy: 63.088683
question-answering-on-boolqParcae (350M)#96Accuracy: 60.73
common-sense-reasoning-on-winograndeParcae-1.3B (T=8, 0-shot)#129Accuracy: 58.800316
question-answering-on-piqaParcae-1.3B (T=8, 10-shot)#132Accuracy: 72.343125
question-answering-on-piqaParcae (350M)#166Accuracy: 62.08
sentence-completion-on-hellaswagParcae-1.3B (T=8, 10-shot)#162Accuracy: 56.598949
sentence-completion-on-hellaswagParcae (350M)#219Accuracy: 29.23