Attention Residuals

Benchmark Model Rank Results
code-generation-on-mbppKimi Linear (48B) with Block AttnRes#47Accuracy: 73.9
sentence-completion-on-hellaswagKimi Linear (48B) with Block AttnRes#50Accuracy: 83.4
math-word-problem-solving-on-mathKimi Linear (48B) with Block AttnRes#62Accuracy: 57.1
code-generation-on-humanevalKimi Linear (48B) with Block AttnRes#92Pass@1: 62.2
multi-task-language-understanding-on-mmlu-proKimi Linear (48B) with Block AttnRes#112Accuracy: 52.2
multiple-choice-question-answering-mcqa-on-gpqa-diamondKimi Linear (48B) with Block AttnRes#120Accuracy: 44.4
arithmetic-reasoning-on-gsm8kKimi Linear (48B) with Block AttnRes#137Accuracy: 82.4