UL2: Unifying Language Learning Paradigms

Open paper
Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kUL2 20B (chain-of-thought)#116Accuracy: 4.4Parameters (Billion): 20
arithmetic-reasoning-on-gsm8kUL2 20B (0-shot)#117Accuracy: 4.1Parameters (Billion): 20
common-sense-reasoning-on-arc-challengeUL2 20B (chain-of-thought + self-consistency)#25Accuracy: 49.5
common-sense-reasoning-on-arc-challengeUL2 20B (chain-of-thought)#32Accuracy: 42.9
common-sense-reasoning-on-arc-challengeUL2 20B (zero-shot)#40Accuracy: 29.8
common-sense-reasoning-on-arc-easyUL2 20B (chain-of-thought + self-consistency)#30Accuracy: 69.8
common-sense-reasoning-on-arc-easyUL2 20B (chain-of-thought)#38Accuracy: 38.4
common-sense-reasoning-on-arc-easyUL2 20B (0-shot)#40Accuracy: 32.2
common-sense-reasoning-on-commonsenseqaUL2 20B (chain-of-thought + self-consistency)#29Accuracy: 55.7
common-sense-reasoning-on-commonsenseqaUL2 20B (chain-of-thought)#31Accuracy: 51.4
common-sense-reasoning-on-commonsenseqaUL2 20B (zero-shot)#33Accuracy: 34.2
coreference-resolution-on-winograd-schemaUL2 20B (fine-tuned)#2Accuracy: 98.1
coreference-resolution-on-winograd-schemaUL2 20B (0-shot)#20Accuracy: 79.9
long-range-modeling-on-scrollsUL2#6Avg.: 37.87GovRep: 53.6 / 26.1 / 28.8SumScr: 32.9 / 7.8 / 19.4…
long-range-modeling-on-scrollsUL2 20B#12CNLI: 88.7
multi-task-language-understanding-on-mmluUL2 20B (5-shot)#26Average (%): 39.2
natural-language-inference-on-rteUL2 20B (fine-tuned)#9Accuracy: 92.1%
natural-language-inference-on-rteUL2 20B (0-shot)#62Accuracy: 60.7%
question-answering-on-boolqUL2 20B (fine-tuned)#6Accuracy: 90.8
question-answering-on-boolqUL2 20B (0-shot)#48Accuracy: 63.1
question-answering-on-copaUL2 20B (fine-tuned)#3Accuracy: 99
question-answering-on-copaUL2 20B (0-shot)#26Accuracy: 85
word-sense-disambiguation-on-words-in-contextUL2 20B (fine-tuned)#5Accuracy: 77.3
word-sense-disambiguation-on-words-in-contextUL2 20B (0-shot)#23Accuracy: 49.8

Rank counts only results with a code link.