Finetuned Language Models Are Zero-Shot Learners

Benchmark Model Rank Results
common-sense-reasoning-on-arc-challengeFLAN 137B (few-shot, k=13)#12Accuracy: 63.8
common-sense-reasoning-on-arc-challengeFLAN 137B (zero-shot)#13Accuracy: 63.1
common-sense-reasoning-on-arc-easyFLAN 137B (few-shot, k=14)#11Accuracy: 80.7
common-sense-reasoning-on-arc-easyFLAN 137B (0-shot)#15Accuracy: 79.6
common-sense-reasoning-on-recordFLAN 137B (prompt-tuned)#8EM: 85.1
common-sense-reasoning-on-recordFLAN 137B (zero-shot)#11EM: 72.5
common-sense-reasoning-on-winograndeFLAN 137B (few-shot, k=16)#31Accuracy: 72.8
common-sense-reasoning-on-winograndeFLAN 137B (0-shot)#33Accuracy: 71.2
coreference-resolution-on-winograd-schemaFLAN 137B (prompt-tuned)#13Accuracy: 86.5
coreference-resolution-on-winograd-schemaFLAN 137B (zero-shot)#17Accuracy: 80.8
machine-translation-on-wmt2014-english-frenchFLAN 137B (zero-shot)#46BLEU score: 33.9
machine-translation-on-wmt2014-english-frenchFLAN 137B (few-shot, k=9)#47BLEU score: 33.8
machine-translation-on-wmt2016-english-1FLAN 137B (few-shot, k=9)#18BLEU score: 20.5
machine-translation-on-wmt2016-english-1FLAN 137B (zero-shot)#19BLEU score: 18.9
machine-translation-on-wmt2016-english-germanFLAN 137B (zero-shot)#4BLEU score: 27.0
machine-translation-on-wmt2016-english-germanFLAN 137B (few-shot, k=11)#6BLEU score: 26.1
machine-translation-on-wmt2016-german-englishFLAN 137B (few-shot, k=11)#1BLEU score: 40.7
machine-translation-on-wmt2016-german-englishFLAN 137B (zero-shot)#2BLEU score: 38.9
machine-translation-on-wmt2016-romanianFLAN 137B (few-shot, k=9)#2BLEU score: 38.1
machine-translation-on-wmt2016-romanianFLAN 137B (zero-shot)#3BLEU score: 37.3
natural-language-inference-on-rteFLAN 137B (prompt-tuned)#11Accuracy: 91.7%
natural-language-inference-on-rteFLAN 137B (8-shot)#25Accuracy: 84.5%
natural-language-inference-on-rteFLAN 137B (0-shot)#26Accuracy: 84.1%
natural-language-inference-on-wnliFLAN 137B (zero-shot)#12Accuracy: 74.6
natural-language-inference-on-wnliFLAN 137B (few-shot, k=4)#15Accuracy: 70.4
question-answering-on-boolqFLAN 137B (prompt-tuned)#12Accuracy: 86.3
question-answering-on-boolqFLAN 137B (4-shot)#17Accuracy: 84.6
question-answering-on-boolqFLAN 137B (0-shot)#22Accuracy: 82.9
question-answering-on-copaFLAN 137B (prompt-tuned)#8Accuracy: 94
question-answering-on-copaFLAN 137B (zero-shot)#12Accuracy: 91
question-answering-on-copaFLAN 137B (few-shot, k=16)#21Accuracy: 87
question-answering-on-multircFLAN 137B (prompt-tuned)#9F1: 83.4
question-answering-on-multircFLAN 137B (zero-shot)#10F1: 77.5
question-answering-on-multircFLAN 137B (1-shot)#12F1: 72.1
question-answering-on-naturalqaFLAN 137B (zero-shot)#1EM: 20.7
question-answering-on-obqaFLAN 137B (zero-shot)#1Accuracy: 78.4
question-answering-on-obqaFLAN 137B (few-shot, k=16)#2Accuracy: 78.2
question-answering-on-piqaFLAN 137B (few-shot, k=10)#26Accuracy: 81.7
question-answering-on-piqaFLAN 137B (0-shot)#30Accuracy: 80.5
question-answering-on-storyclozeFLAN 137B (few-shot, k=10)#3Accuracy: 94.7
question-answering-on-storyclozeFLAN 137B (zero-shot)#5Accuracy: 93.4
question-answering-on-triviaqaFLAN 137B (zero-shot)#25EM: 56.7
sentence-completion-on-hellaswagFLAN 137B (3-shot)#50Accuracy: 59.2
sentence-completion-on-hellaswagFLAN 137B (0-shot)#52Accuracy: 56.7
sentiment-analysis-on-imdbFLAN 137B (few-shot, k=2)#17Accuracy: 95
sentiment-analysis-on-imdbFLAN 137B (zero-shot)#21Accuracy: 94.3