Finetuned Language Models Are Zero-Shot Learners

Benchmark Model Rank Results
machine-translation-on-wmt2016-german-englishFLAN 137B (few-shot, k=11)#1BLEU score: 40.7
machine-translation-on-wmt2016-german-englishFLAN 137B (zero-shot)#2BLEU score: 38.9
question-answering-on-naturalqaFLAN 137B (zero-shot)#1EM: 20.7
question-answering-on-obqaFLAN 137B (zero-shot)#1Accuracy: 78.4
question-answering-on-obqaFLAN 137B (few-shot, k=16)#2Accuracy: 78.2
machine-translation-on-wmt2016-romanian-englishFLAN 137B (few-shot, k=9)#2BLEU score: 38.1
machine-translation-on-wmt2016-romanian-englishFLAN 137B (zero-shot)#3BLEU score: 37.3
question-answering-on-storyclozeFLAN 137B (few-shot, k=10)#3Accuracy: 94.7
question-answering-on-storyclozeFLAN 137B (zero-shot)#5Accuracy: 93.4
machine-translation-on-wmt2016-english-germanFLAN 137B (zero-shot)#4BLEU score: 27.0
machine-translation-on-wmt2016-english-germanFLAN 137B (few-shot, k=11)#6BLEU score: 26.1
common-sense-reasoning-on-recordFLAN 137B (prompt-tuned)#8EM: 85.1
common-sense-reasoning-on-recordFLAN 137B (zero-shot)#11EM: 72.5
question-answering-on-copaFLAN 137B (prompt-tuned)#8Accuracy: 94
question-answering-on-copaFLAN 137B (zero-shot)#12Accuracy: 91
question-answering-on-copaFLAN 137B (few-shot, k=16)#21Accuracy: 87
question-answering-on-multircFLAN 137B (prompt-tuned)#9F1: 83.4
question-answering-on-multircFLAN 137B (zero-shot)#10F1: 77.5
question-answering-on-multircFLAN 137B (1-shot)#12F1: 72.1
natural-language-inference-on-rteFLAN 137B (prompt-tuned)#11Accuracy: 91.7%
natural-language-inference-on-rteFLAN 137B (8-shot)#27Accuracy: 84.5%
natural-language-inference-on-rteFLAN 137B (0-shot)#28Accuracy: 84.1%
common-sense-reasoning-on-arc-challengeFLAN 137B (few-shot, k=13)#12Accuracy: 63.8
common-sense-reasoning-on-arc-challengeFLAN 137B (zero-shot)#13Accuracy: 63.1
coreference-resolution-on-winograd-schema-challengeFLAN 137B (prompt-tuned)#13Accuracy: 86.5
coreference-resolution-on-winograd-schema-challengeFLAN 137B (zero-shot)#17Accuracy: 80.8
natural-language-inference-on-wnliFLAN 137B (zero-shot)#14Accuracy: 74.6
natural-language-inference-on-wnliFLAN 137B (few-shot, k=4)#17Accuracy: 70.4
question-answering-on-boolqFLAN 137B (prompt-tuned)#17Accuracy: 86.3
question-answering-on-boolqFLAN 137B (4-shot)#27Accuracy: 84.6
question-answering-on-boolqFLAN 137B (0-shot)#36Accuracy: 82.9
sentiment-analysis-on-imdbFLAN 137B (few-shot, k=2)#17Accuracy: 95
sentiment-analysis-on-imdbFLAN 137B (zero-shot)#21Accuracy: 94.3
machine-translation-on-wmt2016-english-romanianFLAN 137B (few-shot, k=9)#18BLEU score: 20.5
machine-translation-on-wmt2016-english-romanianFLAN 137B (zero-shot)#19BLEU score: 18.9
common-sense-reasoning-on-arc-easyFLAN 137B (few-shot, k=14)#21Accuracy: 80.7
common-sense-reasoning-on-arc-easyFLAN 137B (0-shot)#25Accuracy: 79.6
question-answering-on-triviaqaFLAN 137B (zero-shot)#32EM: 56.7
question-answering-on-piqaFLAN 137B (few-shot, k=10)#44Accuracy: 81.7
question-answering-on-piqaFLAN 137B (0-shot)#57Accuracy: 80.5
machine-translation-on-wmt2014-english-frenchFLAN 137B (zero-shot)#46BLEU score: 33.9
machine-translation-on-wmt2014-english-frenchFLAN 137B (few-shot, k=9)#47BLEU score: 33.8
common-sense-reasoning-on-winograndeFLAN 137B (few-shot, k=16)#67Accuracy: 72.8
common-sense-reasoning-on-winograndeFLAN 137B (0-shot)#73Accuracy: 71.2
sentence-completion-on-hellaswagFLAN 137B (3-shot)#153Accuracy: 59.2
sentence-completion-on-hellaswagFLAN 137B (0-shot)#161Accuracy: 56.7