Scaling Instruction-Finetuned Language Models

Open paper
Benchmark Model Rank Results
coreference-resolution-on-winograd-schemaFlan-T5 XXL (zero -shot)#8Accuracy: 89.82
cross-lingual-question-answering-on-tydiqaFlan-U-PaLM 540B (direct-prompting)#2EM: 68.3
cross-lingual-question-answering-on-tydiqaFlan-PaLM 540B (direct-prompting)#3EM: 67.8
multi-task-language-understanding-on-bbh-algFlan-PaLM 540B (3-shot, fine-tuned, CoT + SC)#2Average (%): 66.5
multi-task-language-understanding-on-bbh-algPaLM 540B (CoT + self-consistency)#3Average (%): 62.2
multi-task-language-understanding-on-bbh-algFlan-PaLM 540B (3-shot, fine-tuned, CoT)#4Average (%): 61.3
multi-task-language-understanding-on-bbh-algPaLM 540B (CoT)#5Average (%): 57.6
multi-task-language-understanding-on-bbh-algFlan-PaLM 540B (3-shot, fine-tuned)#6Average (%): 48.2
multi-task-language-understanding-on-bbh-algPaLM 540B#7Average (%): 38.3
multi-task-language-understanding-on-bbh-nlpFlan-PaLM 540B (3-shot, fine-tuned, CoT + SC)#1Average (%): 78.4
multi-task-language-understanding-on-bbh-nlpPaLM 540B (CoT + self-consistency)#2Average (%): 78.2
multi-task-language-understanding-on-bbh-nlpFlan-PaLM 540B (3-shot, fine-tuned, CoT)#4Average (%): 72.4
multi-task-language-understanding-on-bbh-nlpPaLM 540B (CoT)#5Average (%): 71.2
multi-task-language-understanding-on-bbh-nlpFlan-PaLM 540B (5-shot, finetuned)#6Average (%): 70.0
multi-task-language-understanding-on-bbh-nlpPaLM 540B#7Average (%): 62.7
multi-task-language-understanding-on-mgsmFlan-PaLM 540B (8-shot, fine-tuned, CoT + SC)#3Average (%): 72.0
multi-task-language-understanding-on-mgsmFlan-U-PaLM 540B (CoT)#4Average (%): 60.4
multi-task-language-understanding-on-mgsmFlan-PaLM 540B (8-shot, fine-tuned, CoT)#5Average (%): 57.0
multi-task-language-understanding-on-mgsmtext-davinci-003#7Average (%): 36
multi-task-language-understanding-on-mgsmcode-davinci-002#8Average (%): 35
multi-task-language-understanding-on-mgsmtext-davinci-002#9Average (%): 23.7
multi-task-language-understanding-on-mgsmFlan-PaLM 540B (8-shot, fine-tuned)#10Average (%): 21.2
multi-task-language-understanding-on-mgsmGPT-3 Davinci 175B#11Average (%): 5.7
multi-task-language-understanding-on-mmlullama 2(65b)#4Average (%): 73.5
multi-task-language-understanding-on-mmluGPT-3 Davinci 175B (CoT)#14Average (%): 59.5
multi-task-language-understanding-on-mmluFlan-T5-XL 3B (CoT)#20Average (%): 45.5
multi-task-language-understanding-on-mmluFlan-T5-Large 780M#22Average (%): 45.1
multi-task-language-understanding-on-mmluFlan-T5-Large 780M (CoT)#24Average (%): 40.5
multi-task-language-understanding-on-mmluGPT-3 Davinci 175B (5-shot)#25Average (%): 39.7
multi-task-language-understanding-on-mmluFlan-T5-Base 250M#31Average (%): 35.9
multi-task-language-understanding-on-mmluFlan-T5-Base 250M (CoT)#32Average (%): 33.7
multi-task-language-understanding-on-mmluFlan-T5-Small 80M#35Average (%): 28.7

Rank counts only results with a code link.