LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

Benchmark Model Rank Results
common-sense-reasoning-on-winograndeFLAN-T5-Large 783M#47Accuracy: 59.9
common-sense-reasoning-on-winograndeGPT-2-XL 1.5B#51Accuracy: 58.3
common-sense-reasoning-on-winograndeLaMini-F-T5 783M#54Accuracy: 56
common-sense-reasoning-on-winograndeLaMini-GPT 1.5B#55Accuracy: 56
common-sense-reasoning-on-winograndeT5-Large 738M#56Accuracy: 55.2
common-sense-reasoning-on-winograndeLaMini-T5 738M#57Accuracy: 54.9
coreference-resolution-on-winograd-schemaGPT-2-XL 1.5B#24Accuracy: 73.3
coreference-resolution-on-winograd-schemaLaMini-GPT 1.5B#30Accuracy: 69.6
coreference-resolution-on-winograd-schemaT5-Large 738M#35Accuracy: 66.7
coreference-resolution-on-winograd-schemaLaMini-F-T5 783M#38Accuracy: 64.1
coreference-resolution-on-winograd-schemaLaMini-T5 738M#50Accuracy: 59
natural-language-inference-on-multinliT5-Large 738M#37Matched: 72.4Mismatched: 72
natural-language-inference-on-multinliLaMini-GPT 1.5B#44Matched: 67.5Mismatched: 69.3
natural-language-inference-on-multinliLaMini-F-T5 783M#45Matched: 61.4Mismatched: 61
natural-language-inference-on-multinliLaMini-T5 738M#46Matched: 54.7Mismatched: 55.8
natural-language-inference-on-multinliGPT-2-XL 1.5B#47Matched: 36.5Mismatched: 37
natural-language-inference-on-rteT5-Large 738M#17Accuracy: 87.4%
natural-language-inference-on-rteLaMini-GPT 1.5B#54Accuracy: 67.9%
natural-language-inference-on-rteLaMini-F-T5 783M#57Accuracy: 65%
natural-language-inference-on-rteLaMini-T5 738M#72Accuracy: 57%
natural-language-inference-on-rteGPT-2-XL 1.5B#78Accuracy: 52.3%
question-answering-on-openbookqaLaMini-GPT 1.5B#23Accuracy: 39.8
question-answering-on-openbookqaLaMini-T5 738M#24Accuracy: 36
question-answering-on-openbookqaLaMini-F-T5 783M#25Accuracy: 34
question-answering-on-openbookqaT5-Large 738M#26Accuracy: 32.8
question-answering-on-openbookqaGPT-2-XL 1.5B#27Accuracy: 32
question-answering-on-openbookqaFLAN-T5-Large 783M#28Accuracy: 31.2
question-answering-on-piqaFLAN-T5-Large 783M#51Accuracy: 72.2
question-answering-on-piqaLaMini-GPT 1.5B#52Accuracy: 71.3
question-answering-on-piqaLaMini-F-T5 783M#53Accuracy: 70.6
question-answering-on-piqaGPT-2-XL 1.5B#54Accuracy: 70.5
question-answering-on-piqaLaMini-T5 738M#57Accuracy: 67.2
question-answering-on-piqaT5-Large 738M#60Accuracy: 55.9
sentence-completion-on-hellaswagGPT-2-XL 1.5B#54Accuracy: 50.9
sentence-completion-on-hellaswagFLAN-T5-Large 783M#55Accuracy: 48.7
sentence-completion-on-hellaswagLaMini-GPT 1.5B#56Accuracy: 48.3
sentence-completion-on-hellaswagLaMini-F-T5 783M#58Accuracy: 43.7
sentence-completion-on-hellaswagLaMini-T5 738M#62Accuracy: 40.6
sentence-completion-on-hellaswagT5-Large 738M#64Accuracy: 38.9
word-sense-disambiguation-on-words-in-contextFLAN-T5-Large 783M#10Accuracy: 64.7
word-sense-disambiguation-on-words-in-contextLaMini-F-T5 783M#11Accuracy: 63.8
word-sense-disambiguation-on-words-in-contextLaMini-GPT 1.5B#16Accuracy: 52.4
word-sense-disambiguation-on-words-in-contextLaMini-T5 738M#21Accuracy: 50.5
word-sense-disambiguation-on-words-in-contextGPT-2-XL 1.5B#24Accuracy: 49.8