LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

Benchmark Model Rank Results
word-sense-disambiguation-on-words-in-contextFLAN-T5-Large 783M#10Accuracy: 64.7
word-sense-disambiguation-on-words-in-contextLaMini-F-T5 783M#11Accuracy: 63.8
word-sense-disambiguation-on-words-in-contextLaMini-GPT 1.5B#16Accuracy: 52.4
word-sense-disambiguation-on-words-in-contextLaMini-T5 738M#21Accuracy: 50.5
word-sense-disambiguation-on-words-in-contextGPT-2-XL 1.5B#24Accuracy: 49.8
natural-language-inference-on-rteT5-Large 738M#19Accuracy: 87.4%
natural-language-inference-on-rteLaMini-GPT 1.5B#58Accuracy: 67.9%
natural-language-inference-on-rteLaMini-F-T5 783M#61Accuracy: 65%
natural-language-inference-on-rteLaMini-T5 738M#76Accuracy: 57%
natural-language-inference-on-rteGPT-2-XL 1.5B#83Accuracy: 52.3%
question-answering-on-openbookqaLaMini-GPT 1.5B#23Accuracy: 39.8
question-answering-on-openbookqaLaMini-T5 738M#24Accuracy: 36
question-answering-on-openbookqaLaMini-F-T5 783M#25Accuracy: 34
question-answering-on-openbookqaT5-Large 738M#26Accuracy: 32.8
question-answering-on-openbookqaGPT-2-XL 1.5B#27Accuracy: 32
question-answering-on-openbookqaFLAN-T5-Large 783M#28Accuracy: 31.2
coreference-resolution-on-winograd-schema-challengeGPT-2-XL 1.5B#24Accuracy: 73.3
coreference-resolution-on-winograd-schema-challengeLaMini-GPT 1.5B#30Accuracy: 69.6
coreference-resolution-on-winograd-schema-challengeT5-Large 738M#35Accuracy: 66.7
coreference-resolution-on-winograd-schema-challengeLaMini-F-T5 783M#38Accuracy: 64.1
coreference-resolution-on-winograd-schema-challengeLaMini-T5 738M#50Accuracy: 59
natural-language-inference-on-multinliT5-Large 738M#38Matched: 72.4Mismatched: 72
natural-language-inference-on-multinliLaMini-GPT 1.5B#45Matched: 67.5Mismatched: 69.3
natural-language-inference-on-multinliLaMini-F-T5 783M#46Matched: 61.4Mismatched: 61
natural-language-inference-on-multinliLaMini-T5 738M#47Matched: 54.7Mismatched: 55.8
natural-language-inference-on-multinliGPT-2-XL 1.5B#48Matched: 36.5Mismatched: 37
common-sense-reasoning-on-winograndeFLAN-T5-Large 783M#119Accuracy: 59.9
common-sense-reasoning-on-winograndeGPT-2-XL 1.5B#134Accuracy: 58.3
common-sense-reasoning-on-winograndeLaMini-F-T5 783M#144Accuracy: 56
common-sense-reasoning-on-winograndeLaMini-GPT 1.5B#145Accuracy: 56
common-sense-reasoning-on-winograndeT5-Large 738M#151Accuracy: 55.2
common-sense-reasoning-on-winograndeLaMini-T5 738M#152Accuracy: 54.9
question-answering-on-piqaFLAN-T5-Large 783M#140Accuracy: 72.2
question-answering-on-piqaLaMini-GPT 1.5B#149Accuracy: 71.3
question-answering-on-piqaLaMini-F-T5 783M#152Accuracy: 70.6
question-answering-on-piqaGPT-2-XL 1.5B#153Accuracy: 70.5
question-answering-on-piqaLaMini-T5 738M#160Accuracy: 67.2
question-answering-on-piqaT5-Large 738M#176Accuracy: 55.9
sentence-completion-on-hellaswagGPT-2-XL 1.5B#178Accuracy: 50.9
sentence-completion-on-hellaswagFLAN-T5-Large 783M#184Accuracy: 48.7
sentence-completion-on-hellaswagLaMini-GPT 1.5B#186Accuracy: 48.3
sentence-completion-on-hellaswagLaMini-F-T5 783M#192Accuracy: 43.7
sentence-completion-on-hellaswagLaMini-T5 738M#202Accuracy: 40.6
sentence-completion-on-hellaswagT5-Large 738M#204Accuracy: 38.9