TinyBERT: Distilling BERT for Natural Language Understanding

Benchmark Model Rank Results
linguistic-acceptability-on-colaTinyBERT-4 14.5M#32Accuracy: 43.3%
linguistic-acceptability-on-cola-devTinyBERT-6 67M#4Accuracy: 54
natural-language-inference-on-multinliTinyBERT-6 67M#26Matched: 84.6Mismatched: 83.2
natural-language-inference-on-multinliTinyBERT-4 14.5M#30Matched: 82.5Mismatched: 81.8
natural-language-inference-on-multinli-devTinyBERT-6 67M#1Matched: 84.5Mismatched: 84.5
natural-language-inference-on-qnliTinyBERT-6 67M#28Accuracy: 90.4%
natural-language-inference-on-qnliTinyBERT-4 14.5M#33Accuracy: 87.7%
natural-language-inference-on-rteTinyBERT-6 67M#56Accuracy: 66%
natural-language-inference-on-rteTinyBERT-4 14.5M#59Accuracy: 62.9%
paraphrase-identification-on-quora-questionTinyBERT#13F1: 71.3
question-answering-on-squad11-devTinyBERT-6 67M#15EM: 79.7F1: 87.5
question-answering-on-squad20-devTinyBERT-6 67M#12F1: 73.4EM: 69.9
semantic-textual-similarity-on-mrpcTinyBERT-6 67M#23Accuracy: 87.3%
semantic-textual-similarity-on-mrpcTinyBERT-4 14.5M#27Accuracy: 86.4%
semantic-textual-similarity-on-sts-benchmarkTinyBERT-4 14.5M#22Pearson Correlation: 0.799
sentiment-analysis-on-sst-2-binaryTinyBERT-6 67M#39Accuracy: 93.1
sentiment-analysis-on-sst-2-binaryTinyBERT-4 14.5M#42Accuracy: 92.6