SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Benchmark Model Rank Results
paraphrase-identification-on-quora-question-pairsALICE#1F1: 90.7
paraphrase-identification-on-quora-question-pairsFreeLB#24Accuracy: 74.8Dev Accuracy: 92.6
paraphrase-identification-on-quora-question-pairsSMART-BERT#26Dev Accuracy: 91.5Dev F1: 88.5
semantic-textual-similarity-on-mrpcMT-DNN-SMART#1Accuracy: 93.7%F1: 91.7
semantic-textual-similarity-on-mrpcSMART#5Accuracy: 91.3%
semantic-textual-similarity-on-sts-benchmarkMT-DNN-SMART#1Pearson Correlation: 0.929Spearman Correlation: 0.925
semantic-textual-similarity-on-sts-benchmarkSMARTRoBERTa#58Dev Pearson Correlation: 92.8Dev Spearman Correlation: 92.6
semantic-textual-similarity-on-sts-benchmarkSMART-BERT#59Dev Pearson Correlation: 90.0Dev Spearman Correlation: 89.4
natural-language-inference-on-multinliT5#2Matched: 92.0Mismatched: 91.7
natural-language-inference-on-multinliMT-DNN-SMARTv0#52Accuracy: 85.7
natural-language-inference-on-multinliMT-DNN-SMART#53Accuracy: 85.7
natural-language-inference-on-multinliSMART+BERT-BASE#54Accuracy: 85.6
natural-language-inference-on-multinliSMARTRoBERTa#56Dev Matched: 91.1Dev Mismatched: 91.3
natural-language-inference-on-multinliSMART-BERT#57Dev Matched: 85.6Dev Mismatched: 86.0
natural-language-inference-on-qnliALICE#2Accuracy: 99.2%
natural-language-inference-on-qnliMT-DNN-SMART#3Accuracy: 99.2%
sentiment-analysis-on-sst-2-binary-classificationMT-DNN-SMART#2Accuracy: 97.5
sentiment-analysis-on-sst-2-binary-classificationMT-DNN#37Accuracy: 93.6
sentiment-analysis-on-sst-2-binary-classificationSMART+BERT-BASE#42Accuracy: 93
sentiment-analysis-on-sst-2-binary-classificationSMARTRoBERTa#80Dev Accuracy: 96.9
sentiment-analysis-on-sst-2-binary-classificationSMART-MT-DNN#81Dev Accuracy: 96.1
sentiment-analysis-on-sst-2-binary-classificationSMART-BERT#82Dev Accuracy: 93.0
natural-language-inference-on-rteT5-XXL 11B#7Accuracy: 92.5%
natural-language-inference-on-rteSMARTRoBERTa#10Accuracy: 92.0%
natural-language-inference-on-rteSMART-BERT#46Accuracy: 71.2%
natural-language-inference-on-rteSMART#47Accuracy: 71.2%
natural-language-inference-on-scitailMT-DNN-SMART_100%ofTrainingData#7Dev Accuracy: 96.1
natural-language-inference-on-scitailMT-DNN-SMART_10%ofTrainingData#8Dev Accuracy: 91.3
natural-language-inference-on-scitailMT-DNN-SMART_1%ofTrainingData#9Dev Accuracy: 88.6
natural-language-inference-on-scitailMT-DNN-SMART_0.1%ofTrainingData#10Dev Accuracy: 82.3
natural-language-inference-on-scitailMT-DNN-SMARTLARGEv0#11% Dev Accuracy: 96.6% Test Accuracy: 95.2
natural-language-inference-on-snliMT-DNN-SMARTLARGEv0#7% Test Accuracy: 91.7% Dev Accuracy: 92.6
natural-language-inference-on-snliMT-DNN-SMART_100%ofTrainingData#69Dev Accuracy: 91.6
natural-language-inference-on-snliMT-DNN-SMART_10%ofTrainingData#70Dev Accuracy: 88.7
natural-language-inference-on-snliMT-DNN-SMART_1%ofTrainingData#71Dev Accuracy: 86
natural-language-inference-on-snliMT-DNN-SMART_0.1%ofTrainingData#72Dev Accuracy: 82.7