SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

Benchmark Model Rank Results
natural-language-inference-on-multinliT5#2Matched: 92.0Mismatched: 91.7
natural-language-inference-on-multinliMT-DNN-SMARTv0#51Accuracy: 85.7
natural-language-inference-on-multinliMT-DNN-SMART#52Accuracy: 85.7
natural-language-inference-on-multinliSMART+BERT-BASE#53Accuracy: 85.6
natural-language-inference-on-multinliSMARTRoBERTa#55Dev Matched: 91.1Dev Mismatched: 91.3
natural-language-inference-on-multinliSMART-BERT#56Dev Matched: 85.6Dev Mismatched: 86.0
natural-language-inference-on-qnliALICE#2Accuracy: 99.2%
natural-language-inference-on-qnliMT-DNN-SMART#3Accuracy: 99.2%
natural-language-inference-on-rteT5-XXL 11B#7Accuracy: 92.5%
natural-language-inference-on-rteSMARTRoBERTa#10Accuracy: 92.0%
natural-language-inference-on-rteSMART-BERT#43Accuracy: 71.2%
natural-language-inference-on-rteSMART#44Accuracy: 71.2%
natural-language-inference-on-scitailMT-DNN-SMART_100%ofTrainingData#7Dev Accuracy: 96.1
natural-language-inference-on-scitailMT-DNN-SMART_10%ofTrainingData#8Dev Accuracy: 91.3
natural-language-inference-on-scitailMT-DNN-SMART_1%ofTrainingData#9Dev Accuracy: 88.6
natural-language-inference-on-scitailMT-DNN-SMART_0.1%ofTrainingData#10Dev Accuracy: 82.3
natural-language-inference-on-scitailMT-DNN-SMARTLARGEv0#11% Dev Accuracy: 96.6% Test Accuracy: 95.2
natural-language-inference-on-snliMT-DNN-SMARTLARGEv0#7% Test Accuracy: 91.7% Dev Accuracy: 92.6
natural-language-inference-on-snliMT-DNN-SMART_100%ofTrainingData#69Dev Accuracy: 91.6
natural-language-inference-on-snliMT-DNN-SMART_10%ofTrainingData#70Dev Accuracy: 88.7
natural-language-inference-on-snliMT-DNN-SMART_1%ofTrainingData#71Dev Accuracy: 86
natural-language-inference-on-snliMT-DNN-SMART_0.1%ofTrainingData#72Dev Accuracy: 82.7
paraphrase-identification-on-quora-questionALICE#1F1: 90.7
paraphrase-identification-on-quora-questionFreeLB#23Accuracy: 74.8Dev Accuracy: 92.6
paraphrase-identification-on-quora-questionSMART-BERT#25Dev Accuracy: 91.5Dev F1: 88.5
semantic-textual-similarity-on-mrpcMT-DNN-SMART#1Accuracy: 93.7%F1: 91.7
semantic-textual-similarity-on-mrpcSMART#5Accuracy: 91.3%
semantic-textual-similarity-on-sts-benchmarkMT-DNN-SMART#1Pearson Correlation: 0.929Spearman Correlation: 0.925
semantic-textual-similarity-on-sts-benchmarkSMARTRoBERTa#58Dev Pearson Correlation: 92.8Dev Spearman Correlation: 92.6
semantic-textual-similarity-on-sts-benchmarkSMART-BERT#59Dev Pearson Correlation: 90.0Dev Spearman Correlation: 89.4
sentiment-analysis-on-sst-2-binaryMT-DNN-SMART#2Accuracy: 97.5
sentiment-analysis-on-sst-2-binaryMT-DNN#35Accuracy: 93.6
sentiment-analysis-on-sst-2-binarySMART+BERT-BASE#40Accuracy: 93
sentiment-analysis-on-sst-2-binarySMARTRoBERTa#76Dev Accuracy: 96.9
sentiment-analysis-on-sst-2-binarySMART-MT-DNN#77Dev Accuracy: 96.1
sentiment-analysis-on-sst-2-binarySMART-BERT#78Dev Accuracy: 93.0