Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Benchmark Model Rank Results
abstractive-text-summarization-on-cnn-dailyT5#21ROUGE-1: 43.52ROUGE-2: 21.55ROUGE-L: 40.69
answer-generation-on-weibopollsT5#2ROUGE-1: 46.20ROUGE-L: 43.32BLEU-1: 37.77BLEU-3: 25.86
common-sense-reasoning-on-recordT5-XXL 11B (fine-tuned)#4EM: 93.4
common-sense-reasoning-on-recordT5-11B#14F1: 94.1
coreference-resolution-on-winograd-schemaT5-XXL 11B (fine-tuned)#5Accuracy: 93.8
document-summarization-on-cnn-daily-mailT5-11B#9ROUGE-1: 43.52ROUGE-2: 21.55ROUGE-L: 40.69
linguistic-acceptability-on-colaT5-11B#8Accuracy: 70.8%
linguistic-acceptability-on-colaT5-XL 3B#16Accuracy: 67.1%
linguistic-acceptability-on-colaT5-Large 770M#20Accuracy: 61.2%
linguistic-acceptability-on-colaT5-Base#29Accuracy: 51.1%
linguistic-acceptability-on-colaT5-Small#33Accuracy: 41.0%
machine-translation-on-wmt2014-english-frenchT5#9BLEU score: 43.4
machine-translation-on-wmt2014-english-germanT5-11B#4BLEU score: 32.1Number of Params: 11110M
multimodal-intent-recognition-on-photochatT5-3B#2F1: 58.9Precision: 54.1Recall: 64.6
multimodal-intent-recognition-on-photochatT5-base#3F1: 58.1Precision: 58.2Recall: 57.9
natural-language-inference-on-commitmentbankT5-XXL 11B (fine-tuned)#5Accuracy: 96.8F1: 93.9
natural-language-inference-on-commitmentbankT5-Large 770M (fine-tuned)#6Accuracy: 94.4F1: 90.3
natural-language-inference-on-commitmentbankT5-Base 220M (fine-tuned)#7Accuracy: 94F1: 86.2
natural-language-inference-on-multinliT5-XXL 11B (fine-tuned)#1Matched: 92.0
natural-language-inference-on-multinliT5-3B#3Matched: 91.4Mismatched: 91.2
natural-language-inference-on-multinliT5-Large#9Matched: 89.9
natural-language-inference-on-multinliT5-Base#18Matched: 87.1Mismatched: 86.2
natural-language-inference-on-multinliT5-Small#31Matched: 82.4Mismatched: 82.3
natural-language-inference-on-multinliT5-11B#48Mismatched: 91.7
natural-language-inference-on-multinliT5-Large 770M#50Mismatched: 89.6
natural-language-inference-on-qnliT5-11B#5Accuracy: 96.7%
natural-language-inference-on-qnliT5-3B#6Accuracy: 96.3%
natural-language-inference-on-qnliT5-Large 770M#10Accuracy: 94.8%
natural-language-inference-on-qnliT5-Base#16Accuracy: 93.7%
natural-language-inference-on-qnliT5-Small#29Accuracy: 90.3%
natural-language-inference-on-rteT5-XXL 11B (fine-tuned)#6Accuracy: 92.5%
natural-language-inference-on-rteT5-XL 3B#12Accuracy: 91.1%
natural-language-inference-on-rteT5-Large 770M#18Accuracy: 87.2%
natural-language-inference-on-rteT5-Base 220M#31Accuracy: 80.1%
natural-language-inference-on-rteT5-Small#47Accuracy: 69.9%
natural-language-inference-on-wnliT5-XXL 11B#2Accuracy: 93.2
natural-language-inference-on-wnliT5-XL 3B#5Accuracy: 89.7
natural-language-inference-on-wnliT5-Large 770M#8Accuracy: 85.6
natural-language-inference-on-wnliT5-Base 220M#10Accuracy: 78.8
natural-language-inference-on-wnliT5-Small 60M#16Accuracy: 69.2
poll-generation-on-weibopollsT5#2ROUGE-1: 45.33ROUGE-L: 42.69BLEU-1: 37.34BLEU-3: 21.06
question-answering-on-boolqT5-XXL 11B (fine-tuned)#4Accuracy: 91.2
question-answering-on-boolqT5-Large 770M (fine-tuned)#14Accuracy: 85.4
question-answering-on-boolqT5-Base 220M (fine-tuned)#24Accuracy: 81.4
question-answering-on-boolqT5-Small 60M (fine-tuned)#31Accuracy: 76.4
question-answering-on-copaT5-XXL 11B (fine-tuned)#7Accuracy: 94.8
question-answering-on-copaT5-XL 3B (fine-tuned)#9Accuracy: 92
question-answering-on-copaT5-Large 770M (fine-tuned)#31Accuracy: 83.4
question-answering-on-copaT5-Base 220M (fine-tuned)#41Accuracy: 71.2
question-answering-on-multircT5-XXL 11B (fine-tuned)#5F1: 88.1
question-answering-on-multircT5-11B#24EM: 63.3
question-answering-on-quora-question-pairsT5-11B#4Accuracy: 90.4%
question-answering-on-quora-question-pairsT5-Large 770M#8Accuracy: 89.9%
question-answering-on-quora-question-pairsT5-3B#10Accuracy: 89.7%
question-answering-on-quora-question-pairsT5-Base#11Accuracy: 89.4%
question-answering-on-quora-question-pairsT5-Small#15Accuracy: 88.0%
question-answering-on-squad11-devT5-11B#1EM: 90.06F1: 95.64
question-answering-on-squad11-devT5-3B#5EM: 88.53F1: 94.95
question-answering-on-squad11-devT5-Large 770M#6EM: 86.66F1: 93.79
question-answering-on-squad11-devT5-Base#8EM: 85.44F1: 92.08
question-answering-on-squad11-devT5-Small#16EM: 79.1F1: 87.24
question-answering-on-webquestionsT5.1.1-XXL+SSM#11EM: 42.8
question-generation-on-weibopollsT5#2ROUGE-1: 44.46ROUGE-L: 42.06BLEU-1: 36.91BLEU-3: 16.26
semantic-parsing-on-webquestionsspT5-11B (Raffel et al., 2020)#3Accuracy: 56.5
semantic-textual-similarity-on-mrpcT5-11B#11Accuracy: 90.0%F1: 91.9
semantic-textual-similarity-on-mrpcT5-Large#12Accuracy: 89.9%F1: 92.4
semantic-textual-similarity-on-mrpcT5-3B#14Accuracy: 89.2%F1: 92.5
semantic-textual-similarity-on-mrpcT5-Base#19Accuracy: 87.5%F1: 90.7
semantic-textual-similarity-on-mrpcT5-Small#26Accuracy: 86.6%F1: 89.7
semantic-textual-similarity-on-sts-benchmarkT5-11B#4Pearson Correlation: 0.925Spearman Correlation: 0.921
semantic-textual-similarity-on-sts-benchmarkT5-3B#12Pearson Correlation: 0.906Spearman Correlation: 0.898
semantic-textual-similarity-on-sts-benchmarkT5-Large#15Pearson Correlation: 0.899
semantic-textual-similarity-on-sts-benchmarkT5-Base#16Pearson Correlation: 0.894
semantic-textual-similarity-on-sts-benchmarkT5-Small#19Pearson Correlation: 0.856Spearman Correlation: 0.85
semantic-textual-similarity-on-sts-benchmarkT5-Large 770M#29Spearman Correlation: 0.886
sentiment-analysis-on-sst-2-binaryT5-11B#1Accuracy: 97.5
sentiment-analysis-on-sst-2-binaryT5-3B#3Accuracy: 97.4
sentiment-analysis-on-sst-2-binaryT5-Large 770M#16Accuracy: 96.3
sentiment-analysis-on-sst-2-binaryT5-Base#23Accuracy: 95.2
sentiment-analysis-on-sst-2-binaryT5-Small#44Accuracy: 91.8
word-sense-disambiguation-on-words-in-contextT5-XXL 11B#6Accuracy: 76.9