Language Models are Unsupervised Multitask Learners

Benchmark Model Rank Results
coreference-resolution-on-winograd-schemaGPT-2-XL 1.5B#28Accuracy: 70.7
dialogue-state-tracking-on-simmc2-0GPT-2#2Act F1: 94.5Slot F1: 81.7
document-summarization-on-cnn-daily-mailGPT-2#23ROUGE-1: 29.34ROUGE-2: 8.27ROUGE-L: 26.58
language-modelling-on-enwiki8GPT-2 (48 layers, h=1600)#1Bit per Character (BPC): 0.93Number of params: 1542M
language-modelling-on-lambadaGPT-2 1.5B (Zero Shot)#23Accuracy: 63.24Perplexity: 8.63
language-modelling-on-one-billion-wordGPT-2#22PPL: 42.16Number of params: 1.54B
language-modelling-on-penn-treebank-wordGPT-2#3Test perplexity: 35.76Params: 1542M
language-modelling-on-text8GPT-2#1Bit per Character (BPC): 0.98Number of params: 1542M
language-modelling-on-wikitext-103GPT-2 Full#25Test perplexity: 17.48Number of params: 1542M
language-modelling-on-wikitext-103GPT-2 Large#45Test perplexity: 22.05Number of params: 774M
language-modelling-on-wikitext-103GPT-2 Medium#61Test perplexity: 26.37Number of params: 355M
language-modelling-on-wikitext-103GPT-2 Small#72Test perplexity: 37.50Number of params: 124M
language-modelling-on-wikitext-2GPT-2#6Test perplexity: 18.34Number of params: 1542M
language-modelling-on-wikitext-2GPT-2 (large)#7Test perplexity: 19.93Number of params: 762M
language-modelling-on-wikitext-2GPT-2 (medium)#8Test perplexity: 22.76Number of params: 345M
language-modelling-on-wikitext-2GPT-2 (small)#9Test perplexity: 29.41Number of params: 117M
question-answering-on-feverZero-shot#7EM: 50
question-answering-on-webquestionsZero-shot#9EM: 43
response-generation-on-simmc2-0GPT-2#3BLEU: 19.2
sentiment-analysis-on-imdbGPT-2 Finetuned#29Accuracy: 92.36
text-generation-on-openwebtextGPT2-124M#2eval_loss: 3.12