Prune Once for All: Sparse Pre-Trained Language Models

Benchmark Model Rank Results
natural-language-inference-on-multinli-devBERT-Large-uncased-PruneOFA (90% unstruct sparse)#2Matched: 83.74Mismatched: 84.2
natural-language-inference-on-multinli-devBERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)#3Matched: 83.47Mismatched: 84.08
natural-language-inference-on-multinli-devBERT-Base-uncased-PruneOFA (85% unstruct sparse)#4Matched: 82.71Mismatched: 83.67
natural-language-inference-on-multinli-devBERT-Base-uncased-PruneOFA (90% unstruct sparse)#5Matched: 81.45Mismatched: 82.43
natural-language-inference-on-multinli-devBERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)#6Matched: 81.4Mismatched: 82.51
natural-language-inference-on-multinli-devDistilBERT-uncased-PruneOFA (85% unstruct sparse)#7Matched: 81.35Mismatched: 82.03
natural-language-inference-on-multinli-devDistilBERT-uncased-PruneOFA (90% unstruct sparse)#8Matched: 80.68Mismatched: 81.47
natural-language-inference-on-multinli-devDistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)#9Matched: 80.66Mismatched: 81.14
natural-language-inference-on-multinli-devDistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)#10Matched: 78.8Mismatched: 80.4
question-answering-on-squad11-devBERT-Large-uncased-PruneOFA (90% unstruct sparse)#10EM: 83.35F1: 90.2
question-answering-on-squad11-devBERT-Large-uncased-PruneOFA (90% unstruct sparse, QAT Int8)#11EM: 83.22F1: 90.02
question-answering-on-squad11-devBERT-Base-uncased-PruneOFA (85% unstruct sparse)#12EM: 81.1F1: 88.42
question-answering-on-squad11-devBERT-Base-uncased-PruneOFA (85% unstruct sparse, QAT Int8)#13EM: 80.84F1: 88.24
question-answering-on-squad11-devBERT-Base-uncased-PruneOFA (90% unstruct sparse)#14EM: 79.83F1: 87.25
question-answering-on-squad11-devDistilBERT-uncased-PruneOFA (85% unstruct sparse)#19EM: 78.1F1: 85.82
question-answering-on-squad11-devDistilBERT-uncased-PruneOFA (85% unstruct sparse, QAT Int8)#21EM: 77.03F1: 85.13
question-answering-on-squad11-devDistilBERT-uncased-PruneOFA (90% unstruct sparse)#22EM: 76.91F1: 84.82
question-answering-on-squad11-devDistilBERT-uncased-PruneOFA (90% unstruct sparse, QAT Int8)#24EM: 75.62F1: 83.87