LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale

Benchmark Model Rank Results
language-modelling-on-c4Zeropoint LLM.int8 13B (vector-wise + decomp)#2Perplexity: 12.45
language-modelling-on-c4LLM.float32 6.7B#5Perplexity: 13.3
language-modelling-on-c4LLM.float32 2.7B#6Perplexity: 14.43
language-modelling-on-c4LLM.float32 1.3B#9Perplexity: 15.91
linguistic-acceptability-on-colaRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#12Accuracy: 68.6%
natural-language-inference-on-multinliRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#8Matched: 90.2
natural-language-inference-on-qnliRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#11Accuracy: 94.7%
natural-language-inference-on-rteRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#22Accuracy: 85.4%
semantic-textual-similarity-on-mrpcRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#6Accuracy: 91.0%
semantic-textual-similarity-on-sts-benchmarkRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#7Pearson Correlation: 0.919
sentiment-analysis-on-sst-2-binaryRoBERTa-large 355M (MLP quantized vector-wise, fine-tuned)#15Accuracy: 96.4