SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs

Benchmark Model Rank Results
language-modelling-on-lambadaLlama-3-70B with SQ-format (W4A(SQ6), static, bb=16)–Perplexity: 2.64
question-answering-on-piqaLlama-3-70B with SQ-format (W4A(SQ6))–Accuracy: 84.06
sentence-completion-on-hellaswagLlama-3-70B with SQ-format (W4A(SQ6))–Accuracy: 84.44