Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Open paper
Benchmark Model Rank Results
question-answering-on-piqaGQE (renorm. scoring + shared head)–Accuracy: 64.69
sentence-completion-on-hellaswagGQE (renorm. scoring + shared head)–Accuracy: 41.01

Rank counts only results with a code link.