Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

Open paper

Benchmark	Model	Rank	Results
question-answering-on-natural-questions	GDN 1.3B (3:1 hybrid, 50B tokens)	#53	EM: 24.9
question-answering-on-piqa	GDN 1.3B (3:1 hybrid, 50B tokens)	#148	Accuracy: 71.55