General-purpose, long-context autoregressive modeling with Perceiver AR

Open paper

Benchmark	Model	Rank	Results
language-modelling-on-wikitext-103	Perceiver AR 358M	#36	Test perplexity: 18.4