Improving language models by retrieving from trillions of tokens
Open paper
Benchmark
Model
Rank
Results
language-modelling-on-wikitext-103
RETRO (7.5B)
#1
Test perplexity: 2.4
Number of params: 7532M
question-answering-on-natural-questions
RETRO + DPR (full)
#9
EM: 45.5
Rank counts only results with a code link.