Deep Equilibrium Models

Benchmark Model Rank Results
language-modelling-on-penn-treebank-word-levelDEQ-TrellisNet#28Test perplexity: 57.1Params: 24M
language-modelling-on-wikitext-103DEQ-Transformer (medium, adaptive embed)#53Test perplexity: 23.2Number of params: 110M
language-modelling-on-wikitext-103DEQ-TrellisNet#68Test perplexity: 29.0Number of params: 180M
language-modelling-on-wikitext-103DEQ-Transformer (small)#72Test perplexity: 32.4Number of params: 138M