| Mogrifier LSTM | ✓ Link | 1.083 | 24M | Mogrifier LSTM + dynamic eval | 2019-09-04 |
| Mogrifier LSTM | ✓ Link | 1.120 | 24M | Mogrifier LSTM | 2019-09-04 |
| Recurrent Highway Networks with Grouped Auxiliary Memory | ✓ Link | 1.147 | 16.0M | GAM-RHN-5 | 2019-12-13 |
| Trellis Networks for Sequence Modeling | ✓ Link | 1.158 | 13.4M | Trellis Network | 2018-10-15 |
| Addressing Some Limitations of Transformers with Feedback Memory | ✓ Link | 1.160 | 10.7M | Feedback Transformer | 2020-02-21 |
| Improved Language Modeling by Decoding the Past | | 1.169 | 13.8M | Past Decode Reg. + AWD-LSTM-MoS + dyn. eval. | 2018-08-14 |
| An Analysis of Neural Language Modeling at Multiple Scales | ✓ Link | 1.175 | 13.8M | 3-layer AWD-LSTM | 2018-03-22 |
| Deep Independently Recurrent Neural Network (IndRNN) | ✓ Link | 1.18 | | Dense IndRNN | 2019-10-11 |
| An Analysis of Neural Language Modeling at Multiple Scales | ✓ Link | 1.187 | 13.8M | 6-layer QRNN | 2018-03-22 |
| Fast-Slow Recurrent Neural Networks | ✓ Link | 1.190 | 27M | FS-LSTM-4 | 2017-05-24 |
| Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN | ✓ Link | 1.19 | | IndRNN | 2018-03-13 |
| Fast-Slow Recurrent Neural Networks | ✓ Link | 1.193 | 27M | FS-LSTM-2 | 2017-05-24 |
| Neural Architecture Search with Reinforcement Learning | ✓ Link | 1.214 | 16.3M | NAS-RL | 2016-11-05 |
| HyperNetworks | ✓ Link | 1.219 | 14.4M | 2-layer Norm HyperLSTM | 2016-09-27 |
| R-Transformer: Recurrent Neural Network Enhanced Transformer | ✓ Link | 1.24 | | R-Transformer | 2019-07-12 |
| Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling | ✓ Link | 1.3 | 5.9M | Seq-U-Net | 2019-11-14 |
| Gating Revisited: Deep Multi-layer RNNs That Can Be Trained | ✓ Link | 1.30 | | STAR | 2019-11-25 |
| Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling | ✓ Link | 1.31 | 5.9M | TCN | 2019-11-14 |
| An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling | ✓ Link | 1.31 | | Temporal Convolutional Network | 2018-03-04 |
| Discrete Flows: Invertible Generative Models of Discrete Data | ✓ Link | 1.38 | | Bipartite Flow | 2019-05-24 |