How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?

Benchmark Model Rank Results
language-modelling-on-wikitext-103LSTM#83Validation perplexity: 52.73
language-modelling-on-wikitext-103GRU#84Validation perplexity: 53.78
language-modelling-on-wikitext-103Decay RNN#85Validation perplexity: 76.67