How much complexity does an RNN architecture need to learn syntax-sensitive dependencies?

Benchmark Model Rank Results
language-modelling-on-wikitext-103LSTM#79Validation perplexity: 52.73
language-modelling-on-wikitext-103GRU#80Validation perplexity: 53.78
language-modelling-on-wikitext-103Decay RNN#81Validation perplexity: 76.67