FNet: Mixing Tokens with Fourier Transforms

Benchmark Model Rank Results
paraphrase-identification-on-quora-question-pairsFNet-Large#5F1: 85
linguistic-acceptability-on-colaFNet-Large#7Accuracy: 78%
natural-language-understanding-on-glue-devFNet-Hybrid-Large#11GLUE (dev): 83.6
natural-language-inference-on-multinliBERT-Large#14Matched: 88Mismatched: 88
natural-language-inference-on-multinliFNet-Large#37Matched: 78Mismatched: 76
semantic-textual-similarity-on-mrpcFNet-Large#17Accuracy: 88%
sentiment-analysis-on-sst-2-binary-classificationFNet-Large#36Accuracy: 94
natural-language-inference-on-qnliFNet-Large#40Accuracy: 85%
semantic-textual-similarity-on-sts-benchmarkFNet-Large#44Spearman Correlation: 0.84
natural-language-inference-on-rteFNet-Large#55Accuracy: 69%