A Dot Product Attention Free Transformer

Benchmark Model Rank Results
image-classification-on-imagenetDAFT-conv (16 heads)Top 1 Accuracy: 80.2%Number of params: 20.3M
image-classification-on-imagenetDAFT-conv (384 heads, 200 epochs)Top 1 Accuracy: 80.1%Number of params: 23M
image-classification-on-imagenetDAFT-conv (384 heads, 300 epochs)Top 1 Accuracy: 80.8%Number of params: 23M
image-classification-on-imagenetDAFT-fullTop 1 Accuracy: 79.8%Number of params: 22.6M