The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles

Benchmark Model Rank Results
graph-regression-on-pcqm4mv2-lscEGT+SSA+Self-ensemble#14Validation MAE: 0.0865
graph-regression-on-pcqm4mv2-lscEGT+SSA#16Validation MAE: 0.0876
language-modelling-on-enwiki8Transformer+SSA#21Bit per Character (BPC): 1.024
language-modelling-on-wikitext-103Transformer+SSA+Self-ensemble#27Test perplexity: 17.18Validation perplexity: 16.54
language-modelling-on-wikitext-103Transformer+SSA#31Test perplexity: 17.60Validation perplexity: 16.91
image-classification-on-imagenetSwin-T+SSA#638Top 1 Accuracy: 81.89%