The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles

Benchmark Model Rank Results
graph-regression-on-pcqm4mv2-lscEGT+SSA+Self-ensemble#14Validation MAE: 0.0865
graph-regression-on-pcqm4mv2-lscEGT+SSA#16Validation MAE: 0.0876
image-classification-on-imagenetSwin-T+SSA#581Top 1 Accuracy: 81.89%
language-modelling-on-enwiki8Transformer+SSA#21Bit per Character (BPC): 1.024
language-modelling-on-wikitext-103Transformer+SSA+Self-ensemble#23Test perplexity: 17.18Validation perplexity: 16.54
language-modelling-on-wikitext-103Transformer+SSA#27Test perplexity: 17.60Validation perplexity: 16.91