SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis

Benchmark Model Rank Results
knowledge-distillation-on-imagenetSpectralKD (T:Swin-S S:Swin-T)#7Top-1 accuracy %: 82.7model size: 22MCRD training setting: โœ˜
knowledge-distillation-on-imagenetSpectralKD (T:Cait-S24 S:DeiT-S)#11Top-1 accuracy %: 82.2model size: 22MCRD training setting: โœ˜
knowledge-distillation-on-imagenetSpectralKD (T:Cait-S24 S:DeiT-T)#20Top-1 accuracy %: 77.4model size: 6MCRD training setting: โœ˜