Understanding the Role of the Projector in Knowledge Distillation

Benchmark Model Rank Results
knowledge-distillation-on-cifar-100SRD (T:resnet-32x4, S:shufflenet-v2)#1Top-1 Accuracy (%): 79.86
knowledge-distillation-on-imagenetSRD (T:RegNety 160 S:DeiT-S)#12Top-1 accuracy %: 82.1model size: 22MCRD training setting: โœ˜
knowledge-distillation-on-imagenetSRD (T:RegNety 160 S:DeIT-Ti)#21Top-1 accuracy %: 77.2model size: 6MCRD training setting: โœ˜
knowledge-distillation-on-imagenetSRD (T: ResNet-34 S:ResNet-18)#41Top-1 accuracy %: 71.87CRD training setting: โœ“