Improving Knowledge Distillation via Regularizing Feature Norm and Direction

Open paper
Benchmark Model Rank Results
knowledge-distillation-on-coco-2017-valReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet50))#1AP@0.5: 61.80AP@0.75: 44.94mAP: 41.03
knowledge-distillation-on-coco-2017-valReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(resnet18))#2AP@0.5: 57.96AP@0.75: 40.15mAP: 37.43
knowledge-distillation-on-coco-2017-valReviewKD++(T: faster rcnn(resnet101), S:faster rcnn(mobilenet-v2))#3AP@0.5: 55.18AP@0.75: 37.21mAP: 34.51
knowledge-distillation-on-imagenetKD++(T: regnety-16GF S:ViT-B)#5Top-1 accuracy %: 83.60model size: 87MCRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T:resnet-152 S:resnet-101)#16Top-1 accuracy %: 79.15model size: 44.5MCRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T:resnet-152 S:resnet-50)#19Top-1 accuracy %: 77.48CRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T:resnet152 S:resnet34)#25Top-1 accuracy %: 75.53CRD training setting: ✘
knowledge-distillation-on-imagenetReviewKD++(T:resnet50, S:mobilenet-v1)#27Top-1 accuracy %: 72.96
knowledge-distillation-on-imagenetKD++(T:resnet-152 S:resnet18)#28Top-1 accuracy %: 72.54CRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T:renset101 S:resnet18)#29Top-1 accuracy %: 72.54CRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T:resnet50 S:resnet18)#30Top-1 accuracy %: 72.53CRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T: ResNet-34 S:ResNet-18)#37Top-1 accuracy %: 72.07CRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T:ViT-B, S:resnet18)#42Top-1 accuracy %: 71.84CRD training setting: ✘
knowledge-distillation-on-imagenetKD++(T: ViT-S, S:resnet18)#49Top-1 accuracy %: 71.46CRD training setting: ✘
knowledge-distillation-on-cifar-100ReviewKD++(T:resnet-32x4, S:shufflenet-v2)#6Top-1 Accuracy (%): 77.93
knowledge-distillation-on-cifar-100ReviewKD++(T:resnet-32x4, S:shufflenet-v1)#7Top-1 Accuracy (%): 77.68
knowledge-distillation-on-cifar-100DKD++(T:resnet-32x4, S:resnet-8x4)#14Top-1 Accuracy (%): 76.28
knowledge-distillation-on-cifar-100ReviewKD++(T:WRN-40-2, S:WRN-40-1)#15Top-1 Accuracy (%): 75.66
knowledge-distillation-on-cifar-100KD++(T:resnet56, S:resnet20)#23Top-1 Accuracy (%): 72.53
knowledge-distillation-on-cifar-100DKD++(T:resnet50, S:mobilenetv2)#26Top-1 Accuracy (%): 70.82

Rank counts only results with a code link.