Information Theoretic Representation Distillation

Benchmark Model Rank Results
knowledge-distillation-on-cifar-100resnet8x4 (T: resnet32x4 S: resnet8x4)#9Top-1 Accuracy (%): 76.68
knowledge-distillation-on-cifar-100vgg8 (T:vgg13 S:vgg8)#15Top-1 Accuracy (%): 74.93
knowledge-distillation-on-cifar-100resnet110 (T:resnet110 S:resnet20)#21Top-1 Accuracy (%): 71.99
knowledge-distillation-on-imagenetITRD (T: ResNet-34 S:ResNet-18)#41Top-1 accuracy %: 71.68model size: 11.69MCRD training setting: ✓
question-answering-on-squad11BERT - 6 Layers#10EM: 81.5F1: 88.5
question-answering-on-squad11BERT - 3 Layers#17EM: 77.7F1: 85.8