TTD: Text-Tag Self-Distillation Enhancing Image-Text Alignment in CLIP to Alleviate Single Tag Bias

Benchmark Model Rank Results
multi-label-text-classification-on-cc3m-tagmaskTTD (w/ fine-tuning)#1Precision: 88.3Recall: 78.0F1: 82.8Accuracy: 88.6mAP: 93.7
multi-label-text-classification-on-cc3m-tagmaskTTD (w/o fine-tuning)#2Precision: 82.9Recall: 74.5F1: 78.5Accuracy: 91.0mAP: 90.3
semantic-segmentation-on-cc3m-tagmaskTTD (TCL)#1mIoU: 65.5
semantic-segmentation-on-cc3m-tagmaskTTD (MaskCLIP)#3mIoU: 50.2
open-vocabulary-semantic-segmentation-on-cityscapesTTD (TCL)#3mIoU: 32.0
open-vocabulary-semantic-segmentation-on-cityscapesTTD (MaskCLIP)#5mIoU: 27.0
open-vocabulary-semantic-segmentation-on-coco-stuff-171TTD (TCL)#4mIoU: 23.7
open-vocabulary-semantic-segmentation-on-coco-stuff-171TTD (MaskCLIP)#7mIoU: 19.4
unsupervised-semantic-segmentation-with-language-image-pre-training-on-cityscapes-valTTD (MaskCLIP)#5mIoU: 32.0
unsupervised-semantic-segmentation-with-language-image-pre-training-on-cityscapes-valTTD (TCL)#7mIoU: 27.0
unsupervised-semantic-segmentation-with-language-image-pre-training-on-coco-objectTTD (TCL)#5mIoU: 37.4
unsupervised-semantic-segmentation-with-language-image-pre-training-on-coco-objectTTD (MaskCLIP)#11mIoU: 26.5
unsupervised-semantic-segmentation-with-language-image-pre-training-on-pascal-context-59TTD (TCL)#6mIoU: 37.4
unsupervised-semantic-segmentation-with-language-image-pre-training-on-pascal-context-59TTD (MaskCLIP)#9mIoU: 31.0
unsupervised-semantic-segmentation-with-language-image-pre-training-on-pascal-vocTTD (TCL)#7mIoU: 61.1
unsupervised-semantic-segmentation-with-language-image-pre-training-on-pascal-vocTTD (MaskCLIP)#10mIoU: 43.1
unsupervised-semantic-segmentation-with-language-image-pre-training-on-coco-stuff-171TTD (TCL)#8mIoU: 23.7
unsupervised-semantic-segmentation-with-language-image-pre-training-on-coco-stuff-171TTD (MaskCLIP)#11mIoU: 19.4
unsupervised-semantic-segmentation-with-language-image-pre-training-on-ade20kTTD (TCL)#9Mean IoU (val): 17.0
unsupervised-semantic-segmentation-with-language-image-pre-training-on-ade20kTTD (MaskCLIP)#11Mean IoU (val): 12.7
open-vocabulary-semantic-segmentation-on-ade20k-150TTD (TCL)#22mIoU: 17.0
open-vocabulary-semantic-segmentation-on-ade20k-150TTD (MaskCLIP)#24mIoU: 12.7
open-vocabulary-semantic-segmentation-on-pascal-context-59TTD (TCL)#22mIoU: 37.4
open-vocabulary-semantic-segmentation-on-pascal-context-59TTD (MaskCLIP)#25mIoU: 31.0