Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Benchmark Model Rank Results
cross-modal-retrieval-on-coco-2014BEiT-3#3Text-to-image R@1: 67.2Text-to-image R@5: 92.8
cross-modal-retrieval-on-flickr30kBEiT-3#3Image-to-text R@1: 98.0Image-to-text R@5: 100.0
instance-segmentation-on-cocoBEiT-3#5mask AP: 54.8
object-detection-on-cocoBEiT-3#13box mAP: 63.7
semantic-segmentation-on-ade20kBEiT-3#5Validation mIoU: 62.8Params (M): 1900
semantic-segmentation-on-ade20k-valBEiT-3#1mIoU: 62.8
visual-question-answering-on-vqa-v2-test-devBEiT-3#2Accuracy: 84.19
visual-question-answering-on-vqa-v2-test-stdBEiT-3#1overall: 84.03
visual-reasoning-on-nlvr2-devBEiT-3#1Accuracy: 91.51
visual-reasoning-on-nlvr2-testBEiT-3#1Accuracy: 92.58
zero-shot-cross-modal-retrieval-on-flickr30kBEiT-3#2Image-to-text R@1: 94.9Image-to-text R@5: 99.9