Scaling Vision Transformers to 22 Billion Parameters

Benchmark Model Rank Results
action-classification-on-kinetics-400ViT-22B#24Acc@1: 88.0
image-classification-on-imagenetViT-L/16 (384res, distilled from ViT-22B)#16Top 1 Accuracy: 89.6%Number of params: 307M
image-classification-on-imagenetViT-B/16#37Top 1 Accuracy: 88.6%Number of params: 86M
object-recognition-on-shape-biasViT-22B-384#4shape bias: 86.4
object-recognition-on-shape-biasViT-22B-560#5shape bias: 83.8
object-recognition-on-shape-biasViT-22B-224#7shape bias: 78.0
zero-shot-transfer-image-classification-on-1LiT-22B#3Accuracy (Private): 85.9
zero-shot-transfer-image-classification-on-3LiT-22B#1Accuracy (Private): 80.9
zero-shot-transfer-image-classification-on-4LiT-22B#3Accuracy: 96.0
zero-shot-transfer-image-classification-on-5LiT-22B#2Accuracy (Private): 90.1
zero-shot-transfer-image-classification-on-6LiT-22B#1Accuracy (Private): 87.6