Scaling Vision with Sparse Mixture of Experts

Open paper

Benchmark	Model	Rank	Results
few-shot-image-classification-on-imagenet-1-shot	ViT-MoE-15B (Every-2)	#1	Top 1 Accuracy: 68.66
few-shot-image-classification-on-imagenet-1-shot	V-MoE-H/14 (Every-2)	#3	Top 1 Accuracy: 63.38
few-shot-image-classification-on-imagenet-1-shot	V-MoE-H/14 (Last-5)	#4	Top 1 Accuracy: 62.95
few-shot-image-classification-on-imagenet-1-shot	V-MoE-L/16 (Every-2)	#5	Top 1 Accuracy: 62.41
few-shot-image-classification-on-imagenet-1-shot	VIT-H/14	#6	Top 1 Accuracy: 62.34
few-shot-image-classification-on-imagenet-5-shot	ViT-MoE-15B (Every-2)	#1	Top 1 Accuracy: 82.78
few-shot-image-classification-on-imagenet-5-shot	V-MoE-H/14 (Every-2)	#5	Top 1 Accuracy: 78.21
few-shot-image-classification-on-imagenet-5-shot	V-MoE-H/14 (Last-5)	#6	Top 1 Accuracy: 78.08
few-shot-image-classification-on-imagenet-5-shot	V-MoE-L/16 (Every-2)	#7	Top 1 Accuracy: 77.1
few-shot-image-classification-on-imagenet-5-shot	VIT-H/14	#8	Top 1 Accuracy: 76.95
image-classification-on-jft-300m	V-MoE-H/14 (Every-2)	#1	prec@1: 60.62
image-classification-on-jft-300m	V-MoE-H/14 (Last-5)	#2	prec@1: 60.12
image-classification-on-jft-300m	V-MoE-L/16 (Every-2)	#3	prec@1: 57.65
image-classification-on-jft-300m	VIT-H/14	#4	prec@1: 56.68
few-shot-image-classification-on-imagenet-10-shot	ViT-MoE-15B (Every-2)	#2	Top 1 Accuracy: 84.29
few-shot-image-classification-on-imagenet-10-shot	V-MoE-H/14 (Every-2)	#5	Top 1 Accuracy: 80.33
few-shot-image-classification-on-imagenet-10-shot	V-MoE-H/14 (Last-5)	#6	Top 1 Accuracy: 80.1
few-shot-image-classification-on-imagenet-10-shot	VIT-H/14	#7	Top 1 Accuracy: 79.01
image-classification-on-imagenet	V-MoE-H/14 (Every-2)	#56	Top 1 Accuracy: 88.36%Number of params: 7200M
image-classification-on-imagenet	VIT-H/14	#66	Top 1 Accuracy: 88.08%Number of params: 656M
image-classification-on-imagenet	V-MoE-L/16 (Every-2)	#95	Top 1 Accuracy: 87.41%Number of params: 3400M