Scaling Vision with Sparse Mixture of Experts

Open paper
Benchmark Model Rank Results
few-shot-image-classification-on-imagenet-1-1ViT-MoE-15B (Every-2)#1Top 1 Accuracy: 68.66
few-shot-image-classification-on-imagenet-1-1V-MoE-H/14 (Every-2)#3Top 1 Accuracy: 63.38
few-shot-image-classification-on-imagenet-1-1V-MoE-H/14 (Last-5)#4Top 1 Accuracy: 62.95
few-shot-image-classification-on-imagenet-1-1V-MoE-L/16 (Every-2)#5Top 1 Accuracy: 62.41
few-shot-image-classification-on-imagenet-1-1VIT-H/14#6Top 1 Accuracy: 62.34
few-shot-image-classification-on-imagenet-10ViT-MoE-15B (Every-2)#2Top 1 Accuracy: 84.29
few-shot-image-classification-on-imagenet-10V-MoE-H/14 (Every-2)#5Top 1 Accuracy: 80.33
few-shot-image-classification-on-imagenet-10V-MoE-H/14 (Last-5)#6Top 1 Accuracy: 80.1
few-shot-image-classification-on-imagenet-10VIT-H/14#7Top 1 Accuracy: 79.01
few-shot-image-classification-on-imagenet-5ViT-MoE-15B (Every-2)#1Top 1 Accuracy: 82.78
few-shot-image-classification-on-imagenet-5V-MoE-H/14 (Every-2)#5Top 1 Accuracy: 78.21
few-shot-image-classification-on-imagenet-5V-MoE-H/14 (Last-5)#6Top 1 Accuracy: 78.08
few-shot-image-classification-on-imagenet-5V-MoE-L/16 (Every-2)#7Top 1 Accuracy: 77.1
few-shot-image-classification-on-imagenet-5VIT-H/14#8Top 1 Accuracy: 76.95
image-classification-on-imagenetV-MoE-H/14 (Every-2)#50Top 1 Accuracy: 88.36%Number of params: 7200M
image-classification-on-imagenetVIT-H/14#60Top 1 Accuracy: 88.08%Number of params: 656M
image-classification-on-imagenetV-MoE-L/16 (Every-2)#87Top 1 Accuracy: 87.41%Number of params: 3400M
image-classification-on-jft-300mV-MoE-H/14 (Every-2)#1prec@1: 60.62
image-classification-on-jft-300mV-MoE-H/14 (Last-5)#2prec@1: 60.12
image-classification-on-jft-300mV-MoE-L/16 (Every-2)#3prec@1: 57.65
image-classification-on-jft-300mVIT-H/14#4prec@1: 56.68

Rank counts only results with a code link.