Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

Benchmark Model Rank Results
video-generation-on-kinetics-600-12-frames-64x64MAGVIT-v2#1FVD: 4.3±0.1
video-prediction-on-kinetics-600-12-frames-64x64MAGVIT-v2#1FVD: 4.3±0.1
video-generation-on-ucf-101MAGVIT-v2#4FVD16: 58±3
video-generation-on-ucf-101MAGVIT-v2 (AR)#8FVD16: 109
image-generation-on-imagenet-512x512MAGVIT-v2#34FID: 1.91Inception score: 324.3
image-generation-on-imagenet-512x512MAGVIT-v2 (w/o guidance)#61FID: 3.07Inception score: 213.1
action-recognition-in-videos-on-something-something-v2MAGVIT-v2 + ViT-B (tokens as targets) (extra data)#59Top-1 Accuracy: 67.38
image-generation-on-imagenet-256x256MAGVIT-v2#70FID: 1.78
image-generation-on-imagenet-256x256MAGVIT-v2 (w/o guidance)#166FID: 3.65