VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena

Open paper
Benchmark Model Rank Results
image-sentence-alignment-on-valseViLBERT 12-in-1#1average pairwise accuracy: 75.1Average Accuracy: 63.2
image-sentence-alignment-on-valseCLIP#2average pairwise accuracy: 64.0
image-sentence-alignment-on-valseViLBERT#3average pairwise accuracy: 63.7Average Accuracy: 51.3
image-sentence-alignment-on-valseGPT1#4average pairwise accuracy: 60.7
image-sentence-alignment-on-valseGPT2#5average pairwise accuracy: 60.1
image-sentence-alignment-on-valseLXMERT#6average pairwise accuracy: 59.6Average Accuracy: 53.5
image-sentence-alignment-on-valseVisualBERT#7average pairwise accuracy: 46.4Average Accuracy: 48.8
image-sentence-alignment-on-valse-actant-swapGPT2#1pairwise accuracy: 76.9
image-sentence-alignment-on-valse-actant-swapGPT1#2pairwise accuracy: 72.2
image-sentence-alignment-on-valse-actant-swapCLIP#3pairwise accuracy: 68.6
image-sentence-alignment-on-valse-actant-swapViLBERT#4pairwise accuracy: 68.3Accuracy (%): 50.4
image-sentence-alignment-on-valse-actant-swapViLBERT 12-in-1#5pairwise accuracy: 58.9Accuracy (%): 52.2
image-sentence-alignment-on-valse-actant-swapLXMERT#6pairwise accuracy: 45.8Accuracy (%): 48.5
image-sentence-alignment-on-valse-actant-swapVisualBERT#7pairwise accuracy: 44.4Accuracy (%): 49.7
image-sentence-alignment-on-valse-actionCLIP#1pairwise accuracy: 75.6
image-sentence-alignment-on-valse-actionViLBERT#2pairwise accuracy: 70.7Accuracy (%): 52.6
image-sentence-alignment-on-valse-actionGPT2#3pairwise accuracy: 66.8
image-sentence-alignment-on-valse-actionViLBERT 12-in-1#4pairwise accuracy: 65.9Accuracy (%): 57.3
image-sentence-alignment-on-valse-actionGPT1#5pairwise accuracy: 65.4
image-sentence-alignment-on-valse-actionLXMERT#6pairwise accuracy: 54.8Accuracy (%): 51.1
image-sentence-alignment-on-valse-actionVisualBERT#7pairwise accuracy: 49.2Accuracy (%): 48.8
image-sentence-alignment-on-valse-coreferenceViLBERT 12-in-1#1pairwise accuracy: 75.7Accuracy (%): 54.4
image-sentence-alignment-on-valse-coreferenceGPT2#2pairwise accuracy: 54.5
image-sentence-alignment-on-valse-coreferenceCLIP#3pairwise accuracy: 52.1
image-sentence-alignment-on-valse-coreferenceVisualBERT#4pairwise accuracy: 49.5Accuracy (%): 50.0
image-sentence-alignment-on-valse-coreferenceViLBERT#5pairwise accuracy: 47.2Accuracy (%): 50.0
image-sentence-alignment-on-valse-coreferenceLXMERT#6pairwise accuracy: 46.8Accuracy (%): 49.8
image-sentence-alignment-on-valse-coreferenceGPT1#7pairwise accuracy: 45.6
image-sentence-alignment-on-valse-coreference-1ViLBERT 12-in-1#1pairwise accuracy: 69.2Accuracy (%): 54.3
image-sentence-alignment-on-valse-coreference-1GPT2#2pairwise accuracy: 50.0
image-sentence-alignment-on-valse-coreference-1CLIP#3pairwise accuracy: 49.7
image-sentence-alignment-on-valse-coreference-1ViLBERT#4pairwise accuracy: 48.1Accuracy (%): 50.0
image-sentence-alignment-on-valse-coreference-1VisualBERT#5pairwise accuracy: 47.6Accuracy (%): 50.0
image-sentence-alignment-on-valse-coreference-1GPT1#6pairwise accuracy: 45.2
image-sentence-alignment-on-valse-coreference-1LXMERT#7pairwise accuracy: 44.2Accuracy (%): 49.0
image-sentence-alignment-on-valse-countingViLBERT 12-in-1#1pairwise accuracy: 76.7Accuracy (%): 64.9
image-sentence-alignment-on-valse-countingLXMERT#2pairwise accuracy: 62.2Accuracy (%): 52.0
image-sentence-alignment-on-valse-countingCLIP#3pairwise accuracy: 62.1
image-sentence-alignment-on-valse-countingViLBERT#4pairwise accuracy: 58.6Accuracy (%): 50.7
image-sentence-alignment-on-valse-countingGPT2#5pairwise accuracy: 51.6
image-sentence-alignment-on-valse-countingGPT1#6pairwise accuracy: 51.2
image-sentence-alignment-on-valse-countingVisualBERT#7pairwise accuracy: 48.2Accuracy (%): 48.3
image-sentence-alignment-on-valse-counting-1ViLBERT 12-in-1#1pairwise accuracy: 80.2Accuracy (%): 69.2
image-sentence-alignment-on-valse-counting-1LXMERT#2pairwise accuracy: 69.2Accuracy (%): 55.4
image-sentence-alignment-on-valse-counting-1ViLBERT#3pairwise accuracy: 62.9Accuracy (%): 50.6
image-sentence-alignment-on-valse-counting-1CLIP#4pairwise accuracy: 62.5
image-sentence-alignment-on-valse-counting-1GPT2#5pairwise accuracy: 49.8
image-sentence-alignment-on-valse-counting-1GPT1#6pairwise accuracy: 48.7
image-sentence-alignment-on-valse-counting-1VisualBERT#7pairwise accuracy: 48.2Accuracy (%): 47.8
image-sentence-alignment-on-valse-counting-2ViLBERT 12-in-1#1pairwise accuracy: 77.3Accuracy (%): 66.7
image-sentence-alignment-on-valse-counting-2ViLBERT#2pairwise accuracy: 73.7Accuracy (%): 51.8
image-sentence-alignment-on-valse-counting-2GPT1#3pairwise accuracy: 69.5
image-sentence-alignment-on-valse-counting-2CLIP#4pairwise accuracy: 57.5
image-sentence-alignment-on-valse-counting-2VisualBERT#5pairwise accuracy: 50.0Accuracy (%): 50.0
image-sentence-alignment-on-valse-counting-2GPT2#6pairwise accuracy: 45.3
image-sentence-alignment-on-valse-counting-2LXMERT#7pairwise accuracy: 42.6Accuracy (%): 49.9
image-sentence-alignment-on-valse-existenceViLBERT 12-in-1#1pairwise accuracy: 95.6Accuracy (%): 89.0
image-sentence-alignment-on-valse-existenceLXMERT#2pairwise accuracy: 78.6Accuracy (%): 55.8
image-sentence-alignment-on-valse-existenceCLIP#3pairwise accuracy: 66.9
image-sentence-alignment-on-valse-existenceViLBERT#4pairwise accuracy: 66.5Accuracy (%): 2.4
image-sentence-alignment-on-valse-existenceGPT1#5pairwise accuracy: 61.8
image-sentence-alignment-on-valse-existenceGPT2#6pairwise accuracy: 58.0
image-sentence-alignment-on-valse-existenceVisualBERT#7pairwise accuracy: 39.7Accuracy (%): 49.3
image-sentence-alignment-on-valse-foil-itCLIP#1pairwise accuracy: 88.8
image-sentence-alignment-on-valse-foil-itLXMERT#2pairwise accuracy: 87.1Accuracy (%): 70.8
image-sentence-alignment-on-valse-foil-itViLBERT 12-in-1#3pairwise accuracy: 86.9Accuracy (%): 71.5
image-sentence-alignment-on-valse-foil-itViLBERT#4pairwise accuracy: 86.9Accuracy (%): 55.9
image-sentence-alignment-on-valse-foil-itGPT2#5pairwise accuracy: 80.7
image-sentence-alignment-on-valse-foil-itGPT1#6pairwise accuracy: 77.5
image-sentence-alignment-on-valse-foil-itVisualBERT#7pairwise accuracy: 48.5Accuracy (%): 46.6
image-sentence-alignment-on-valse-pluralityViLBERT 12-in-1#1pairwise accuracy: 72.4Accuracy (%): 62.0
image-sentence-alignment-on-valse-pluralityLXMERT#2pairwise accuracy: 64.4Accuracy (%): 55.1
image-sentence-alignment-on-valse-pluralityViLBERT#3pairwise accuracy: 61.2Accuracy (%): 50.3
image-sentence-alignment-on-valse-pluralityCLIP#4pairwise accuracy: 56.2
image-sentence-alignment-on-valse-pluralityGPT1#5pairwise accuracy: 53.1
image-sentence-alignment-on-valse-pluralityGPT2#6pairwise accuracy: 51.9
image-sentence-alignment-on-valse-pluralityVisualBERT#7pairwise accuracy: 45.7Accuracy (%): 46.5
image-sentence-alignment-on-valse-spatialGPT1#1pairwise accuracy: 77.2
image-sentence-alignment-on-valse-spatialGPT2#2pairwise accuracy: 75.0
image-sentence-alignment-on-valse-spatialViLBERT 12-in-1#3pairwise accuracy: 67.7Accuracy (%): 53.4
image-sentence-alignment-on-valse-spatialCLIP#4pairwise accuracy: 64.3
image-sentence-alignment-on-valse-spatialLXMERT#5pairwise accuracy: 60.2Accuracy (%): 50.8
image-sentence-alignment-on-valse-spatialViLBERT#6pairwise accuracy: 57.2Accuracy (%): 49.9
image-sentence-alignment-on-valse-spatialVisualBERT#7pairwise accuracy: 39.7Accuracy (%): 49.3

Rank counts only results with a code link.