| visual-question-answering-vqa-on-ai2d | Gemini Ultra | #1 | EM: 79.5 |
| long-context-understanding-on-mmneedle | Gemini Pro 1.0 | #4 | 1 Image, 4*4 Stitching, Exact Accuracy: 24.78… |
| temporal-casual-qa-on-next-qa | Gemini Ultra (zero-shot) | #4 | WUPS: 29.9 |
| temporal-casual-qa-on-next-qa | Gemini Pro (zero-shot) | #5 | WUPS: 28.0 |
| video-question-answering-on-perception-test-test | Gemini Ultra | #5 | Accuracy: 54.7 |
| multi-task-language-understanding-on-mgsm | Gemini Ultra (8-shot) | #6 | Average (%): 79.0 |
| visual-question-answering-on-mm-vet-v2 | Gemini Pro Vision | #7 | GPT-4 score: 57.2±0.2 |
| visual-question-answering-vqa-on-infographicvqa | Gemini Ultra (pixel only) | #7 | ANLS: 80.3 |
| visual-question-answering-vqa-on-textvqa-val | Gemini Ultra | #12 | Accuracy: 82.3 |
| sentence-completion-on-hellaswag | Gemini Ultra (10-shot) | #22 | Accuracy: 87.8 |
| visual-question-answering-on-mm-vet | Gemini 1.0 Pro Vision (gemini-pro-vision) | #28 | GPT-4 score: 64.3±0.4 |
| visual-question-answering-on-docvqa-test | Gemini Ultra | #37 | ANLS: 0.909 |
| multimodal-reasoning-on-mmmu | Gemini Ultra | #61 | Accuracy: 59.4 |
| chart-question-answering-on-chartqa | Gemini Ultra | #67 | 1:1 Accuracy: 80.8 |
| code-generation-on-humaneval | Gemini Ultra | #75 | Pass@1: 74.4 |
| math-word-problem-solving-on-math | Gemini Ultra (4-shot) | #75 | Accuracy: 53.2 |
| math-word-problem-solving-on-math | Gemini Pro (4-shot) | #133 | Accuracy: 32.6 |
| multimodal-reasoning-on-mathvista | Gemini Ultra | #105 | Accuracy: 53.0 |
| arithmetic-reasoning-on-gsm8k | Gemini Pro (maj1@32) | #109 | Accuracy: 86.5 |