| common-sense-reasoning-on-record | Base Layers 10B (0-shot) | – | EM: 60.7 |
| common-sense-reasoning-on-record | Gshard 9B | – | EM: 72.4 |
| common-sense-reasoning-on-record | HASH Layers 10B (0-shot) | – | EM: 67.2 |
| common-sense-reasoning-on-record | Switch Transformer 9B | – | EM: 79.9 |
| common-sense-reasoning-on-record | sMLP – deterministic 9.4B (0-shot) | – | EM: 73.4 |
| common-sense-reasoning-on-winogrande | Base Layers 10B (0-shot) | – | Accuracy: 51 |
| common-sense-reasoning-on-winogrande | Gshard 9B (0-shot) | – | Accuracy: 51.1 |
| common-sense-reasoning-on-winogrande | HASH Layers 10B (0-shot) | – | Accuracy: 51.7 |
| common-sense-reasoning-on-winogrande | Switch Transformer 9B (0-shot) | – | Accuracy: 53.4 |
| common-sense-reasoning-on-winogrande | sMLP – deterministic 9.4B (0-shot) | – | Accuracy: 54.3 |
| question-answering-on-copa | Base Layers 10B (0-shot) | – | Accuracy: 63 |
| question-answering-on-copa | Gshard 9B | – | Accuracy: 76 |
| question-answering-on-copa | HASH Layers 10B (0-shot) | – | Accuracy: 64 |
| question-answering-on-copa | Switch Transformer 9B | – | Accuracy: 75 |
| question-answering-on-copa | sMLP – deterministic 9.4B (0-shot) | – | Accuracy: 79 |
| question-answering-on-piqa | Base Layers 10B (0-shot) | – | Accuracy: 63.8 |
| question-answering-on-piqa | Gshard 9B | – | Accuracy: 68.1 |
| question-answering-on-piqa | HASH Layers 10B (0-shot) | – | Accuracy: 63.8 |
| question-answering-on-piqa | sMLP - deterministic 9.4B (0-shot) | – | Accuracy: 73 |
| question-answering-on-storycloze | Base Layers 10B (0-shot) | – | Accuracy: 61.4 |
| question-answering-on-storycloze | Gshard 9B | – | Accuracy: 67.9 |
| question-answering-on-storycloze | HASH Layers 10B (0-shot) | – | Accuracy: 64.7 |
| question-answering-on-storycloze | Switch Transformer 9B | – | Accuracy: 73.3 |
| question-answering-on-storycloze | sMLP – deterministic 9.4B (0-shot) | – | Accuracy: 74.7 |
| sentence-completion-on-hellaswag | Base Layers 10B (0-shot) | – | Accuracy: 30.2 |
| sentence-completion-on-hellaswag | Gshard 9B | – | Accuracy: 38 |
| sentence-completion-on-hellaswag | HASH Layers 10B (0-shot) | – | Accuracy: 33 |
| sentence-completion-on-hellaswag | Switch Transformer 9B | – | Accuracy: 52.5 |
| sentence-completion-on-hellaswag | sMLP – deterministic 9.4B (0-shot) | – | Accuracy: 54.5 |