Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks

Open paper
Benchmark Model Rank Results
common-sense-reasoning-on-parusRandom weighted–Accuracy: 0.48
common-sense-reasoning-on-parusheuristic majority–Accuracy: 0.478
common-sense-reasoning-on-parusmajority_class–Accuracy: 0.498
common-sense-reasoning-on-rucosRandom weighted–Average F1: 0.25EM: 0.247
common-sense-reasoning-on-rucosheuristic majority–Average F1: 0.26EM: 0.257
common-sense-reasoning-on-rucosmajority_class–Average F1: 0.25EM: 0.247
common-sense-reasoning-on-rwsdRandom weighted–Accuracy: 0.597
common-sense-reasoning-on-rwsdheuristic majority–Accuracy: 0.669
common-sense-reasoning-on-rwsdmajority_class–Accuracy: 0.669
natural-language-inference-on-lidirusRandom weighted–MCC: 0
natural-language-inference-on-lidirusheuristic majority–MCC: 0.147
natural-language-inference-on-lidirusmajority_class–MCC: 0
natural-language-inference-on-rcbRandom weighted–Average F1: 0.319Accuracy: 0.374
natural-language-inference-on-rcbheuristic majority–Average F1: 0.4Accuracy: 0.438
natural-language-inference-on-rcbmajority_class–Average F1: 0.217Accuracy: 0.484
natural-language-inference-on-terraRandom weighted–Accuracy: 0.483
natural-language-inference-on-terraheuristic majority–Accuracy: 0.549
natural-language-inference-on-terramajority_class–Accuracy: 0.513
question-answering-on-danetqaRandom weighted–Accuracy: 0.52
question-answering-on-danetqaheuristic majority–Accuracy: 0.642
question-answering-on-danetqamajority_class–Accuracy: 0.503
reading-comprehension-on-musercRandom weighted–Average F1: 0.45EM: 0.071
reading-comprehension-on-musercheuristic majority–Average F1: 0.671EM: 0.237
reading-comprehension-on-musercmajority_class–Average F1: 0.0EM: 0.0
word-sense-disambiguation-on-russeRandom weighted–Accuracy: 0.528
word-sense-disambiguation-on-russeheuristic majority–Accuracy: 0.595
word-sense-disambiguation-on-russemajority_class–Accuracy: 0.587

Rank counts only results with a code link.