Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema

Benchmark Model Rank Results
common-sense-reasoning-on-winograndeALBERT-base 11M–Accuracy: 52.8
common-sense-reasoning-on-winograndeALBERT-xxlarge 235M–Accuracy: 58.7
common-sense-reasoning-on-winograndeBERT-base 110M–Accuracy: 53.1
common-sense-reasoning-on-winograndeBERT-large 345M–Accuracy: 55.6
common-sense-reasoning-on-winograndeRandom baseline–Accuracy: 50
common-sense-reasoning-on-winograndeRoBERTa-base 125M–Accuracy: 56.3
common-sense-reasoning-on-winograndeRoBERTa-large 355M–Accuracy: 54.9
coreference-resolution-on-winograd-schema-challengeALBERT-base 11M–Accuracy: 55.4
coreference-resolution-on-winograd-schema-challengeALBERT-xxlarge 235M–Accuracy: 78.8
coreference-resolution-on-winograd-schema-challengeBERT-base 110M–Accuracy: 56.5
coreference-resolution-on-winograd-schema-challengeBERT-large 340M–Accuracy: 61.4
coreference-resolution-on-winograd-schema-challengeRandom chance baseline–Accuracy: 50
coreference-resolution-on-winograd-schema-challengeRoBERTa-base 125M–Accuracy: 63
coreference-resolution-on-winograd-schema-challengeRoBERTa-large 354M–Accuracy: 73.9