ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics

Benchmark Model Rank Results
machine-translation-on-acesHWTSC-Teacher-Sim#1Score: 19.97
machine-translation-on-acesMS-COMET-22#2Score: 19.89
machine-translation-on-acesMS-COMET-QE-22#3Score: 19.76
machine-translation-on-acesKG-BERTScore#4Score: 17.28
machine-translation-on-acesmetricx_xl_DA_2019#5Score: 17.17
machine-translation-on-acesCOMET-QE#6Score: 16.8
machine-translation-on-acesCOMET-22#7Score: 16.31
machine-translation-on-acesUniTE-src#8Score: 15.68
machine-translation-on-acesUniTE-ref#9Score: 15.38
machine-translation-on-acesmetricx_xxl_DA_2019#10Score: 15.24
machine-translation-on-acesUniTE#11Score: 14.76
machine-translation-on-acesCross-QE#12Score: 14.07
machine-translation-on-aceschrF#13Score: 13.57
machine-translation-on-acesmetricx_xl_MQM_2020#14Score: 13.08
machine-translation-on-acesCOMET-20#15Score: 12.06
machine-translation-on-acesBLEURT-20#16Score: 11.9
machine-translation-on-acesYiSi-1#17Score: 11.38
machine-translation-on-acesBERTScore#18Score: 10.47
machine-translation-on-acesf200spBLEU#19Score: -0.18
machine-translation-on-acesf101spBLEU#20Score: -0.33
machine-translation-on-acesBLEU#21Score: -3.13