Lyrics Transcription for Humans: A Readability-Aware Benchmark

Open paper
Benchmark Model Rank Results
automatic-lyrics-transcription-on-jam-altAudioShake v3#1Word Error Rate (WER): 16.1Case-Sensitive Word Error Rate: 20.1…
automatic-lyrics-transcription-on-jam-altWhisper v2 +lang#3Word Error Rate (WER): 27.9Case-Sensitive Word Error Rate: 32.6…
automatic-lyrics-transcription-on-jam-altWhisper v3 +lang#4Word Error Rate (WER): 32.6Case-Sensitive Word Error Rate: 37.2…
automatic-lyrics-transcription-on-jam-altWhisper v2 +demucs +lang#5Word Error Rate (WER): 33.5Case-Sensitive Word Error Rate: 39.3…
automatic-lyrics-transcription-on-jam-altWhisper v3#7Word Error Rate (WER): 35.5Case-Sensitive Word Error Rate: 39.7…
automatic-lyrics-transcription-on-jam-altWhisper v2#9Word Error Rate (WER): 37.8Case-Sensitive Word Error Rate: 42.1…
automatic-lyrics-transcription-on-jam-altWhisper v2 +demucs#11Word Error Rate (WER): 44.5Case-Sensitive Word Error Rate: 49.8…
automatic-lyrics-transcription-on-jam-altWhisper v3 +demucs +lang#12Word Error Rate (WER): 46.6Case-Sensitive Word Error Rate: 50.4…
automatic-lyrics-transcription-on-jam-altWhisper v3 +demucs#14Word Error Rate (WER): 48.0Case-Sensitive Word Error Rate: 51.6…
automatic-lyrics-transcription-on-jam-altOWSM v3.1 +demucs +lang#15Word Error Rate (WER): 66.5Case-Sensitive Word Error Rate: 72.6…
automatic-lyrics-transcription-on-jam-altOWSM v3.1 +lang#16Word Error Rate (WER): 69.3Case-Sensitive Word Error Rate: 75.0…
automatic-lyrics-transcription-on-jam-alt-1AudioShake v3#1Word Error Rate (WER): 17.3Case-Sensitive Word Error Rate: 20.9…
automatic-lyrics-transcription-on-jam-alt-1LyricWhiz#4Word Error Rate (WER): 24.6Case-Sensitive Word Error Rate: 28.0…
automatic-lyrics-transcription-on-jam-alt-1Whisper v2 +demucs#6Word Error Rate (WER): 33.3Case-Sensitive Word Error Rate: 39.1…
automatic-lyrics-transcription-on-jam-alt-1Whisper v2 +demucs +lang#7Word Error Rate (WER): 35.6Case-Sensitive Word Error Rate: 41.3…
automatic-lyrics-transcription-on-jam-alt-1Whisper v3 +lang#8Word Error Rate (WER): 36.4Case-Sensitive Word Error Rate: 41.4…
automatic-lyrics-transcription-on-jam-alt-1Whisper v3#10Word Error Rate (WER): 37.7Case-Sensitive Word Error Rate: 42.5…
automatic-lyrics-transcription-on-jam-alt-1Whisper v2 +lang#11Word Error Rate (WER): 39.7Case-Sensitive Word Error Rate: 43.7…
automatic-lyrics-transcription-on-jam-alt-1Whisper v3 +demucs#13Word Error Rate (WER): 43.0Case-Sensitive Word Error Rate: 47.2…
automatic-lyrics-transcription-on-jam-alt-1Whisper v3 +demucs +lang#14Word Error Rate (WER): 43.0Case-Sensitive Word Error Rate: 47.2…
automatic-lyrics-transcription-on-jam-alt-1Whisper v2#16Word Error Rate (WER): 43.8Case-Sensitive Word Error Rate: 47.5…
automatic-lyrics-transcription-on-jam-alt-1OWSM v3.1 +demucs +lang#17Word Error Rate (WER): 63.4Case-Sensitive Word Error Rate: 69.4…
automatic-lyrics-transcription-on-jam-alt-1OWSM v3.1 +lang#18Word Error Rate (WER): 68.6Case-Sensitive Word Error Rate: 74.0…
automatic-lyrics-transcription-on-jam-alt-2AudioShake v3#1Word Error Rate (WER): 12.6Case-Sensitive Word Error Rate: 17.7…
automatic-lyrics-transcription-on-jam-alt-2Whisper v2 +lang#2Word Error Rate (WER): 21.9Case-Sensitive Word Error Rate: 27.7…
automatic-lyrics-transcription-on-jam-alt-2Whisper v3 +lang#3Word Error Rate (WER): 22.4Case-Sensitive Word Error Rate: 28.0…
automatic-lyrics-transcription-on-jam-alt-2Whisper v2#6Word Error Rate (WER): 25.8Case-Sensitive Word Error Rate: 31.5…
automatic-lyrics-transcription-on-jam-alt-2Whisper v3#8Word Error Rate (WER): 28.6Case-Sensitive Word Error Rate: 33.6…
automatic-lyrics-transcription-on-jam-alt-2Whisper v2 +demucs +lang#9Word Error Rate (WER): 34.9Case-Sensitive Word Error Rate: 42.2…
automatic-lyrics-transcription-on-jam-alt-2Whisper v2 +demucs#11Word Error Rate (WER): 39.6Case-Sensitive Word Error Rate: 46.5…
automatic-lyrics-transcription-on-jam-alt-2Whisper v3 +demucs +lang#12Word Error Rate (WER): 58.6Case-Sensitive Word Error Rate: 62.1…
automatic-lyrics-transcription-on-jam-alt-2Whisper v3 +demucs#14Word Error Rate (WER): 61.5Case-Sensitive Word Error Rate: 64.9…
automatic-lyrics-transcription-on-jam-alt-2OWSM v3.1 +demucs +lang#15Word Error Rate (WER): 70.8Case-Sensitive Word Error Rate: 76.0…
automatic-lyrics-transcription-on-jam-alt-2OWSM v3.1 +lang#16Word Error Rate (WER): 73.3Case-Sensitive Word Error Rate: 78.5…
automatic-lyrics-transcription-on-jam-alt-3AudioShake v3#1Word Error Rate (WER): 12.6Case-Sensitive Word Error Rate: 17.5…
automatic-lyrics-transcription-on-jam-alt-3Whisper v2 +lang#2Word Error Rate (WER): 19.9Case-Sensitive Word Error Rate: 26.0…
automatic-lyrics-transcription-on-jam-alt-3Whisper v2 +demucs +lang#3Word Error Rate (WER): 23.9Case-Sensitive Word Error Rate: 30.4…
automatic-lyrics-transcription-on-jam-alt-3Whisper v3 +lang#5Word Error Rate (WER): 35.9Case-Sensitive Word Error Rate: 40.4…
automatic-lyrics-transcription-on-jam-alt-3Whisper v3#7Word Error Rate (WER): 40.7Case-Sensitive Word Error Rate: 44.6…
automatic-lyrics-transcription-on-jam-alt-3Whisper v3 +demucs +lang#8Word Error Rate (WER): 40.8Case-Sensitive Word Error Rate: 44.9…
automatic-lyrics-transcription-on-jam-alt-3Whisper v3 +demucs#10Word Error Rate (WER): 43.5Case-Sensitive Word Error Rate: 47.4…
automatic-lyrics-transcription-on-jam-alt-3OWSM v3.1 +demucs +lang#12Word Error Rate (WER): 51.8Case-Sensitive Word Error Rate: 62.0…
automatic-lyrics-transcription-on-jam-alt-3Whisper v2#13Word Error Rate (WER): 54.5Case-Sensitive Word Error Rate: 59.3…
automatic-lyrics-transcription-on-jam-alt-3OWSM v3.1 +lang#14Word Error Rate (WER): 63.3Case-Sensitive Word Error Rate: 71.8…
automatic-lyrics-transcription-on-jam-alt-3Whisper v2 +demucs#16Word Error Rate (WER): 65.2Case-Sensitive Word Error Rate: 70.4…
automatic-lyrics-transcription-on-jam-alt-4AudioShake v3#1Word Error Rate (WER): 20.8Case-Sensitive Word Error Rate: 23.5…
automatic-lyrics-transcription-on-jam-alt-4Whisper v2 +lang#2Word Error Rate (WER): 27.1Case-Sensitive Word Error Rate: 30.5…
automatic-lyrics-transcription-on-jam-alt-4Whisper v2#4Word Error Rate (WER): 27.7Case-Sensitive Word Error Rate: 31.1…
automatic-lyrics-transcription-on-jam-alt-4Whisper v3#6Word Error Rate (WER): 34.7Case-Sensitive Word Error Rate: 38.0…
automatic-lyrics-transcription-on-jam-alt-4Whisper v3 +lang#7Word Error Rate (WER): 34.7Case-Sensitive Word Error Rate: 38.0…
automatic-lyrics-transcription-on-jam-alt-4Whisper v2 +demucs +lang#9Word Error Rate (WER): 38.2Case-Sensitive Word Error Rate: 42.1…
automatic-lyrics-transcription-on-jam-alt-4Whisper v2 +demucs#11Word Error Rate (WER): 43.3Case-Sensitive Word Error Rate: 46.9…
automatic-lyrics-transcription-on-jam-alt-4Whisper v3 +demucs#13Word Error Rate (WER): 44.9Case-Sensitive Word Error Rate: 48.2…
automatic-lyrics-transcription-on-jam-alt-4Whisper v3 +demucs +lang#14Word Error Rate (WER): 44.9Case-Sensitive Word Error Rate: 48.3…
automatic-lyrics-transcription-on-jam-alt-4OWSM v3.1 +lang#15Word Error Rate (WER): 71.6Case-Sensitive Word Error Rate: 75.7…
automatic-lyrics-transcription-on-jam-alt-4OWSM v3.1 +demucs +lang#16Word Error Rate (WER): 78.5Case-Sensitive Word Error Rate: 82.1…

Rank counts only results with a code link.