Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark

Benchmark Model Rank Results
automatic-lyrics-transcription-on-jam-altAudioShake v1#2Word Error Rate (WER): 26.0Case Error Rate: 3.4Punctuation F1: 50.5
automatic-lyrics-transcription-on-jam-altWhisper v3#6Word Error Rate (WER): 35.5Case Error Rate: 4.3Punctuation F1: 41.6
automatic-lyrics-transcription-on-jam-altWhisper v2#8Word Error Rate (WER): 35.7Case Error Rate: 4.5Punctuation F1: 41.7
automatic-lyrics-transcription-on-jam-altWhisper v2 +demucs#10Word Error Rate (WER): 44.0Case Error Rate: 5.3Punctuation F1: 28.0
automatic-lyrics-transcription-on-jam-altWhisper v3 +demucs#13Word Error Rate (WER): 47.9Case Error Rate: 3.8Punctuation F1: 29.0
automatic-lyrics-transcription-on-jam-alt-1AudioShake v1#2Word Error Rate (WER): 22.1Case Error Rate: 3.4Punctuation F-1: 59.0
automatic-lyrics-transcription-on-jam-alt-1LyricWhiz#3Word Error Rate (WER): 24.6Case Error Rate: 3.5Punctuation F-1: 34.0
automatic-lyrics-transcription-on-jam-alt-1Whisper v2 +demucs#5Word Error Rate (WER): 32.3Case Error Rate: 5.3Punctuation F-1: 39.2
automatic-lyrics-transcription-on-jam-alt-1Whisper v3#9Word Error Rate (WER): 37.7Case Error Rate: 4.8Punctuation F-1: 40.9
automatic-lyrics-transcription-on-jam-alt-1Whisper v3 +demucs#12Word Error Rate (WER): 43.0Case Error Rate: 4.1Punctuation F-1: 23.3
automatic-lyrics-transcription-on-jam-alt-1Whisper v2#15Word Error Rate (WER): 43.8Case Error Rate: 3.5Punctuation F-1: 31.3
automatic-lyrics-transcription-on-jam-alt-2AudioShake v1#4Word Error Rate (WER): 22.5Case Error Rate: 4.1Punctuation F-1: 47.8
automatic-lyrics-transcription-on-jam-alt-2Whisper v2#5Word Error Rate (WER): 25.7Case Error Rate: 6.5Punctuation F-1: 50.0
automatic-lyrics-transcription-on-jam-alt-2Whisper v3#7Word Error Rate (WER): 28.6Case Error Rate: 5.0Punctuation F-1: 41.9
automatic-lyrics-transcription-on-jam-alt-2Whisper v2 +demucs#10Word Error Rate (WER): 38.8Case Error Rate: 7.1Punctuation F-1: 17.2
automatic-lyrics-transcription-on-jam-alt-2Whisper v3 +demucs#13Word Error Rate (WER): 61.5Case Error Rate: 3.6Punctuation F-1: 28.7
automatic-lyrics-transcription-on-jam-alt-3AudioShake v1#4Word Error Rate (WER): 24.4Case Error Rate: 4.1Punctuation F-1: 48.5
automatic-lyrics-transcription-on-jam-alt-3Whisper v3#6Word Error Rate (WER): 40.7Case Error Rate: 4.0Punctuation F-1: 41.2
automatic-lyrics-transcription-on-jam-alt-3Whisper v3 +demucs#9Word Error Rate (WER): 43.5Case Error Rate: 4.4Punctuation F-1: 34.0
automatic-lyrics-transcription-on-jam-alt-3Whisper v2#11Word Error Rate (WER): 45.4Case Error Rate: 5.3Punctuation F-1: 38.7
automatic-lyrics-transcription-on-jam-alt-3Whisper v2 +demucs#15Word Error Rate (WER): 65.2Case Error Rate: 5.9Punctuation F-1: 30.2
automatic-lyrics-transcription-on-jam-alt-4Whisper v2#3Word Error Rate (WER): 27.7Case Error Rate: 3.2Punctuation F-1: 45.8
automatic-lyrics-transcription-on-jam-alt-4Whisper v3#5Word Error Rate (WER): 34.7Case Error Rate: 3.3Punctuation F-1: 42.4
automatic-lyrics-transcription-on-jam-alt-4AudioShake v1#8Word Error Rate (WER): 34.9Case Error Rate: 2.0Punctuation F-1: 45.8
automatic-lyrics-transcription-on-jam-alt-4Whisper v2 +demucs#10Word Error Rate (WER): 43.3Case Error Rate: 3.2Punctuation F-1: 34.9
automatic-lyrics-transcription-on-jam-alt-4Whisper v3 +demucs#12Word Error Rate (WER): 44.9Case Error Rate: 3.2Punctuation F-1: 30.9