Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark

Benchmark Model Rank Results
automatic-lyrics-transcription-on-jam-altAudioShake v1#2Word Error Rate (WER): 26.0Case Error Rate: 3.4Punctuation F1: 50.5…
automatic-lyrics-transcription-on-jam-altWhisper v3#6Word Error Rate (WER): 35.5Case Error Rate: 4.3Punctuation F1: 41.6…
automatic-lyrics-transcription-on-jam-altWhisper v2#8Word Error Rate (WER): 35.7Case Error Rate: 4.5Punctuation F1: 41.7…
automatic-lyrics-transcription-on-jam-altWhisper v2 +demucs#10Word Error Rate (WER): 44.0Case Error Rate: 5.3Punctuation F1: 28.0…
automatic-lyrics-transcription-on-jam-altWhisper v3 +demucs#13Word Error Rate (WER): 47.9Case Error Rate: 3.8Punctuation F1: 29.0…
automatic-lyrics-transcription-on-jam-alt-englishAudioShake v1#2Word Error Rate (WER): 22.1Case Error Rate: 3.4Punctuation F-1: 59.0…
automatic-lyrics-transcription-on-jam-alt-englishLyricWhiz#3Word Error Rate (WER): 24.6Case Error Rate: 3.5Punctuation F-1: 34.0…
automatic-lyrics-transcription-on-jam-alt-englishWhisper v2 +demucs#5Word Error Rate (WER): 32.3Case Error Rate: 5.3Punctuation F-1: 39.2…
automatic-lyrics-transcription-on-jam-alt-englishWhisper v3#9Word Error Rate (WER): 37.7Case Error Rate: 4.8Punctuation F-1: 40.9…
automatic-lyrics-transcription-on-jam-alt-englishWhisper v3 +demucs#12Word Error Rate (WER): 43.0Case Error Rate: 4.1Punctuation F-1: 23.3…
automatic-lyrics-transcription-on-jam-alt-englishWhisper v2#15Word Error Rate (WER): 43.8Case Error Rate: 3.5Punctuation F-1: 31.3…
automatic-lyrics-transcription-on-jam-alt-frenchWhisper v2#3Word Error Rate (WER): 27.7Case Error Rate: 3.2Punctuation F-1: 45.8…
automatic-lyrics-transcription-on-jam-alt-frenchWhisper v3#5Word Error Rate (WER): 34.7Case Error Rate: 3.3Punctuation F-1: 42.4…
automatic-lyrics-transcription-on-jam-alt-frenchAudioShake v1#8Word Error Rate (WER): 34.9Case Error Rate: 2.0Punctuation F-1: 45.8…
automatic-lyrics-transcription-on-jam-alt-frenchWhisper v2 +demucs#10Word Error Rate (WER): 43.3Case Error Rate: 3.2Punctuation F-1: 34.9…
automatic-lyrics-transcription-on-jam-alt-frenchWhisper v3 +demucs#12Word Error Rate (WER): 44.9Case Error Rate: 3.2Punctuation F-1: 30.9…
automatic-lyrics-transcription-on-jam-alt-germanAudioShake v1#4Word Error Rate (WER): 24.4Case Error Rate: 4.1Punctuation F-1: 48.5…
automatic-lyrics-transcription-on-jam-alt-germanWhisper v3#6Word Error Rate (WER): 40.7Case Error Rate: 4.0Punctuation F-1: 41.2…
automatic-lyrics-transcription-on-jam-alt-germanWhisper v3 +demucs#9Word Error Rate (WER): 43.5Case Error Rate: 4.4Punctuation F-1: 34.0…
automatic-lyrics-transcription-on-jam-alt-germanWhisper v2#11Word Error Rate (WER): 45.4Case Error Rate: 5.3Punctuation F-1: 38.7…
automatic-lyrics-transcription-on-jam-alt-germanWhisper v2 +demucs#15Word Error Rate (WER): 65.2Case Error Rate: 5.9Punctuation F-1: 30.2…
automatic-lyrics-transcription-on-jam-alt-spanishAudioShake v1#4Word Error Rate (WER): 22.5Case Error Rate: 4.1Punctuation F-1: 47.8…
automatic-lyrics-transcription-on-jam-alt-spanishWhisper v2#5Word Error Rate (WER): 25.7Case Error Rate: 6.5Punctuation F-1: 50.0…
automatic-lyrics-transcription-on-jam-alt-spanishWhisper v3#7Word Error Rate (WER): 28.6Case Error Rate: 5.0Punctuation F-1: 41.9…
automatic-lyrics-transcription-on-jam-alt-spanishWhisper v2 +demucs#10Word Error Rate (WER): 38.8Case Error Rate: 7.1Punctuation F-1: 17.2…
automatic-lyrics-transcription-on-jam-alt-spanishWhisper v3 +demucs#13Word Error Rate (WER): 61.5Case Error Rate: 3.6Punctuation F-1: 28.7…