Ducho meets Elliot: Large-scale Benchmarks for Multimodal Recommendation

Benchmark Model Rank Results
multimodal-recommendation-on-amazon-babyFREEDOM (CLIP)#1Recall: 8.95nDCG: 4.36Hit Ratio: 14.45
multimodal-recommendation-on-amazon-babyFREEDOM (ResNet50 + Sentence Bert)#2Recall: 8.81nDCG: 4.31Hit Ratio: 14.28
multimodal-recommendation-on-amazon-babyLATTICE (ResNet50 + Sentence Bert)#3Recall: 8.41nDCG: 4.06Hit Ratio: 13.69
multimodal-recommendation-on-amazon-babyLATTICE (MMFashion + Sentence Bert)#4Recall: 8.38nDCG: 4.13Hit Ratio: 13.63
multimodal-recommendation-on-amazon-babyBM3 (AltCLIP)#5Recall: 8.15nDCG: 4.10Hit Ratio: 13.53
multimodal-recommendation-on-amazon-babyBM3 (ResNet50 + Sentence Bert)#6Recall: 8.05nDCG: 3.91Hit Ratio: 13.29
multimodal-recommendation-on-amazon-babyNGCF-M (Align)#7Recall: 7.70nDCG: 3.66Hit Ratio: 12.61
multimodal-recommendation-on-amazon-babyLightGCN#8Recall: 7.56nDCG: 3.82Hit Ratio: 12.60
multimodal-recommendation-on-amazon-babyNGCF-M (ResNet50 + Sentence Bert)#9Recall: 7.18nDCG: 3.50Hit Ratio: 11.91
multimodal-recommendation-on-amazon-babyVBPR (MMFashion + Sentence Bert)#10Recall: 6.42nDCG: 3.12Hit Ratio: 10.39
multimodal-recommendation-on-amazon-babyVBPR (ResNet50 + Sentence Bert)#11Recall: 6.21nDCG: 2.99Hit Ratio: 10.18
multimodal-recommendation-on-amazon-babyDGCF#12Recall: 6.08nDCG: 3.03Hit Ratio: 10.26
multimodal-recommendation-on-amazon-babySGL#13Recall: 5.77nDCG: 2.93Hit Ratio: 9.40
multimodal-recommendation-on-amazon-babyBPRMF#14Recall: 5.48nDCG: 2.67Hit Ratio: 9.04
multimodal-recommendation-on-amazon-babyGRCN (ResNet50 + Sentence Bert)#15Recall: 5.29nDCG: 2.48Hit Ratio: 8.81
multimodal-recommendation-on-amazon-babyGRCN (Align)#16Recall: 5.21nDCG: 2.43Hit Ratio: 8.76
multimodal-recommendation-on-amazon-babyNGCF#17Recall: 5.09nDCG: 2.39Hit Ratio: 8.59
multimodal-recommendation-on-amazon-babyItemKNN#18Recall: 2.46nDCG: 1.19Hit Ratio: 4.21
multimodal-recommendation-on-amazon-beautyLATTICE (ALIGN)#1Recall: 13.93nDCG: 7.21Hit Ratio: 21.31
multimodal-recommendation-on-amazon-beautyFREEDOM (MMFashion + Sentecen Bert)#2Recall: 13.87nDCG: 7.17Hit Ratio: 21.18
multimodal-recommendation-on-amazon-beautyFREEDOM (ResNet50 + Sentence Bert)#3Recall: 13.85nDCG: 7.24Hit Ratio: 21.11
multimodal-recommendation-on-amazon-beautyLATTICE (ResNet50 + Sentence Bert)#4Recall: 13.44nDCG: 7.03Hit Ratio: 20.65
multimodal-recommendation-on-amazon-beautyLightGCN#5Recall: 12.30nDCG: 6.42Hit Ratio: 19.03
multimodal-recommendation-on-amazon-beautyVBPR (AltCLIP)#6Recall: 11.94nDCG: 6.15Hit Ratio: 18.19
multimodal-recommendation-on-amazon-beautyNGCF-M (MMFashion + Sentecen Bert)#7Recall: 11.93nDCG: 6.21Hit Ratio: 18.22
multimodal-recommendation-on-amazon-beautySGL#8Recall: 11.82nDCG: 6.50Hit Ratio: 18.17
multimodal-recommendation-on-amazon-beautyNGCF-M (ResNet50 + Sentence Bert)#9Recall: 11.72nDCG: 6.11Hit Ratio: 18.12
multimodal-recommendation-on-amazon-beautyBM3 (ALIGN)#10Recall: 11.67nDCG: 6.04Hit Ratio: 18.04
multimodal-recommendation-on-amazon-beautyVBPR (ResNet50 + Sentence Bert)#11Recall: 11.54nDCG: 6.08Hit Ratio: 17.64
multimodal-recommendation-on-amazon-beautyBM3 (ResNet50 + Sentence Bert)#12Recall: 11.28nDCG: 5.83Hit Ratio: 17.65
multimodal-recommendation-on-amazon-beautyBPRMF#13Recall: 10.72nDCG: 5.36Hit Ratio: 16.55
multimodal-recommendation-on-amazon-beautyNGCF#14Recall: 10.42nDCG: 5.27Hit Ratio: 16.21
multimodal-recommendation-on-amazon-beautyDGCF#15Recall: 10.42nDCG: 5.27Hit Ratio: 16.21
multimodal-recommendation-on-amazon-beautyGRCN (ALIGN)#16Recall: 10.26nDCG: 5.15Hit Ratio: 16.09
multimodal-recommendation-on-amazon-beautyGRCN (ResNet50 + Sentence Bert)#17Recall: 9.57nDCG: 4.83Hit Ratio: 14.89
multimodal-recommendation-on-amazon-beautyItemKNN#18Recall: 6.97nDCG: 3.85Hit Ratio: 10.89
multimodal-recommendation-on-amazon-digital-musicLATTICE (ResNet50 + Sentence Bert)#1Recall: 29.40nDCG: 16.07Hit Ratio: 43.60
multimodal-recommendation-on-amazon-digital-musicFREEDOM (ResNet50 + Sentence Bert)#2Recall: 29.05nDCG: 16.15Hit Ratio: 43.46
multimodal-recommendation-on-amazon-digital-musicLightGCN#3Recall: 28.66nDCG: 14.95Hit Ratio: 43.19
multimodal-recommendation-on-amazon-digital-musicVBPR (ResNet50 + Sentence Bert)#4Recall: 28.37nDCG: 15.22Hit Ratio: 43.54
multimodal-recommendation-on-amazon-digital-musicNGCF-M (ResNet50 + Sentence Bert)#5Recall: 27.84nDCG: 15.35Hit Ratio: 41.91
multimodal-recommendation-on-amazon-digital-musicBRMF#6Recall: 27.32nDCG: 14.94Hit Ratio: 41.13
multimodal-recommendation-on-amazon-digital-musicSGL#7Recall: 27.09nDCG: 15.03Hit Ratio: 40.81
multimodal-recommendation-on-amazon-digital-musicBM3 (ResNet50 + Sentence Bert)#8Recall: 27.07nDCG: 14.34Hit Ratio: 41.42
multimodal-recommendation-on-amazon-digital-musicDGCF#9Recall: 26.47nDCG: 14.46Hit Ratio: 40.46
multimodal-recommendation-on-amazon-digital-musicNGCF#10Recall: 26.46nDCG: 14.58Hit Ratio: 40.14
multimodal-recommendation-on-amazon-digital-musicGRCN (ResNet50 + Sentence Bert)#11Recall: 22.88nDCG: 12.17Hit Ratio: 36.25
multimodal-recommendation-on-amazon-digital-musicItemKNN#12Recall: 21.74nDCG: 12.00Hit Ratio: 34.51
multimodal-recommendation-on-amazon-office-productsLATTICE (ResNet50+ Sentence Bert)#1Recall: 15.75nDCG: 7.71Hit Ratio: 25.79
multimodal-recommendation-on-amazon-office-productsLATTICE (ALIGN)#2Recall: 15.71nDCG: 7.63Hit Ratio: 25.65
multimodal-recommendation-on-amazon-office-productsFREEDOM (CLIP)#3Recall: 15.64nDCG: 7.66Hit Ratio: 25.88
multimodal-recommendation-on-amazon-office-productsFREEDOM (ResNet50+ Sentence Bert)#4Recall: 15.58nDCG: 7.57Hit Ratio: 23.59
multimodal-recommendation-on-amazon-office-productsNGCF-M (CLIP)#5Recall: 14.99nDCG: 7.43Hit Ratio: 24.85
multimodal-recommendation-on-amazon-office-productsNGCF-M (ResNet50+ Sentence Bert)#6Recall: 14.35nDCG: 7.14Hit Ratio: 24.04
multimodal-recommendation-on-amazon-office-productsLightGCN#7Recall: 13.99nDCG: 6.93Hit Ratio: 23.95
multimodal-recommendation-on-amazon-office-productsBM3 (ALIGN)#8Recall: 13.84nDCG: 6.75Hit Ratio: 23.40
multimodal-recommendation-on-amazon-office-productsBM3 (ResNet50+ Sentence Bert)#9Recall: 13.13nDCG: 6.42Hit Ratio: 22.5
multimodal-recommendation-on-amazon-office-productsGRCN (CLIP)#10Recall: 13.10nDCG: 6.47Hit Ratio: 22.32
multimodal-recommendation-on-amazon-office-productsVBPR (ResNet50+ Sentence Bert)#11Recall: 12.83nDCG: 6.18Hit Ratio: 22.01
multimodal-recommendation-on-amazon-office-productsVBPR (CLIP)#12Recall: 12.78nDCG: 6.23Hit Ratio: 22.10
multimodal-recommendation-on-amazon-office-productsGRCN (ResNet50+ Sentence Bert)#13Recall: 12.31nDCG: 6.08Hit Ratio: 21.20
multimodal-recommendation-on-amazon-office-productsDGCF#14Recall: 12.19nDCG: 5.89Hit Ratio: 20.89
multimodal-recommendation-on-amazon-office-productsSGL#15Recall: 11.85nDCG: 5.89Hit Ratio: 20.49
multimodal-recommendation-on-amazon-office-productsItemKNN#16Recall: 11.35nDCG: 5.76Hit Ratio: 20.33
multimodal-recommendation-on-amazon-office-productsBPRMF#17Recall: 11.28nDCG: 5.35Hit Ratio: 19.70
multimodal-recommendation-on-amazon-office-productsNGCF#18Recall: 11.05nDCG: 5.45Hit Ratio: 19.62
multimodal-recommendation-on-amazon-toys-gamesFREEDOM (MMFashion + Sentence Bert)#1Recall: 13.73nDCG: 7.10Hit Ratio: 20.70
multimodal-recommendation-on-amazon-toys-gamesFREEDOM (ResNet50 + Sentence Bert)#2Recall: 13.67nDCG: 7.04Hit Ratio: 20.64
multimodal-recommendation-on-amazon-toys-gamesLATTICE (ALIGN)#3Recall: 12.73nDCG: 6.64Hit Ratio: 19.27
multimodal-recommendation-on-amazon-toys-gamesLATTICE (ResNet50 + Sentence Bert)#4Recall: 12.42nDCG: 6.45Hit Ratio: 18.95
multimodal-recommendation-on-amazon-toys-gamesNGCF-M (ALIGN)#5Recall: 11.12nDCG: 5.80Hit Ratio: 17.16
multimodal-recommendation-on-amazon-toys-gamesVBPR (ALIGN)#6Recall: 11.06nDCG: 5.85Hit Ratio: 16.86
multimodal-recommendation-on-amazon-toys-gamesNGCF-M (ResNet50 + Sentence Bert)#7Recall: 10.85nDCG: 5.73Hit Ratio: 16.73
multimodal-recommendation-on-amazon-toys-gamesVBPR (ResNet50 + Sentence Bert)#8Recall: 10.83nDCG: 5.70Hit Ratio: 16.54
multimodal-recommendation-on-amazon-toys-gamesSGL#9Recall: 10.76nDCG: 5.93Hit Ratio: 16.68
multimodal-recommendation-on-amazon-toys-gamesLightGCN#10Recall: 10.59nDCG: 5.58Hit Ratio: 16.63
multimodal-recommendation-on-amazon-toys-gamesBM3 (ALIGN)#11Recall: 10.07nDCG: 5.24Hit Ratio: 15.78
multimodal-recommendation-on-amazon-toys-gamesBM3 (ResNet50 + Sentence Bert)#12Recall: 9.94nDCG: 5.14Hit Ratio: 15.56
multimodal-recommendation-on-amazon-toys-gamesGRCN (ALIGN)#13Recall: 9.94nDCG: 5.07Hit Ratio: 15.35
multimodal-recommendation-on-amazon-toys-gamesGRCN (ResNet50 + Sentence Bert)#14Recall: 9.67nDCG: 9.675.07Hit Ratio: 15.00
multimodal-recommendation-on-amazon-toys-gamesBPRMF#15Recall: 9.51nDCG: 5.02Hit Ratio: 14.75
multimodal-recommendation-on-amazon-toys-gamesDGCF#16Recall: 9.43nDCG: 5.12Hit Ratio: 14.71
multimodal-recommendation-on-amazon-toys-gamesNGCF#17Recall: 9.24nDCG: 4.87Hit Ratio: 14.44
multimodal-recommendation-on-amazon-toys-gamesItemKNN#18Recall: 6.97nDCG: 3.91Hit Ratio: 11.06