Ducho meets Elliot: Large-scale Benchmarks for Multimodal Recommendation

Benchmark Model Rank Results
multimodal-recommendation-on-amazon-babyFREEDOM (CLIP)#1Recall: 8.95nDCG: 4.36Hit Ratio: 14.45
multimodal-recommendation-on-amazon-babyFREEDOM (ResNet50 + Sentence Bert)#2Recall: 8.81nDCG: 4.31Hit Ratio: 14.28
multimodal-recommendation-on-amazon-babyLATTICE (ResNet50 + Sentence Bert)#3Recall: 8.41nDCG: 4.06Hit Ratio: 13.69
multimodal-recommendation-on-amazon-babyLATTICE (MMFashion + Sentence Bert)#4Recall: 8.38nDCG: 4.13Hit Ratio: 13.63
multimodal-recommendation-on-amazon-babyBM3 (AltCLIP)#5Recall: 8.15nDCG: 4.10Hit Ratio: 13.53
multimodal-recommendation-on-amazon-babyBM3 (ResNet50 + Sentence Bert)#6Recall: 8.05nDCG: 3.91Hit Ratio: 13.29
multimodal-recommendation-on-amazon-babyNGCF-M (Align)#7Recall: 7.70nDCG: 3.66Hit Ratio: 12.61
multimodal-recommendation-on-amazon-babyLightGCN#8Recall: 7.56nDCG: 3.82Hit Ratio: 12.60
multimodal-recommendation-on-amazon-babyNGCF-M (ResNet50 + Sentence Bert)#9Recall: 7.18nDCG: 3.50Hit Ratio: 11.91
multimodal-recommendation-on-amazon-babyVBPR (MMFashion + Sentence Bert)#10Recall: 6.42nDCG: 3.12Hit Ratio: 10.39
multimodal-recommendation-on-amazon-babyVBPR (ResNet50 + Sentence Bert)#11Recall: 6.21nDCG: 2.99Hit Ratio: 10.18
multimodal-recommendation-on-amazon-babyDGCF#12Recall: 6.08nDCG: 3.03Hit Ratio: 10.26
multimodal-recommendation-on-amazon-babySGL#13Recall: 5.77nDCG: 2.93Hit Ratio: 9.40
multimodal-recommendation-on-amazon-babyBPRMF#14Recall: 5.48nDCG: 2.67Hit Ratio: 9.04
multimodal-recommendation-on-amazon-babyGRCN (ResNet50 + Sentence Bert)#15Recall: 5.29nDCG: 2.48Hit Ratio: 8.81
multimodal-recommendation-on-amazon-babyGRCN (Align)#16Recall: 5.21nDCG: 2.43Hit Ratio: 8.76
multimodal-recommendation-on-amazon-babyNGCF#17Recall: 5.09nDCG: 2.39Hit Ratio: 8.59
multimodal-recommendation-on-amazon-babyItemKNN#18Recall: 2.46nDCG: 1.19Hit Ratio: 4.21
multimodal-recommendation-on-amazon-beautyLATTICE (ALIGN)#1Recall: 13.93nDCG: 7.21Hit Ratio: 21.31
multimodal-recommendation-on-amazon-beautyFREEDOM (MMFashion + Sentecen Bert)#2Recall: 13.87nDCG: 7.17Hit Ratio: 21.18
multimodal-recommendation-on-amazon-beautyFREEDOM (ResNet50 + Sentence Bert)#3Recall: 13.85nDCG: 7.24Hit Ratio: 21.11
multimodal-recommendation-on-amazon-beautyLATTICE (ResNet50 + Sentence Bert)#4Recall: 13.44nDCG: 7.03Hit Ratio: 20.65
multimodal-recommendation-on-amazon-beautyLightGCN#5Recall: 12.30nDCG: 6.42Hit Ratio: 19.03
multimodal-recommendation-on-amazon-beautyVBPR (AltCLIP)#6Recall: 11.94nDCG: 6.15Hit Ratio: 18.19
multimodal-recommendation-on-amazon-beautyNGCF-M (MMFashion + Sentecen Bert)#7Recall: 11.93nDCG: 6.21Hit Ratio: 18.22
multimodal-recommendation-on-amazon-beautySGL#8Recall: 11.82nDCG: 6.50Hit Ratio: 18.17
multimodal-recommendation-on-amazon-beautyNGCF-M (ResNet50 + Sentence Bert)#9Recall: 11.72nDCG: 6.11Hit Ratio: 18.12
multimodal-recommendation-on-amazon-beautyBM3 (ALIGN)#10Recall: 11.67nDCG: 6.04Hit Ratio: 18.04
multimodal-recommendation-on-amazon-beautyVBPR (ResNet50 + Sentence Bert)#11Recall: 11.54nDCG: 6.08Hit Ratio: 17.64
multimodal-recommendation-on-amazon-beautyBM3 (ResNet50 + Sentence Bert)#12Recall: 11.28nDCG: 5.83Hit Ratio: 17.65
multimodal-recommendation-on-amazon-beautyBPRMF#13Recall: 10.72nDCG: 5.36Hit Ratio: 16.55
multimodal-recommendation-on-amazon-beautyNGCF#14Recall: 10.42nDCG: 5.27Hit Ratio: 16.21
multimodal-recommendation-on-amazon-beautyDGCF#15Recall: 10.42nDCG: 5.27Hit Ratio: 16.21
multimodal-recommendation-on-amazon-beautyGRCN (ALIGN)#16Recall: 10.26nDCG: 5.15Hit Ratio: 16.09
multimodal-recommendation-on-amazon-beautyGRCN (ResNet50 + Sentence Bert)#17Recall: 9.57nDCG: 4.83Hit Ratio: 14.89
multimodal-recommendation-on-amazon-beautyItemKNN#18Recall: 6.97nDCG: 3.85Hit Ratio: 10.89
multimodal-recommendation-on-amazon-digitalLATTICE (ResNet50 + Sentence Bert)#1Recall: 29.40nDCG: 16.07Hit Ratio: 43.60
multimodal-recommendation-on-amazon-digitalFREEDOM (ResNet50 + Sentence Bert)#2Recall: 29.05nDCG: 16.15Hit Ratio: 43.46
multimodal-recommendation-on-amazon-digitalLightGCN#3Recall: 28.66nDCG: 14.95Hit Ratio: 43.19
multimodal-recommendation-on-amazon-digitalVBPR (ResNet50 + Sentence Bert)#4Recall: 28.37nDCG: 15.22Hit Ratio: 43.54
multimodal-recommendation-on-amazon-digitalNGCF-M (ResNet50 + Sentence Bert)#5Recall: 27.84nDCG: 15.35Hit Ratio: 41.91
multimodal-recommendation-on-amazon-digitalBRMF#6Recall: 27.32nDCG: 14.94Hit Ratio: 41.13
multimodal-recommendation-on-amazon-digitalSGL#7Recall: 27.09nDCG: 15.03Hit Ratio: 40.81
multimodal-recommendation-on-amazon-digitalBM3 (ResNet50 + Sentence Bert)#8Recall: 27.07nDCG: 14.34Hit Ratio: 41.42
multimodal-recommendation-on-amazon-digitalDGCF#9Recall: 26.47nDCG: 14.46Hit Ratio: 40.46
multimodal-recommendation-on-amazon-digitalNGCF#10Recall: 26.46nDCG: 14.58Hit Ratio: 40.14
multimodal-recommendation-on-amazon-digitalGRCN (ResNet50 + Sentence Bert)#11Recall: 22.88nDCG: 12.17Hit Ratio: 36.25
multimodal-recommendation-on-amazon-digitalItemKNN#12Recall: 21.74nDCG: 12.00Hit Ratio: 34.51
multimodal-recommendation-on-amazon-officeLATTICE (ResNet50+ Sentence Bert)#1Recall: 15.75nDCG: 7.71Hit Ratio: 25.79
multimodal-recommendation-on-amazon-officeLATTICE (ALIGN)#2Recall: 15.71nDCG: 7.63Hit Ratio: 25.65
multimodal-recommendation-on-amazon-officeFREEDOM (CLIP)#3Recall: 15.64nDCG: 7.66Hit Ratio: 25.88
multimodal-recommendation-on-amazon-officeFREEDOM (ResNet50+ Sentence Bert)#4Recall: 15.58nDCG: 7.57Hit Ratio: 23.59
multimodal-recommendation-on-amazon-officeNGCF-M (CLIP)#5Recall: 14.99nDCG: 7.43Hit Ratio: 24.85
multimodal-recommendation-on-amazon-officeNGCF-M (ResNet50+ Sentence Bert)#6Recall: 14.35nDCG: 7.14Hit Ratio: 24.04
multimodal-recommendation-on-amazon-officeLightGCN#7Recall: 13.99nDCG: 6.93Hit Ratio: 23.95
multimodal-recommendation-on-amazon-officeBM3 (ALIGN)#8Recall: 13.84nDCG: 6.75Hit Ratio: 23.40
multimodal-recommendation-on-amazon-officeBM3 (ResNet50+ Sentence Bert)#9Recall: 13.13nDCG: 6.42Hit Ratio: 22.5
multimodal-recommendation-on-amazon-officeGRCN (CLIP)#10Recall: 13.10nDCG: 6.47Hit Ratio: 22.32
multimodal-recommendation-on-amazon-officeVBPR (ResNet50+ Sentence Bert)#11Recall: 12.83nDCG: 6.18Hit Ratio: 22.01
multimodal-recommendation-on-amazon-officeVBPR (CLIP)#12Recall: 12.78nDCG: 6.23Hit Ratio: 22.10
multimodal-recommendation-on-amazon-officeGRCN (ResNet50+ Sentence Bert)#13Recall: 12.31nDCG: 6.08Hit Ratio: 21.20
multimodal-recommendation-on-amazon-officeDGCF#14Recall: 12.19nDCG: 5.89Hit Ratio: 20.89
multimodal-recommendation-on-amazon-officeSGL#15Recall: 11.85nDCG: 5.89Hit Ratio: 20.49
multimodal-recommendation-on-amazon-officeItemKNN#16Recall: 11.35nDCG: 5.76Hit Ratio: 20.33
multimodal-recommendation-on-amazon-officeBPRMF#17Recall: 11.28nDCG: 5.35Hit Ratio: 19.70
multimodal-recommendation-on-amazon-officeNGCF#18Recall: 11.05nDCG: 5.45Hit Ratio: 19.62
multimodal-recommendation-on-amazon-toysFREEDOM (MMFashion + Sentence Bert)#1Recall: 13.73nDCG: 7.10Hit Ratio: 20.70
multimodal-recommendation-on-amazon-toysFREEDOM (ResNet50 + Sentence Bert)#2Recall: 13.67nDCG: 7.04Hit Ratio: 20.64
multimodal-recommendation-on-amazon-toysLATTICE (ALIGN)#3Recall: 12.73nDCG: 6.64Hit Ratio: 19.27
multimodal-recommendation-on-amazon-toysLATTICE (ResNet50 + Sentence Bert)#4Recall: 12.42nDCG: 6.45Hit Ratio: 18.95
multimodal-recommendation-on-amazon-toysNGCF-M (ALIGN)#5Recall: 11.12nDCG: 5.80Hit Ratio: 17.16
multimodal-recommendation-on-amazon-toysVBPR (ALIGN)#6Recall: 11.06nDCG: 5.85Hit Ratio: 16.86
multimodal-recommendation-on-amazon-toysNGCF-M (ResNet50 + Sentence Bert)#7Recall: 10.85nDCG: 5.73Hit Ratio: 16.73
multimodal-recommendation-on-amazon-toysVBPR (ResNet50 + Sentence Bert)#8Recall: 10.83nDCG: 5.70Hit Ratio: 16.54
multimodal-recommendation-on-amazon-toysSGL#9Recall: 10.76nDCG: 5.93Hit Ratio: 16.68
multimodal-recommendation-on-amazon-toysLightGCN#10Recall: 10.59nDCG: 5.58Hit Ratio: 16.63
multimodal-recommendation-on-amazon-toysBM3 (ALIGN)#11Recall: 10.07nDCG: 5.24Hit Ratio: 15.78
multimodal-recommendation-on-amazon-toysBM3 (ResNet50 + Sentence Bert)#12Recall: 9.94nDCG: 5.14Hit Ratio: 15.56
multimodal-recommendation-on-amazon-toysGRCN (ALIGN)#13Recall: 9.94nDCG: 5.07Hit Ratio: 15.35
multimodal-recommendation-on-amazon-toysGRCN (ResNet50 + Sentence Bert)#14Recall: 9.67nDCG: 9.675.07Hit Ratio: 15.00
multimodal-recommendation-on-amazon-toysBPRMF#15Recall: 9.51nDCG: 5.02Hit Ratio: 14.75
multimodal-recommendation-on-amazon-toysDGCF#16Recall: 9.43nDCG: 5.12Hit Ratio: 14.71
multimodal-recommendation-on-amazon-toysNGCF#17Recall: 9.24nDCG: 4.87Hit Ratio: 14.44
multimodal-recommendation-on-amazon-toysItemKNN#18Recall: 6.97nDCG: 3.91Hit Ratio: 11.06