UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection

Benchmark Model Rank Results
video-grounding-on-qvhighlightsUMT#5R@1,IoU=0.7: 41.18R@1,IoU=0.5: 56.23
highlight-detection-on-tvsumUMT#7mAP: 83.1
highlight-detection-on-youtube-highlightsUMT#7mAP: 74.9
highlight-detection-on-qvhighlightsUMT (w. PT)#13mAP: 39.12
highlight-detection-on-qvhighlightsUMT#18mAP: 38.18
moment-retrieval-on-qvhighlightsUMT (w/ audio + PT ASR Cpations)#34mAP: 38.08
moment-retrieval-on-qvhighlightsUMT#36mAP: 36.12
moment-retrieval-on-charades-staUMT (VO)#58R@1 IoU=0.5: 49.35R@1 IoU=0.7: 26.16R@5 IoU=0.5: 89.41…
moment-retrieval-on-charades-staUMT (VA)#61R@1 IoU=0.5: 48.31R@1 IoU=0.7: 29.25R@5 IoU=0.5: 88.79…