HowToCaption: Prompting LLMs to Transform Video Annotations at Scale

Benchmark Model Rank Results
video-captioning-on-msr-vtt-1HowToCaption#8CIDEr: 65.3METEOR: 32.2ROUGE-L: 66.3BLEU-4: 49.8
video-captioning-on-msvd-1HowToCaption#5CIDEr: 154.2BLEU-4: 70.4METEOR: 46.4ROUGE-L: 83.2
video-captioning-on-youcook2HowToCaption#8BLEU-4: 8.8CIDEr: 116.4ROUGE-L: 37.3METEOR: 15.9
zero-shot-video-retrieval-on-lsmdcVAST, HowToCaption-finetuned#3text-to-video R@1: 27.7text-to-video R@5: 46.5
zero-shot-video-retrieval-on-lsmdcHowToCaption#8text-to-video R@1: 17.3text-to-video R@5: 31.7
zero-shot-video-retrieval-on-msr-vttVAST, HowToCaption-finetuned#4text-to-video R@1: 50text-to-video R@5: 73.2text-to-video R@10: 81.4
zero-shot-video-retrieval-on-msr-vttHowToCaption#16text-to-video R@1: 37.6text-to-video R@5: 62text-to-video R@10: 73.3
zero-shot-video-retrieval-on-msvdVAST, HowToCaption-finetuned#3text-to-video R@1: 54.8text-to-video R@5: 80.9
zero-shot-video-retrieval-on-msvdHowToCaption#8text-to-video R@1: 44.5text-to-video R@5: 73.3
zero-shot-video-retrieval-on-youcook2VAST, HowToCaption-finetuned#3text-to-video R@1: 19.7text-to-video R@5: 43.6
zero-shot-video-retrieval-on-youcook2HowToCaption#5text-to-video R@1: 13.4text-to-video R@5: 33.1