TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation

Benchmark Model Rank Results
description-guided-molecule-generation-on-tomgClaude-3.5#1wAcc: 35.92
description-guided-molecule-generation-on-tomgGemini-1.5-pro#2wAcc: 34.80
description-guided-molecule-generation-on-tomgGPT-4-turbo#3wAcc: 34.23
description-guided-molecule-generation-on-tomgGPT-4o#4wAcc: 32.29
description-guided-molecule-generation-on-tomgClaude-3#5wAcc: 30.47
description-guided-molecule-generation-on-tomgLlama-3.1-8B (OpenMolIns-large)#6wAcc: 27.22
description-guided-molecule-generation-on-tomgGalactica-125M (OpenMolIns-xlarge)#7wAcc: 25.73
description-guided-molecule-generation-on-tomgLlama3-70B-Instruct (INT4)#8wAcc: 23.93
description-guided-molecule-generation-on-tomgGalactica-125M (OpenMolIns-large)#9wAcc: 23.42
description-guided-molecule-generation-on-tomgGalactica-125M (OpenMolIns-medium)#10wAcc: 19.89
description-guided-molecule-generation-on-tomgGPT-3.5-turbo#11wAcc: 18.58
description-guided-molecule-generation-on-tomgGalactica-125M (OpenMolIns-small)#12wAcc: 15.18
description-guided-molecule-generation-on-tomgLlama3.1-8B-Instruct#13wAcc: 14.09
description-guided-molecule-generation-on-tomgLlama3-8B-Instruct#14wAcc: 13.75
description-guided-molecule-generation-on-tomgchatglm-9B#15wAcc: 13.137
description-guided-molecule-generation-on-tomgGalactica-125M (OpenMolIns-light)#16wAcc: 13.136
description-guided-molecule-generation-on-tomgLlama3.2-1B (OpenMolIns-large)#17wAcc: 8.10
description-guided-molecule-generation-on-tomgyi-1.5-9B#18wAcc: 7.32
description-guided-molecule-generation-on-tomgMistral-7B-Instruct-v0.2#19wAcc: 4.81
description-guided-molecule-generation-on-tomgBioT5-base#20wAcc: 4.21
description-guided-molecule-generation-on-tomgMolT5-large#21wAcc: 2.89
description-guided-molecule-generation-on-tomgLlama-3.1-1B-Instruct#22wAcc: 1.99
description-guided-molecule-generation-on-tomgMolT5-base#23wAcc: 1.300
description-guided-molecule-generation-on-tomgMolT5-small#24wAcc: 1.299
description-guided-molecule-generation-on-tomgQwen2-7B-Instruct#25wAcc: 0.15