OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

Open paper
Benchmark Model Rank Results
common-sense-reasoning-on-arc-easyGPT-2 XL + OPUS–Accuracy: 50.62
question-answering-on-piqaGPT-2 XL + OPUS–Accuracy: 72.2
sentence-completion-on-hellaswagGPT-2 XL + OPUS–Accuracy: 50.54

Rank counts only results with a code link.