Self-Improving Pretraining: using post-trained models to pretrain better models

Benchmark Model Rank Results
question-answering-on-boolqSelf-Improving Pretraining (1.4B)–Accuracy: 70.3
question-answering-on-piqaSelf-Improving Pretraining (1.4B)–Accuracy: 75.8
sentence-completion-on-hellaswagSelf-Improving Pretraining (1.4B)–Accuracy: 69.4