Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs

Benchmark Model Rank Results
language-modelling-on-the-pileTest-Time Fine-Tuning with SIFT + Llama-3.2 (3B)#1Bits per byte: 0.557
language-modelling-on-the-pileTest-Time Fine-Tuning with SIFT + Phi-3 (3.8B)#2Bits per byte: 0.595
language-modelling-on-the-pileTest-Time Fine-Tuning with SIFT + Llama-3.2 (1B)#3Bits per byte: 0.606
language-modelling-on-the-pileGemma-2 27B#4Bits per byte: 0.629
language-modelling-on-the-pileLlama-3.2 3B#6Bits per byte: 0.640
language-modelling-on-the-pilePhi-3 14B#8Bits per byte: 0.651
language-modelling-on-the-pileGemma-2 9B#9Bits per byte: 0.670
language-modelling-on-the-pilePhi-3 7B#10Bits per byte: 0.678
language-modelling-on-the-pilePhi-3 3.8B#11Bits per byte: 0.679
language-modelling-on-the-pileLlama-3.2 1B#12Bits per byte: 0.697
language-modelling-on-the-pileGemma-2 2B#14Bits per byte: 0.721
language-modelling-on-the-pileLlama-3.2-Instruct 3B#15Bits per byte: 0.737
language-modelling-on-the-pileTest-Time Fine-Tuning with SIFT + GPT-2 (774M)#17Bits per byte: 0.762
language-modelling-on-the-pileLlama-3.2-Instruct 1B#19Bits per byte: 0.807
language-modelling-on-the-pileTest-Time Fine-Tuning with SIFT + GPT-2 (124M)#21Bits per byte: 0.862