Cross-Tokenizer LLM Distillation through a Byte-Level Interface

Benchmark Model Rank Results
arithmetic-reasoning-on-gsm8kGemma2 (2B) with BLD (distilled from OpenMath2-Llama3.1-8B)–Accuracy: 62.55
math-word-problem-solving-on-mathGemma2 (2B) with BLD (distilled from OpenMath2-Llama3.1-8B)–Accuracy: 20.08
question-answering-on-boolqLlama3.2 (3B) with BLCTD (transferred to Qwen2 tokenizer)–Accuracy: 77.34
question-answering-on-piqaBLD–Accuracy: 75.68
question-answering-on-piqaLlama3.2 (3B) with BLCTD (transferred to Qwen2 tokenizer)–Accuracy: 75.68