arxivcs.LGcs.AIcs.CL2026-07-24
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Hao Wang, Kun Yuan, Wenlin Zhong, Minglei Zhang, Han Xiao, Ming Sun, et al.
Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer me…