Jeongbae Son

SAD-LoRA: Spectral Alignment for Low-Rank Knowledge Distillation

Omer Tariq, Syed Muhammad Raza, Jeongbae Son

Distilling a fine-tuned teacher into a LoRA-adapted student is a standard recipe for parameter-efficient compression, but output-level KD does not explicitly control which rank-$r$ weight subspace the adapter occupies. We propose \textbf{SAD-LoRA} (\textbf{S}pectral \textbf{A}lig…Read more