Wenwu Tang

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training

Dong Wang, Wenwu Tang, Yun Cheng, Olga Saukh

Large language models have driven recent progress in language and multimodal AI, yet pre-training them at scale is prohibitively expensive. Low-rank pre-training, which factorizes each weight matrix into a rank-r product to reduce both parameters and FLOPs, is a promising respons…Read more