Sudong Cai

Nonlinearity-Aware LoRA: Structured Gate Adaptation under Low-Rank Constraints

Shuai Yuan, Sudong Cai, Bingzhi Chen, Shuyuan Zheng, Chuan Xiao, Makoto Onizuka, et al.

Low-rank adaptation (LoRA) is commonly viewed as an update-space approximation to full fine-tuning, yet this view is incomplete for self-gated Transformer feed-forward networks. In gated FFNs, a low-rank residual can change not only projected features but also the nonlinear selec…Read more