arxivcs.DCcs.LG2026-07-08
GIFT: Geometry-Informed Low-precision Gradient Communication for LLM Pretraining
Jieying Wang, Shuyuan Fan, Mingkai Zheng, Zhao Zhang
Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, such as FP8 and NVFP4, can significantly reduce the communication volume. Existing methods quantize gradients via linear or nonlinea…