openalexeScholarship (California Digital Library)
Optimizing Ring AllReduce for Sparse Data
The distributed training of machine learning models via gradient descent is generally conducted by iteratively computing the local gradients of a loss function and aggregating them across all processors. Communicating these gradients during aggregation is often a major cost but s…