Stefan Horoi

Can Model Merging Improve Aggregation in DiLoCo?

Stefan Horoi, Benjamin Thérien, Guy Wolf, Eugene Belilovsky

Model merging techniques, which aggregate independently finetuned models into one to combine their capabilities, have become a topic of significant interest in recent years, with a broad array of methods having been proposed to tackle this problem. Simultaneously, an emerging tre…Read more