CheckWork: Enabling Trace-Driven Analysis of Checkpointing Overhead in Distributed ML Training
Eldar T. Hasanov, Adel Sefiane, Alireza Farshin, Marios Kogias
TL;DR: CheckWork is presented, a framework that generates checkpoint-aware Chakra execution traces by augmenting training DAGs with checkpoint operations that enable realistic simulation of checkpointing strategies using existing system simulators and analyse interference between checkpoint traffic and training communication in multi-job environments.
Checkpointing is a fundamental mechanism for fault tolerance in large-scale distributed machine learning (ML) training, but it can introduce significant overhead due to interactions with compute and communication phases. Evaluating checkpointing strategies on production clusters is costly, difficult to reproduce, and limits the ability to systematically explore alternative designs. We present CheckWork, a framework that generates checkpoint-aware Chakra execution traces by augmenting training DAGs with checkpoint operations. These traces enable realistic simulation of checkpointing strategies using existing system simulators. Our evaluation shows that the generated traces reproduce checkpointing behaviour previously observed on real test-beds and reported in systems such as CheckFreq and Gemini, capturing both computational overheads and network-level communication interference. Using the traces, we also analyse interference between checkpoint traffic and training communication in multi-job environments and propose a mechanism to mitigate this effect.