Marios Kogias

CheckWork: Enabling Trace-Driven Analysis of Checkpointing Overhead in Distributed ML Training

Eldar T. Hasanov, Adel Sefiane, Alireza Farshin, Marios Kogias

TL;DR: CheckWork is presented, a framework that generates checkpoint-aware Chakra execution traces by augmenting training DAGs with checkpoint operations that enable realistic simulation of checkpointing strategies using existing system simulators and analyse interference between checkpoint traffic and training communication in multi-job environments.

Checkpointing is a fundamental mechanism for fault tolerance in large-scale distributed machine learning (ML) training, but it can introduce significant overhead due to interactions with compute and communication phases. Evaluating checkpointing strategies on production clusters…Read more