arxivcs.LGcs.AIcs.DCcs.PF2026-07-06
Adaptive Inference Batching using Policy Gradients
Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic. We investigate whether reinforcement learning (RL) can…