Zijie Zhou

Online Linear Programming for Multi-Objective Routing in LLM Serving

Zixi Chen, Yinyu Ye, Zijie Zhou

We study the online routing problem in large language model serving, where requests arrive sequentially and must be dispatched to parallel decode workers under tight batch-size and KV-cache constraints. Unlike widely used routing heuristics that are not tied to explicit service-l…Read more