Anisha Dasgupta

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

Yuchen Yang, Yifan Zhao, Anisha Dasgupta, Sasa Misailovic

Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We…Read more