Criticality-Aware Gen-AI Inference via Dynamic Step Control in Resource-Efficient Vehicular Computing

Jinmei Shu, Jia Hu, Geyong Min, Zhu Xiao, Hongbo Jiang

TL;DR: A Criticality-aware Resource-efficient Inference (CARIN) framework, where inference steps are dynamically adjusted to balance delay and quality for multi-criticality tasks, is developed.

Integrating Generative AI (Gen-AI) into vehicular computing can significantly enhance road safety and driving experience. The performance of Gen-AI inference in vehicular computing is highly sensitive to the number of inference steps, where even minor adjustments can disrupt the balance between delay and inference quality. This raises a critical question: how can inference steps be optimally determined for diverse tasks, given the inherent delay-quality trade-offs? Existing approaches fail to offer an optimal solution due to the lack of flexible inference services and efficient memory bandwidth allocation strategies. To address these challenges, we develop a Criticality-aware Resource-efficient Inference (CARIN) framework, where inference steps are dynamically adjusted to balance delay and quality for multi-criticality tasks. Leveraging an accurate inference model, CARIN fully exploits in-vehicle resources to accelerate both parameter loading and task computing during inference. The joint optimization problem of step control, memory bandwidth allocation, and compute resource scheduling is formulated as a mixed-integer nonlinear programming (MINLP) problem, and solved by a novel learning-to-optimize (L2O) algorithm efficiently. Experimental results demonstrate that, for high-criticality tasks, the proposed approach achieves 28.4% latency reduction and 36% quality improvement over baseline methods.