arxivcs.LGcs.AIcs.CL2026-07-16
Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization
Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li, Deng Cai, Min Chen, et al.
Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CP…