arxivcs.CLcs.AI2026-07-09
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Zongyou Yang, Yinghan Hou, Xiaokun Yang
An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practic…