Guijia Zhang

Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure

Guijia Zhang, Yuxun Chen, Yuheng Qi, Harry Yang

Multimodal GUI agents read an interface through two redundant channels: the rendered pixels of a screenshot and a serialized structure such as a document object model or accessibility tree. Before acting, an agent forms a belief about the current interface state, but existing ben…Read more