HiViG: Elevating GUI Interaction with History and Visual Grounding

Graphical User Interface (GUI) environments present unique challenges for Computer Use Agents (CUAs). Existing models falter with short-sighted decision-making and lack visual grounding, leading to errors. Enter HiViG, a framework that introduces a fresh perspective on addressing these issues.

Reimagining Critiques

HiViG stands out by incorporating a multimodal critic trained on real GUI trajectories. This critic compacts past interactions into a clear record, aiding in the evaluation of future actions with a visually grounded approach. It's a significant upgrade from previous scalar and verbal critics. But why does this matter?

The paper's key contribution is its ability to integrate macro-action history with visually grounded critiques. This combination not only reduces short-sighted planning but also intercepts execution errors. HiViG isn't just a minor improvement. It's a leap forward.

Performance That Speaks

Across web, mobile, and desktop benchmarks, HiViG consistently outshines existing critics. For models like Qwen3-VL-32B and Gemini-3-Flash, HiViG boosts success rates by 5.8% and 9.0%, respectively. This isn't just statistical noise. It's a clear edge in performance.

This builds on prior work from the CUA community, addressing the lack of long-horizon planning and grounding errors. The ablation study reveals that HiViG's macro-action history and visual critiques are indispensable for effective test-time scaling in GUI tasks.

Why It Matters

Why should this matter to anyone beyond the technical community? The answer lies in efficiency. By minimizing execution errors and enhancing decision-making, HiViG could drastically improve how we interact with digital environments.

Think of the implications for automated web or app navigation. Improved CUAs mean smoother interactions and less user frustration. But the question remains: will developers embrace this technology? Its success hinges on adoption and integration into existing systems.

HiViG represents a critical step forward. It's not just about solving a technical issue. It's about paving the way for more intelligent, efficient GUI interactions. Code and data are available at the project's repository, inviting further exploration and enhancement.