DragOn: The Dataset That's Changing the Way We Think About GUI Automation

JUST IN: DragOn is here to shake things up GUI agents. These vision-based models, the ones that can handle desktops, web browsers, and mobile devices, have been waiting for this kind of boost. While datasets for simple clicks have grown rapidly, drag-based tasks like swipe, highlight, and drag-and-drop have lagged behind. DragOn is stepping in to fill that gap.

What's DragOn All About?

DragOn introduces a benchmark and training dataset specifically for drag grounding, and it's no small feat. We're talking about 286,000 training screenshots, 3.5 million tasks, and a 2000-example held-out evaluation suite spread across four domains: text highlighting, cell selection, element resizing, and slider manipulation. The sheer scale is wild.

The big question: why should we care? Simple. Automation of digital tasks is the future, and DragOn is a massive step in that direction. Imagine your computer doing the grunt work of selecting text or resizing elements while you focus on the big picture. That's the power DragOn promises.

The Models and The Madness

Evaluated models include some of the biggest names in AI: GPT, Claude, Qwen, Kimi, and Holo. And there's a twist, DragOn also fine-tuned a Qwen VLM on this rich dataset. The results? They're hinting that DragOn might just be the key to unlocking the next level of performance for these models in complex computer-use tasks.

Sources confirm: This changes the landscape. It's not just about having more data. It's about the kind of data. Drag-based tasks are complex, and DragOn provides the nuanced examples needed to train models to tackle them effectively. And just like that, the leaderboard shifts.

Why It Matters

Here's the kicker: the digital world is moving at breakneck speed, but the tools we use to manage it aren't keeping up. The current models are good with clicks but struggle with drags. DragOn is the answer to bridging that gap, potentially making digital interactions as smooth as silk.

Is it a done deal? Not yet, but DragOn is offering a glimpse into a future where automation isn't just a buzzword. It's part of the daily workflow, making our digital lives easier and more efficient. The labs are scrambling to keep up. If you're in the field, DragOn isn't just a story. It's where the story starts.