AI's Struggle with Real Intelligence: A Deep Dive into ARC-AGI Challenges

In the AI race to mimic human intelligence, the Abstraction and Reasoning Corpus (ARC-AGI) stands as a stark reminder of how far there's still to go. You'd think with all the buzzwords flying around about AI's potential, we'd be closer to reaching human-like reasoning. But the data says otherwise.

Performance Drops: The Numbers Don't Lie

The latest cross-generation analysis of 82 AI approaches revealed a worrying trend. From ARC-AGI-1 to ARC-AGI-3, program synthesis, neuro-symbolic, and neural methods showed a 2-3x performance drop. Initial systems scored as high as 93% on ARC-AGI-1. Fast forward to ARC-AGI-2 and 3, and those scores plummeted to 68.8% and 13%, respectively. Meanwhile, humans continue to ace these tests, hitting nearly perfect scores across the board.

A huge cost reduction has been noted, with costs dropping 390 times in a year. That's from $4,500 per task with older models to just $12 with GPT-5.2. But don't let the price tag fool you. This drop is more about cutting back on test-time parallelism than any major leap in efficiency.

Human vs. Machine: The Great Divide

So, what does this mean for AI's grand plans? While AI can crunch numbers at lightning speed, it's still grappling with what we humans do naturally, reasoning through complex problems. Sure, trillions of parameters in models make for impressive tech specs. But intelligence, size doesn't always matter.

Kaggle-constrained entries, ranging from 660 million to 8 billion parameters, managed to perform competitively without breaking the bank. This aligns with François Chollet's thesis that true intelligence is measured by how efficiently skills are acquired. Yet, even after thousands of synthetic examples, ARC Prize 2025 winners only achieved a 24% success rate on ARC-AGI-2.

Why Should We Care?

It's time to question the real progress we're making. If AI can't handle abstract reasoning, are we truly progressing? Test-time adaptation and refinement loops are essential yet still underdeveloped, while compositional reasoning remains a puzzle.

The gap between AI's potential and its reality is glaring. The press release might say AI revolution, but the data says it’s more like a slow crawl. In a world obsessed with faster and cheaper, the real question we should be asking is whether we're investing in the right kind of AI progress. Is the future of AI just about bigger models, or should we be focusing on smarter, more adaptable systems?