HRM: Master of 'Guessing' in Reasoning Tasks?

In the race to create models that can solve complex reasoning tasks, Hierarchical Reasoning Models (HRMs) have emerged as a frontrunner. Yet, despite their impressive performance compared to large language model-based reasoners, a deeper look reveals some unexpected pitfalls. Could HRMs be more about 'guessing' than actual reasoning?

Surprising Weaknesses

The paper, published in Japanese, reveals that HRMs, while strong, falter on surprisingly simple puzzles. For instance, they can stumble on puzzles with only one unknown cell. This unexpected failure stems from a fundamental flaw: the violation of the fixed point property. This means HRMs might not always converge to a correct solution, a critical oversight in their design.

the dynamics within HRMs show a peculiar 'grokking' pattern. Answers don't improve steadily. Instead, there's a sudden leap to correctness at a critical reasoning step. This erratic behavior suggests that HRMs might be making educated guesses rather than employing true deductive reasoning.

Guessing vs. Reasoning

The benchmark results speak for themselves. Another eye-opener is the existence of multiple fixed points. HRMs often latch onto the first fixed point they encounter, whether it's correct or not, and may remain stuck indefinitely. This limitation implies that HRMs operate more like guessers than reasoners.

So, why does this matter? In a world where AI is expected to solve increasingly complex problems, relying on a model that guesses could lead to catastrophic failures in critical applications.

Strategic Enhancements

Recognizing these deficiencies, researchers have devised strategies to scale HRM's guessing capabilities. Data augmentation, input perturbation, and model bootstrapping are employed to enhance the quality and quantity of guesses. This approach transformed HRM's accuracy on Sudoku-Extreme puzzles from a modest 54.5% to an impressive 96.9%. But the question remains: are we truly enhancing reasoning, or are we merely improving guesswork?

These advancements, while significant on the surface, invite a deeper investigation into the nature of reasoning models. What the English-language press missed: the distinction between guessing and reasoning could redefine how we evaluate AI's effectiveness in reasoning tasks.