Reverse Engineering with LLMs: Promising Yet Unpolished

Reverse engineering (RE) plays a important role in software security, especially cryptographic programs. These programs handle sensitive data and are always vulnerable to exploitation. Yet, despite its importance, RE remains a labor-intensive task requiring a high level of expertise. Enter large language models (LLMs) as potential game-changers. But, frankly, are they up to the task?

The CREBench Benchmark

A new benchmark called CREBench aims to evaluate LLMs on cryptographic binary RE tasks. Comprising 432 challenges based on 48 standard cryptographic algorithms and three insecure usage scenarios, CREBench offers a rigorous testing ground. The challenges reflect real-world scenarios, styled as Capture-the-Flag (CTF) tasks that require models to decode cryptographic logic and recover correct inputs.

LLMs are put through four sub-tasks in the CREBench framework, from identifying algorithms to recovering the correct flag. It's a thorough test, but how do these models fare? The best-performing model, GPT-5.4, scored 64.03 out of 100, recovering flags in 59% of challenges. Not bad, but not exactly groundbreaking either. Compare this to a human expert baseline that scores 92.19 points, and the gap becomes clear.

Human Expertise vs. Machine Efficiency

Here's what the benchmarks actually show: humans still outperform machines in cryptographic RE tasks. Despite the hype surrounding AI, these numbers suggest that human intuition and expertise aren't easily replicated by LLMs, at least not yet.

Should we be disappointed? Not necessarily. LLMs still show promise, especially as tools to aid human experts. Strip away the marketing and you get a technology that's evolving, not yet replacing. The architecture matters more than the parameter count, and future iterations could quite possibly narrow the gap.

Why Should Readers Care?

This isn't just an academic exercise. As cybersecurity threats grow, the efficiency of reverse engineering becomes critical. If LLMs can automate even a portion of this work, it could free up human experts for the more intricate tasks. But until these models can match human performance, they serve best as complementary tools rather than replacements.

The reality is, in a field where precision is key, humans still hold the edge. So, when will LLMs truly step up? That's the billion-dollar question.