When LLMs Get Lost in Translation: The JavaScript Deobfuscation Dilemma

In a fascinating study involving the large language model Claude Opus 4.6, researchers took a deep dive into JavaScript deobfuscation. If you've ever trained a model, you know that quirks and unexpected behaviors can surface. This seems to be the case here too. Across 192 inference runs on two code styles, force-directed graph simulation and A* pathfinding, the model consistently struggled with poisoned identifier names. But what does this mean for AI-generated code?

The Persistent Poisoned Names

Think of it this way: you're trying to translate a sentence, but some words are garbled nonsense. That's what happened in these experiments. In every baseline run, poisoned names stuck around like unwelcome guests. Eight out of eight runs in physics and five out of five in pathfinding showed this persistence, proving a troublesome pattern. Notably, even terms with zero semantic fit clung on when the string table lacked a coherent alternative.

Understanding Versus Execution

Interestingly, while the model flubbed variable names, it managed to get the semantics right in comments 15 out of 17 times. It's like a student who can explain the theory but messes up the math on the test. This duality raises a question: is the model truly understanding, or just regurgitating learned patterns?

Here's why this matters for everyone, not just researchers. If AI systems can't reliably clean up their code, the implications for software development are significant. Companies relying on AI for code generation might find themselves mired in debugging sessions, chasing down these persistent errors.

Can Reframing Help?

The analogy I keep coming back to is reframing a problem can sometimes change your perspective. In this study, simply changing the task framing from "deobfuscate this" to "write a fresh implementation" dramatically reduced the persistence of these poisoned names. This shift cut down the propagation from 100% to a mere 0-20% in physics and eliminated it entirely in pathfinding.

But here's the thing, the broader issue of domain-level coherence and recoverability still looms large. The model's performance varied in earlier domain-gradient experiments, but it was tied to the coherence of the replacement table.

Ultimately, this research offers a glimpse into the current limitations of large language models in realistic coding scenarios. While generalization beyond these two archetypes is needed, the findings highlight a key area for improvement in AI code generation.