RuleForge: Automating Cybersecurity in the Face of a CVE Deluge

Security teams are drowning in vulnerabilities. In 2025, the National Vulnerability Database unveiled an overwhelming 48,000 new vulnerabilities. It's a tsunami of data that's impossible to manage manually. Enter RuleForge, a system developed by AWS to tackle this avalanche with automation.

Inside RuleForge

RuleForge is designed to automatically generate detection rules from structured Nuclei templates. These JSON-based patterns are meant to identify malicious HTTP requests that exploit specific vulnerabilities. The process starts with YAML-based vulnerability descriptions that provide the input for rule generation.

The real star of the show is RuleForge's novel LLM-as-a-judge system. This Large Language Model acts as a judge to validate the rules by weighing sensitivity against specificity. In simpler terms, how well does the rule avoid missing real threats without crying wolf too often? The results are promising: an AUROC score of 0.75 and a 67% reduction in false positives compared to the usual synthetic tests.

The Methodology Behind the Magic

RuleForge employs a 5x5 generation strategy, which means it generates five parallel candidates with up to five refinement attempts each. This is paired with continuous feedback loops that aim to consistently improve rule quality. However, color me skeptical, but can such an automated system really replace the nuanced judgment of a seasoned cybersecurity expert?

Extending its capabilities, RuleForge can also generate rules from unstructured data sources and manage multi-event-type detection. It sounds like a dream come true for any security team inundated by data. But what they're not telling you: RuleForge still relies heavily on human-in-the-loop validation to ensure that these rules are up to scratch.

A Word of Caution

The developers acknowledge lessons learned in the process, such as the need to mitigate overconfidence in AI-driven systems and the vital role of domain expertise in crafting effective prompts and reviewing generated rules. To be fair, these are valuable insights, but I've seen this pattern before. Automation is touted as a cure-all, yet it often introduces its own set of challenges.

So, is RuleForge the silver bullet the cybersecurity world needs? how well it performs in the high-stakes game of cyber defense. One thing is certain: the conversation about AI's role in cybersecurity is far from over.