The Debate Over Multi-Agent Reasoning: Constructor vs. Auditor

Multi-agent debate systems are often judged solely by the accuracy of their final answers. This approach, however, neglects the richness of intermediate reasoning that such debates are meant to showcase. A recent study shifts focus to the intricacies of reasoning within these systems. By examining token-level log-probability distributions and LLM-as-judge rubric scores, researchers aim to understand how internal confidence signals map to external assessments of reasoning quality.

The Role of Confidence

The study's architects propose an intriguing framework with a two-agent debate architecture. It consists of a Constructor and an Auditor, adjudicated by an LLM-as-judge. This judge scores based on instruction following, justification quality, and evidence grounding. A critical-failure flag also plays a role. Within the rubric-scoring domain, experiments show a four-phase confidence trajectory and reveal a significant role asymmetry. Notably, confidence aligns with judged reasoning quality about twice as effectively for the Constructor compared to the Auditor.

Why does this asymmetry matter? The Constructor displays a more reliable confidence-based detection of critical reasoning failures, with an AUROC of 0.804, starkly contrasting the Auditor's 0.634. This gap suggests that the Constructor's role in reasoning is more predictive of quality outcomes, which could inform future designs of debate systems.

Cross-Domain Implications

The findings extend beyond a single domain. They encourage a broader cross-domain investigation into how confidence signals in multi-agent systems might generalize across rubric-based scoring, mathematical reasoning, and factual question answering. If internal confidence can reliably indicate reasoning quality, it could revolutionize how we design and assess AI debate systems.

One might wonder, does this mean the Constructor is inherently superior? Not quite. It indicates that the roles aren't equal in their predictive power concerning reasoning quality. This insight is key for developers aiming to refine multi-agent systems, especially in applications requiring high reasoning accuracy.

The paper's key contribution: it highlights the potential for confidence signals to predict external judgments of reasoning quality. This could lead to more nuanced AI systems capable of producing not just correct answers but also high-quality reasoning processes.