The Quest for Scrutable AI: A Deep Dive into Source Faithfulness

The allure of large language models (LLMs) lies in their capacity to generate compelling narratives. Yet, as persuasive as these explanations might be, they often leave users in the dark about the truthfulness of their claims. The critical issue here's scrutability: can users verify whether these AI-generated insights are actually substantiated by solid evidence?

Challenging the Status Quo

In the area of Explainable AI (XAI), there's an increasing focus on faithfulness and traceability. What does this mean? Simply put, explanations should be rooted in, and traceable to, explicit sources, without this, claims are nothing more than speculative fiction. Our current study zeroes in on retrieval-augmented generation (RAG) within the context of programming education, where textbooks stand as the bastions of authoritative evidence.

We benchmarked six LLMs by testing them on 90 Stack Overflow questions grounded in three programming textbooks, measuring source faithfulness through source adherence metrics. The results are eye-opening. Non-RAG models displayed a median source adherence of a staggering 0%, highlighting a gaping chasm in their reliability. Even baseline RAG systems, supposedly a step up, showed only modest adherence rates, ranging from 22% to 40%, depending on the model.

Innovating with Illocutionary Macro-Planning

Drawing inspiration from Achinstein's illocutionary theory of explanation, the concept of illocutionary macro-planning emerges as a novel approach. This methodology involves chain-of-illocution prompting (CoI), which expands a user's query into a series of implicit explanatory questions designed to drive retrieval.

Across the board, CoI demonstrated statistically significant improvements, boosting source adherence by up to 63%. Yet, let's apply some rigor here: while these gains are notable, the absolute adherence levels remain middling at best. It begs the question: is this enough to satisfy the demands of high-stakes applications where accuracy and traceability are key?

Evaluating User Satisfaction

A comprehensive user study, involving 165 retained participants out of 220 recruited, provided further insights. The study revealed that these adherence gains, the direct result of CoI, didn't compromise user satisfaction, relevance, or perceived correctness of the explanations.

What they're not telling you: while CoI may enhance adherence, it doesn't necessarily translate into infallible accuracy. This distinction is important. The quest for scrutability in AI-generated content is far from over, and while the strides made are commendable, there's no room for complacency. In the end, it's the relentless pursuit of verifiable truth that will separate genuine breakthroughs from the smoke and mirrors of persuasive AI explanations.