Bridging the Gap: Aligning Language Model Explanations with Actual Outputs

Large language models (LLMs) have captivated the tech world with their capability to engage in human-like conversations and generate coherent text. However, there's a caveat. While LLMs can provide explanations for their outputs, these rationales often don't align with the underlying features that actually influenced their decisions. This misalignment raises important questions about the trustworthiness of these models.

A New Benchmark: The Post-hoc Self-Consistency Bank

To tackle this challenge, researchers have introduced the Post-hoc Self-Consistency Bank (PSCB), a large-scale benchmark that links model decisions with diverse explanations and attribution vectors. This comprehensive approach spans multiple datasets, methods, and model families. Prior analyses highlighted discrepancies between answers and their explanations. Still, the high computational cost of attribution methods limited large-scale studies. PSCB aims to change that.

Why should we care about this? When LLMs provide explanations that don't match the actual decision-making process, it's as if there's a disconnect between the brain and the mouth. If users can't trust these explanations, the utility of LLMs in critical applications diminishes.

Beyond Cosine Similarity

Traditionally, cosine similarity has been a go-to metric for assessing alignment. However, the research reveals that Spearman rank correlation offers a more reliable signal of alignment between LLM decisions and their explanations. This insight is essential for anyone relying on LLMs for sensitive or high-stakes tasks. A more faithful alignment could enhance the deployment of these models in domains where accuracy and interpretability are important.

Direct Preference Optimization: A big deal?

Building on their findings, researchers applied Direct Preference Optimization (DPO) to optimize attribution-based preference data. Surprisingly, this method improved alignment without sacrificing task accuracy. In contrast, standard supervised fine-tuning on the same data didn't yield comparable gains. This is a significant development. It challenges the notion that traditional methods suffice for optimizing LLM performance.

For businesses and developers, the message is clear: refining how LLMs explain their decisions can open new avenues for applications, especially where transparency is key. This research isn't just about technical curiosity. It's about making LLMs more reliable partners in decision-making processes. If we can't trust the explanations, can we truly trust the output?