Federal Agencies' Language Models: A New Challenge for Policy Transparency

Federal agencies are increasingly leaning on large language models (LLMs) to sift through massive volumes of public comments. In theory, it's a move towards efficiency, categorizing opinions swiftly and, supposedly, fairly. But the documents show a different story. The deployment of these models isn’t as straightforward as it seems. Disagreements between models could mean we're overlooking the nuances of public input, potentially skewing what policymakers see.

The Complexity of Categorizations

In a recent study analyzing 1,260 public comments on a USDA docket using four different LLMs, the results were telling. The thematic divergence between models was greater than the variations observed within a single model using different prompts. This suggests a fundamental issue: LLMs interpret the same data differently. This disparity hints at a more profound problem than just a technical glitch. When models can't agree, it reflects the inherent ambiguity in public input that isn’t being addressed.

The system was deployed without the safeguards the agency promised. Typically, a standard evaluation method focuses on stance accuracy using a small, validated set of data. While this may work on paper, it doesn't capture the full picture when models disagree significantly in categorizing the same input. The gap between promised oversight and actual accountability is widening.

Disagreement as a Diagnostic Tool

The study proposes an innovative approach: using disagreement between models as a diagnostic tool. This disagreement-based evaluation shifts the focus from mere accuracy to understanding interpretive complexity. It encourages human reviewers to pay closer attention to genuinely ambiguous inputs, ensuring that subtle yet significant public opinions don't get lost in the shuffle.

But here's the catch, an expert rubric used in this process only suppresses deep interpretive disagreements without actually resolving them. The affected communities weren't consulted when implementing these models. you've to ask, how can a tool accurately reflect public sentiment when the communities it affects aren't involved in its creation?

Human Insight Versus Machine Consistency

In a two-stage labeling study involving a 40-comment subsample, both human annotators and LLMs labeled the comments independently before revising their labels after reviewing others' work. Notably, the human annotator often introduced unique framings absent from the models' collective output. This underlines the importance of human insight in understanding complex issues that algorithms might overlook.

The push for automated systems in policy-making is understandable, but it shouldn't come at the cost of transparency and accountability. When inter-model disagreement becomes a norm rather than an exception, it's time to rethink the systems in place. Accountability requires transparency. Here's what they won't release: a foolproof solution that marries efficiency with nuanced understanding doesn't yet exist. Until it does, the human touch remains irreplaceable.