Bridging the Gap: LLMs, Self-Reports, and Human Behavior

Large language models (LLMs) have become central to AI research, yet their capacity to predict human-like behavior remains questionable. A recent study unveils critical insights into the mismatch between LLM self-reports and observed conduct.

LLMs and Personality Constructs

The paper's key contribution: a novel psychometric instrument designed from the ground up to measure LLM behavior. This tool, employing exploratory factor analysis (EFA), identifies five core dimensions: Responsiveness, Deference, Boldness, Guardedness, and Verbosity. Researchers administered 300 varied items to 25 LLMs across 17 model families, capturing vast behavioral data.

What's the takeaway? The instrument's reliability is impressive, boasting a Tucker's phi of at least.957 and internal consistency with all alpha values above.930. However, the real issue lies in predicting actual behavior.

Predictive Validity Concerns

The study's evaluation phase collected 2,500 open-ended behavioral samples, judged by both human raters and an LLM ensemble. Interestingly, while human and LLM judge ratings showed agreeability (average correlation of.51), neither aligned with LLM self-reports. The self-report correlation with human ratings was a mere -.01, rising slightly with LLM judges to.13.

Crucially, self-reports on Responsiveness correlated better with LLM judges (.53) than with humans (.04), despite a.59 agreement between humans and LLM judges. This suggests a potential confound in self-report measures that human observers miss.

Implications and Challenges

Why should this matter to us? As AI systems increasingly judge their own performance and that of others, reliance on LLM self-reports without human oversight could skew outcomes. The ablation study reveals that alignment-shaped self-descriptions pose risks, especially in LLM-as-judge scenarios.

This builds on prior work from psychometrics, challenging the accuracy of LLM-based evaluations without human input. What's missing is a more refined understanding of how these models interpret and report their behavior.

Why do LLMs struggle with accurate self-assessment? The gap between human and machine interpretations of personality and behavior remains vast. Until we bridge this divide, expect inconsistencies in AI evaluations.

Code and data are available at the study's repository, offering a essential resource for further exploration. The development of such diagnostic tools marks a step towards more reliable AI systems. But can we trust LLMs to judge themselves accurately? The jury's still out.