AI's Temperament: Understanding How Models Behave Differently

AI models, despite having similar capabilities, can behave differently. A new framework, the Model Temperament Index (MTI), aims to quantify these behavioral differences in AI agents. The MTI measures temperament, not capability, across four axes: Reactivity, Compliance, Sociality, and Resilience. It's a promising tool for deciphering the personality of AI.

Four Axes of Temperament

The MTI's design is rooted in the Four Shell Model from Model Medicine. Each axis offers insight into a model's disposition. Reactivity gauges an AI's environmental sensitivity. Compliance measures how well instructions translate into behavior. Sociality looks at how relational resources are allocated, while Resilience tests stress resistance. Crucially, MTI focuses on what models do, not what they claim about themselves.

Profiling ten small language models, ranging from 1.7 billion to 9 billion parameters, reveals intriguing patterns. A key finding is that the four axes are largely independent among instruction-tuned models, with correlation coefficients (|r|) staying below 0.42. This suggests that AI behavior is multifaceted and not easily predicted by one single dimension.

Key Discoveries and Implications

Notably, within-axis facet dissociations emerged. Compliance splits into two independent facets: formal and stance. Resilience, meanwhile, divides into cognitive and adversarial facets, which are inversely related. This nuanced understanding of AI temperaments could pave the way for more specialized and adaptable AI systems. The ablation study reveals a fascinating Compliance-Resilience paradox, where opinion-yielding and fact-vulnerability work through separate channels.

Another standout observation is how Reinforcement Learning from Human Feedback (RLHF) reshapes AI temperament. It does more than alter axis scores. it creates within-axis facet differentiation absent in unaligned base models. This is a breakthrough in understanding how AI systems can be improved through alignment with human feedback.

Dispelling the Size Myth

The study confirms that temperament is independent of model size, spanning from 1.7 billion to 9 billion parameters. This challenges the common belief that bigger models inherently exhibit more nuanced behavior. Instead, MTI highlights that disposition isn't tied to size. This raises the question: Should we focus more on temperamental profiling than on scaling up models?

The paper's key contribution: it shifts the focus from capability to disposition. By offering a structured way to measure AI temperament, MTI could redefine how we design and deploy AI systems. It's an essential step toward creating more reliable and predictable AI behavior.