Brittlebench: Revealing the Fragility in AI Model Evaluations

Model evaluations often rely on benchmarks that don't reflect real-world conditions. This oversight can inflate perceived performance, especially language models. A newly introduced framework, Brittlebench, aims to address this gap by evaluating how sensitive models are to minor changes in input prompts.

Understanding Model Brittleness

The key concept here's 'brittleness', a model's sensitivity to variations in its input prompts. Using Brittlebench, researchers applied semantics-preserving perturbations to popular benchmarks. The findings were striking: model performance could degrade by as much as 12% under these conditions. But that's not all. These perturbations didn't affect all models equally. In a staggering 63% of cases, even a single change in the prompt altered the relative ranking of models. This challenges our assumptions about comparative model performance.

Dissecting Performance Variance

The study further decomposed the performance variance in both state-of-the-art open-weight and commercial models. The results showed that semantics-preserving input changes could account for nearly half of a model's performance variance. This raises a key question: Are we truly measuring a model's capability, or are we merely assessing how well it handles specific prompts?

The Need for solid Evaluations

Brittlebench underscores the necessity for more resilient evaluations and models. If a model's ranking can be so easily shifted, what confidence can we've in current SOTA claims? The paper's key contribution is highlighting the urgent need for benchmarks that mirror the dynamic nature of user interactions and the unpredictable variability of human-generated text.

What can be done? A reevaluation of current benchmarks is key. To ensure AI models are reliable and effective, they must be tested under conditions that reflect the messy, unpredictable reality of human inputs. Brittlebench provides a step in this direction, offering a framework that could reshape how we assess AI performance.