When Truth Isn't Enough: How Small Transformers Handle Contradictory Data

AI, the allure of larger models often overshadows the nuanced behavior of smaller ones. Notably, recent experiments with language models ranging from 3.5 million to 86 million parameters reveal that these models prefer answers that are structurally compressible over being correct.

Compressibility Over Accuracy

The crux of these findings lies in what researchers term the Compression--Consistency Principle. In controlled experiments, GPT-2 style models were trained on datasets where each mathematical problem came with both correct and incorrect solutions. This setup intentionally models conflicting information. The results are intriguing: when errors in the data are random, models manage to identify the correct solutions with accuracy scaling from 65% to 85% as model size increases.

However, when errors follow a coherent alternative rule system, the accuracy plummets to around 45% to 51%. The data shows that the model can't discern false systems from truth when the errors appear coherent. A fascinating twist arises with the introduction of multiple rule systems. If two competing false rules are introduced, accuracy climbs back to 78%, and with ten competing rules, it reaches 88%.

Real-world Implications

What does this mean for real-world applications? On actual Wikipedia text, this phenomenon is mirrored with models achieving 71% accuracy on incoherent errors versus 46% when errors are coherent. The benchmark results speak for themselves. They challenge the assumption that increasing model size inherently improves truth discernment.

Western coverage has largely overlooked this: the idea that truth bias in models is mainly a byproduct of structural incoherence in errors rather than inherent model skill. With AI systems increasingly used in decision-making processes, understanding these biases becomes key. Are we placing too much trust in the scale of models rather than their design?

Looking Ahead

These findings raise a pointed question: If small models are so easily swayed by the structure of data, how do larger models fare? Can these insights be extrapolated to large-scale pretraining? The paper, published in Japanese, reveals that the answer remains open. It emphasizes that as AI continues to evolve, we must scrutinize what's under the hood rather than get lost in parameter count alone.

The data shows that more isn't always better. Perhaps it's time the AI community rethinks the value of coherence and compressibility in training data. The future may not be about building the biggest model but rather the most thoughtfully designed one.