Cracking the Code of AI Model Sizes: A Sneak Peek

AI developers love their secrets. Among their favorites? The size of their large language models (LLMs). But a new method might just crack open this opaque world, giving us a peek behind the curtain.

The Secret's Out

Here's the deal: most AI companies play their cards close to the chest their model sizes. Yet, understanding these sizes is key. It helps us decode both capabilities and costs. So, how do you find out what they won't tell you? A clever new black-box approach uses nothing but the text outputs these models generate.

The idea is surprisingly simple. The method looks at how well models predict the next word in popular texts. Yep, the kind we all know, like classic literature and religious documents. The better the predictions, the more the model's likely memorized. And memorization capacity points directly to its size.

Tech Meets Practicality

What's truly fascinating here's the use of Principal Component Analysis (PCA). By aggregating accuracy across different texts into a single vector, two nifty tools emerge. One compares models directly. The other estimates an index that maps to parameter counts. It's not just theory, either. Tests on open-weight models prove it's reliable.

Now, why should you care? Simple. These insights reveal the hidden design choices of AI developers. Some companies are pushing the limits, continuously expanding model parameters. Others? Sticking to rigid parameters, suggesting a different growth philosophy. Are these constraints due to strategic decisions or technical limitations? That's up for debate.

Rethinking AI Opacity

This development could challenge the industry's norm of secrecy. As we begin to map out what these models really hold, it forces a conversation about transparency. Is it time for more openness in AI development? After all, if a method like this can pierce the veil, maybe keeping model sizes under wraps isn't the best strategy.

In a world where AI models dictate the future of technology, knowing their size is more than just a number. It's a glimpse into the strategies of the companies that build them. And let's be real. In the hype-filled AI landscape, who wouldn't want a clearer view?