Beyond Token Counting: AI's Real Value Lies in Outcomes

Silicon Valley's latest obsession, 'tokenmaxxing,' touts the idea that increased AI token use equates to enhanced productivity. Yet, Charles Holive, the chief AI officer at BNP Paribas CIB, isn't buying it.

While others count tokens like trophies, Holive measures AI's success in concrete terms: dollars and increased productivity. He argues that fixating on token consumption as a metric is nothing but a 'vanity metric.'

Outcomes vs. Vanity Metrics

Holive’s comments resonate amid growing skepticism from tech giants like Amazon and Uber over the real returns on ballooning AI expenditures. Amazon, for instance, dismantled its AI-use leaderboard after realizing employees were merely gaming the system to climb rankings. Uber’s Andrew Macdonald has similarly questioned whether rising AI costs translate into practical benefits.

At BNP Paribas CIB, Holive flips the script. Instead of asking for token usage stats, he queries employees on their newfound capabilities and speed enhancements. Simple questions: What are you doing now that you couldn't before? How much quicker are you doing it?

Chasing Real Gains

In steering BNP Paribas CIB's AI projects, Holive starts with clear expectations of potential revenue or productivity enhancements. Success is tracked through key performance indicators, assessed monthly or quarterly, to ensure alignment with these outcomes.

This doesn't mean he ignores tokens entirely. Monitoring token consumption remains essential for cost control. But focusing solely on token usage feels misguided. After all, slapping a model on a GPU rental isn't a convergence thesis. It's about achieving tangible business results.

Beyond Metrics

Holive isn't alone in his critique. At a recent AI summit, Amit Kapur of Tata Consultancy Services echoed the sentiment that businesses should prioritize outcomes over mere token metrics. Antoine Pichot from La Banque Postale emphasized evaluating AI on efficiency, customer satisfaction, and financial impacts.

While token usage can indicate adoption levels, especially in software engineering, Holive cautions against letting it become the be-all and end-all. 'We didn’t say, ‘use the tool, do your best,’ he notes. Contrast this with organizations that chase AI usage metrics without critical scrutiny.

Ultimately, the conversation around AI needs a shift from token volumes to the real-world results they enable. If the AI can hold a wallet, who writes the risk model?