Batching LLMs: Cutting Costs Without Losing Quality?

Large language models (LLMs) have become the go-to for text classification in social sciences. But anyone who's tried to code a significant amount of text knows it can get expensive fast. Imagine coding 100,000 texts across four variables. Typically, you'd be looking at around 400,000 API calls. Painful, right?

Batching to the Rescue?

Here's a novel idea: batch the items and stack variables into a single prompt. Slash those API calls to just 4,000. That's a whopping 80% cut in token costs. Sounds like magic. But there's a catch, does batching degrade the quality of coding? It hasn't been clear until now.

This study tested eight production LLMs from four different providers. The task? Process 3,962 expertly coded tweets across four tasks, tweaking batch sizes from just 1 up to a sky-high 1,000 items, stacking up to 25 coding dimensions per prompt. The results? Six out of eight LLMs maintained accuracy within 2 percentage points of the single-item baseline all the way through batch sizes of 100. Promising, right?

Quality vs. Complexity

The idea of stacking variables, up to 10 dimensions, didn't seem to compromise the results much either. Degradation in performance was more about the task's complexity than the length of the prompt. Within the safe zone, the measurement error from batching and stacking is less than the usual inter-coder disagreement in the ground-truth data.

So, what's the takeaway? If you're sticking to simpler tasks, batching might just be the silver bullet to make LLMs more cost-effective without sacrificing the quality of results. But let's not get ahead of ourselves. Is it the right move for every project? Not so fast. Show me consistent retention of quality across a broader range of complex tasks, and then we'll talk.

Why Should You Care?

In a world where efficiency is king, finding ways to cut costs without losing quality is the holy grail. This study might just have a piece of that puzzle. But, as always, proceed with caution. The promise of cheaper text classification is attractive, but make sure to test the complexity of your tasks first. After all, what's the point of saving money if the results aren't up to scratch?

Ultimately, this method could democratize access to LLMs, making them viable for even small-scale research projects. But until I see consistent application across various complexities, I'll believe it when I see those retention numbers.