Layered Prefill Takes MoE Models to New Heights

Running large language models in production is no small feat. They must meet tight service-level objectives for both time-to-first-token (TTFT) and time-between-token (TBT), all while maximizing throughput within fixed resource limits.

Chunked Prefill: The Old Standard

Chunked prefill has been the go-to technique for stabilizing TBT. It splits long prompts along the token dimension and interleaves prefill with decode iterations. While it works, chunked prefill has its downsides, particularly for Mixture-of-Experts (MoE) models. Redundant expert weight loads can crank up memory traffic by 39% and energy use.

Enter Layered Prefill

Layered prefill flips the script. Instead of focusing on tokens, it uses transformer layer groups as the scheduling unit. This method vertically partitions the model, interleaving prefill and decode across these groups. The results are impressive: up to 70% reduction in TTFT, 41% drop in overall latency, and a 22% decrease in per-token energy consumption.

Layered prefill consistently pushes the TTFT-TBT Pareto frontier further than chunked prefill does. It cuts expert-load traffic and energy costs while maintaining stall-free decoding. Strip away the marketing and you get a clear win for efficiency.

Why Does This Matter?

Here's the kicker: shifting focus from tokens to layers opens up new possibilities for high-efficiency MoE serving in co-located environments. It challenges the old guard of token-centric scheduling, showing that the architecture matters more than the parameter count.

In a world where every watt of energy and byte of memory counts, layered prefill isn't just a nice-to-have. It's essential. So the question is: why aren't more systems adopting this approach? The numbers tell a different story.