WebGPU's Real Cost: Debunking the Dispatch Overhead in LLMs

In the race to optimize neural network inference, WebGPU has emerged as a significant player. Yet, its security-focused design may be more of a hindrance than a help. The per-operation validation required by WebGPU compounds across multiple dispatches, especially noticeable in large language models (LLMs), such as Qwen2.5-0.5B and 1.5B.

Understanding the Overhead

Through an extensive study involving four GPU vendors, NVIDIA, AMD, Apple, and Intel, and two native implementations, Dawn and wgpu-native, it becomes clear that the true cost of WebGPU's overhead has been underestimated. Using a sequential-dispatch methodology, we've unveiled that naive benchmarks exaggerate dispatch costs by approximately 20 times. The reality? The WebGPU API overhead stands at 24-36 microseconds on Vulkan and 32-71 microseconds on Metal.

This isn't just a minor technical detail. When Python costs are included, the total per-operation overhead rises to about 95 microseconds. Why does this matter? Because it highlights a critical opportunity for optimization. Kernel fusion on Vulkan significantly improves throughput by 53%, a stark contrast to CUDA fusion, which offers no benefit.

Implications for Inference Efficiency

Our tests across Linux, Windows, and macOS reveal that backend choice is the dominant factor influencing dispatch overhead. But don't discount implementation choice, within a backend like Metal, it can swing by as much as 2.2 times. What does this mean for compute efficiency? At a batch size of 1, within the current dispatch-heavy setup, per-operation overhead is king, overshadowing even kernel quality.

The introduction of torch-webgpu, a unique PyTorch backend, and an FX-to-WebGPU compiler highlights a stark truth. On a reference platform, this setup achieves only 11-12% of CUDA's performance. Even when dtype-matched to float32, the RTX PRO 2000 outperforms WebGPU's throughput by 1.4 times, despite having roughly 6 times less compute than the RTX 5090.

Why Should We Care?

If you're engaged in optimizing LLM inference pipelines, the findings here shouldn't be ignored. The AI-AI Venn diagram is getting thicker, and understanding the real costs associated with dispatch overhead is essential for. With kernel fusion proving beneficial, isn't it time we reconsidered how we structure our pipelines?

Ultimately, the compute layer needs a payment rail, and as we explore these technologies, understanding the true costs and benefits will guide smarter infrastructure decisions. We're building the financial plumbing for machines, and every microsecond counts.