FP8 Overtakes FP64: A Paradigm Shift in HPC

High-performance computing (HPC) has long revered FP64 as the cornerstone of precision in scientific simulations. But recent advancements in AI-optimized GPUs, particularly the B300 series, are challenging this notion. The paper's key contribution reveals that FP8, combined with the Ozaki Scheme II, achieves FP64 accuracy, marking a significant shift in computational paradigms.

Reevaluating FP64

NVIDIA's Blackwell Ultra (B300) series illustrates a dramatic change. Native FP64 performance regresses to approximately 1.3 TFLOPS, a stark 31-fold decrease compared to its predecessor, the B200. This regression renders traditional memory-bound operations like SpMV and GEMV compute-bound instead, fundamentally altering the HPC landscape.

The introduction of the Tensor-Memory Equilibrium (TME) model brings a new perspective. By factoring in a compute multiplier, bandwidth multiplier, and reconstruction latency, this model reshapes our understanding of computational efficiency. Register-level fusion, a key mechanism, drives the bandwidth multiplier effectively to one, making emulation nearly cost-free behind the memory wall.

FP8's Rising Star

Projected performance metrics are nothing short of revolutionary. Using the Ozaki II method, emulated FP64 achieves ~500 TFLOPS on the B300 and ~400 TFLOPS on Rubin R200. This surpasses the native FP64 ceiling of the B200 by over tenfold in compute-bound conditions, while matching the memory roof in bandwidth-bound scenarios.

A comparison against the H100 baseline further emphasizes this shift. The Ozaki II method consistently matches or outperforms the H100 across various workloads, starkly contrasting the up-to-50x regression seen with B300's native FP64.

Implications for the Future

Does this herald the end of native FP64's reign in HPC? The evidence certainly suggests so. With the Ozaki Scheme II and complementary techniques like Kulisch fixed-point reconstruction, FP8 proves itself as a viable, if not superior, alternative. This builds on prior work from computational theory, solidifying FP8's place in production HPC.

The question now is whether the industry will rapidly adopt this new standard. Code and data for these methods are available, encouraging continued exploration and validation. As we witness this shift, it's important for stakeholders to adapt, ensuring they're not left clinging to outdated dogmas.