Fourier Neural Operators: A Resolution Dilemma

Fourier Neural Operators (FNOs) promise a future where AI models can operate across various spatial resolutions, theoretically allowing these models to be trained on a coarse grid and then deployed on a finer one. But, does this assumption hold water in practice?

Testing the Assumptions

When transitioning from a training resolution, let's call it $s$, to a testing resolution $S$ where $S>s$, researchers face a choice. Should they run the FNO directly at this higher resolution? Or should they stick to the original resolution and upsample the results to the finer grid using Fourier zero-padding? Surprisingly, for tasks like Darcy flow, the latter approach often outperforms direct inference at the higher resolution.

This finding isn't just an academic curiosity. It reveals that direct fine-grid inference might not always deliver the anticipated benefits and, in some cases, can even underperform compared to the low-resolution approach. So, what gives?

The Complex Dance of Frequencies

The answer lies in the spectral characteristics of these operators. As layers process data, they tend to concentrate energy in lower frequencies. High-frequency outputs, essential for finer resolutions, are increasingly left to the final stages of processing, such as nonlinear decoders. It's almost like building a puzzle where the most intricate pieces are added only at the end.

This behavior suggests that FNOs can perform admirably with fewer modes but remain sensitive when jumping across resolutions. The reasons are deeply rooted in nonlinear aliasing, a phenomenon that complicates zero-shot resolution equivariance.

Why This Matters

For AI practitioners and engineers, this is a wake-up call. Is the allure of multi-resolution capability just a mirage? Should developers reconsider how they approach resolution scaling in neural operator design?

These findings bring forward a solid baseline for evaluating models across different resolutions. In an era where computational efficiency is constantly weighed against performance fidelity, understanding these nuances could be the key to unlocking new potentials in AI applications.

Ultimately, the real world is coming industry, one asset class at a time, and the infrastructure we build must be ready to meet it.