Revolutionizing Diffusion Models: A New Approach to Sampling

Diffusion models have long been bifurcated between score and energy parameterizations. The energy approach is appealing because it allows for sampling procedures like Markov Chain Monte Carlo (MCMC), which can include a Metropolis-Hastings (MH) correction step. This leads to better sampling quality, especially when combining pre-trained models for unique distribution samples.

Score vs. Energy

Score-based diffusion models, however, dominate the field with a lot of pre-trained models available. Yet, they lack an underlying energy function, rendering MH-based sampling techniques inapplicable. This is a significant drawback given the potential improvements in sampling quality that MH corrections can provide.

Innovative MH-like Rule

In a significant advancement, researchers have introduced an MH-like acceptance rule based on the line integration of the score function. This enables the reuse of existing diffusion models while integrating them with MCMC methods in the reverse process. Essentially, it's an innovative blend of score parameterization with annealed MCMC, promising substantial benefits.

Crucially, experiments on both synthetic and real-world datasets demonstrate that these MH-like samplers offer relative improvements akin to those observed with energy-based models. This is achieved without requiring explicit energy parameterization, marking a breakthrough in the area of diffusion models.

Why It Matters

Why should this matter to those outside the niche field? Simply put, the ability to enhance existing models without overhauling their structure means faster, more efficient development cycles. It also means that the extensive investment in pre-trained score-based models retains its value.

The key contribution here's the balance struck between maintaining score parameterization and achieving the benefits usually reserved for energy-based approaches. Could this be the path to more universal applications for diffusion models?

In a field often hindered by its own complexity, this development is a breath of fresh air. By effectively bridging the gap between score and energy-based methods, researchers are setting the stage for more adaptable and high-performing models. The ablation study reveals the practical viability of this approach, and code and data are available at the usual repositories. The potential here's enormous, and it wouldn't be a surprise if we see more adoption of this method across various applications.