Unlocking Huawei's Ascend NPUs with AscendOptimizer

Huawei's Ascend neural processing units (NPUs) present a unique challenge for developers. Unlike the well-documented CUDA ecosystem, Ascend lacks public reference implementations, creating a knowledge gap. This is where AscendOptimizer steps in, reshaping operator optimization.

Tackling the Knowledge Bottleneck

The AscendC operator optimization faces a two-fold hurdle. First, the absence of public implementations restricts learning opportunities. Second, performance optimization relies on a complex two-part artifact, combining a host-side tiling program with a kernel program for instruction scheduling. AscendOptimizer addresses these challenges head-on.

By employing an episodic agent, AscendOptimizer transforms execution into experience. On the host side, it uses a profiling-in-the-loop evolutionary search, directly harnessing hardware feedback to discover optimal configurations for tiling and data movement. This approach ensures that the optimization evolves with each iteration, improving performance steadily.

Kernel Optimization Redefined

On the kernel side, AscendOptimizer introduces an innovative method by mining transferable optimization motifs. By deliberately de-optimizing optimized kernels, it creates instructive trajectories from "bad" to "good." These trajectories are distilled into a retrievable experience bank, providing a foundation for guided rewriting.

Alternating between host tuning and kernel rewriting in a closed loop, AscendOptimizer not only expands feasibility but also reduces latency. The results speak for themselves. On a benchmark of 127 real AscendC operators, AscendOptimizer achieved a geometric-mean speedup of 1.19x over the open-source baseline, with nearly half of the operators outperforming their references.

Implications for AI Development

Why does this matter? AscendOptimizer's approach not only fills the knowledge void but also sets a new standard for optimization on proprietary hardware. The market map tells the story: when nearly 50% of operators outperform their references, it marks a significant shift in the competitive landscape.

This isn't just about raw performance gains. It's about empowering developers to push the boundaries of what's possible with Huawei's hardware. As AI applications grow increasingly complex, efficient utilization of hardware resources becomes important. AscendOptimizer paves the way for more sophisticated AI models, unlocking new potential.

So, what's the takeaway? The data shows that with the right tools, even the most challenging NPUs can be tamed. AscendOptimizer not only highlights the importance of continuous optimization but also sets a precedent for future advancements in AI hardware utilization. As technology progresses, the question isn't whether such tools will be necessary, but how quickly they can evolve to meet new demands.