FoRA: Redefining Efficiency in Fine-Tuning

Parameter-efficient fine-tuning has seen a surge of interest, particularly with methods like LoRA. However, the original goal, reducing trainable parameters, seems overshadowed by accuracy pursuits. Enter FoRA, a method that revisits this goal with a novel twist.

Rethinking Layer Selection

FoRA deviates from the typical focus on adapter rank by trimming the number of adapted layers instead. How? By employing a single-pass diagonal Fisher score that operates at under 1% of the training cost. This process isn't just efficient. it's transformative. FoRA then trains the LoRA down-projection on a Stiefel manifold, maintaining column orthonormality and effective rank.

Performance Across Architectures

The results are compelling. Across five LLaMA-family backbones, FoRA consistently outperformed both LoRA and DoRA while using only half the parameter budget. It's within 0.7-0.8 accuracy points of AdaLoRA, yet it uses just a quarter of the parameters. That's a significant reduction without a substantial sacrifice in performance.

cross-architecture experiments show FoRA's prowess on twelve backbones from LLaMA, Qwen3, and Gemma families, ranging from 270 million to 32 billion parameters. This isn't a fluke. it's a breakthrough. The integration of Fisher selection and Stiefel constraint isn't just additive, it's super-additive. Fisher selection alone matches rank reduction at the same budget. The Stiefel constraint provides that important edge. Does this spell the end for parameter-heavy fine-tuning?

Why It Matters

In an era where computational resources are a premium, FoRA's ability to deliver competitive performance with fewer parameters shouldn't be underestimated. Reducing the computational burden without losing accuracy opens doors for broader applications, especially for those with limited resources.

Yet, there's a broader question: Will this shift in focus prompt a reevaluation of current best practices in model fine-tuning? If FoRA's results hold across even more architectures and datasets, the industry might need to rethink its obsession with accuracy at the cost of efficiency.