Why Behavior Matters More Than Matching: A New Look at LLM Distillation

Distilling large language models (LLMs) isn't just about making student models mimic teacher outputs. It's about making them act indistinguishably. New research highlights an often-overlooked dimension: bounded behavioral indistinguishability.

The Core of Indistinguishability

Behavioral indistinguishability is defined through a set of constraints: distinguishing advantage (ε), oracle queries (q), computation limits (t), and adversary class (𝒜). This framework demands more than simple output similarity.

In practical terms, researchers applied this to pairs like Qwen and Llama. They used a reliable 5,000-prompt probe to test behavioral fidelity. The aim? Determine if students were genuinely imitating their teachers, beyond just output similarity.

LoRA Distillation: More Than Just Similarity

LoRA distillation raised semantic similarity scores for Qwen from 0.788 to 0.862 and for Llama from 0.814 to 0.874. A notable improvement, sure. But here's the catch: adversarial evaluations showed lingering behavioral gaps. Discriminators still found differences, especially in style, robustness, and domain-specific prompts.

Why does this matter? Because in real-world applications, it's not enough to just sound like the teacher model. The student must behave like it too. The architecture matters more than the parameter count.

Rethinking Evaluation Strategies

Despite the improvements in output, distinguishing advantage didn't vanish. The Qwen model's advantage dropped from 0.158 to 0.081 after LoRA distillation. It's a start, but there's room to grow.

The study also questioned current query-budget strategies. Disagreement-guided acquisition didn't consistently outperform random sampling. This suggests that broad coverage and diversity are still fundamental strategies.

So, what's the takeaway? Semantic fidelity alone won't cut it. Black-box LLM distillation calls for a more nuanced, adversarial, and category-sensitive approach. Are we measuring what truly matters in AI development, or just what's easy to quantify?