Decoder-Only Attention: Redefining Simultaneous Speech Translation

Simultaneous speech-to-text translation is evolving, and Decoder-Only Attention (DOA) is making a splash. The method uses Speech Large Language Models (SpeechLLMs) to translate speech as it happens. Unlike traditional models, DOA relies solely on decoder self-attention, sidestepping the need for explicit alignment signals.

The Challenge of Alignment

Alignment signals have been the linchpin for state-of-the-art translation models, which typically use attention-based encoder-decoder architectures. These models have relied heavily on cross-attention mechanisms to keep things in sync. However, DOA questions whether decoder self-attention can provide a stable enough signal for streaming policies. The answer, according to recent experiments, is a resounding yes. Notably, DOA achieves this without the need for retraining, offering a more efficient path forward.

Breaking New Ground with DOA

What sets DOA apart is its ability to function effectively in long-form translation settings. Traditional approaches have often faltered here, relying on training-based adaptations or heuristic wait-$k$ policies that haven't been validated for extended use. With DOA, the results are clear. Experiments on the Phi4-Multimodal and Qwen3-Omni datasets demonstrate that DOA provides a reliable alignment signal, supporting low-latency and high-quality translations akin to offline decoding.

Why This Matters

The benchmark results speak for themselves. DOA not only holds its own against existing models but does so with off-the-shelf SpeechLLMs. This is a essential development because it reduces the need for specialized training, making advanced speech translation more accessible. Western coverage has largely overlooked this, but the potential impact is significant. Could this be the model that democratizes real-time translation technology? The data shows it might be.

While the rest of the AI community grapples with complex training models, DOA offers a simpler, more efficient path. The question now is whether the industry will embrace this shift or continue down the rabbit hole of ever-complex architectures. As simultaneous translation becomes more integrated into everyday technology, those who adapt will likely lead the charge in making easy communication a reality.