DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs
Abstract
Simultaneous speech-to-text translation (SimulST) generates translations while speech is still unfolding, requiring a streaming policy that decides when to read and when to write. State-of-the-art approaches rely on attention-based encoder-decoder models where cross-attention provides explicit alignment signals. In contrast, Speech Large Language Models (SpeechLLMs) are decoder-only architectures relying solely on self-attention. This raises a central question: whether decoder self-attention contains sufficiently stable alignment signals to guide the streaming policy. Moreover, existing approaches typically rely on training-based adaptations or heuristic wait-$k$ policies and have not been validated in long-form settings. To fill these gaps, we propose Decoder-Only Attention (DOA), a training-free policy that enables long-form simultaneous translation with off-the-shelf SpeechLLMs by deriving a proxy alignment from self-attention. Experiments on Phi4-Multimodal and Qwen3-Omni show that DOA provides an effective alignment signal for supporting streaming decisions, enabling low-latency long-form SimulST with quality close to offline decoding without retraining.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a training-free, decoder-only attention policy for long-form simultaneous translation, demonstrating that self-attention can effectively support streaming decisions in SpeechLLMs. This work significantly advances the field by providing a practical solution that enhances the capabilities of existing models without the need for extensive retraining, thus promoting broader accessibility and application in real-time translation tasks.
Methodology
The paper introduces the Decoder-Only Attention (DOA) policy, which is a novel approach that leverages self-attention from decoder-only Speech Large Language Models (SpeechLLMs) to derive a proxy alignment signal for simultaneous speech-to-text translation (SimulST). This method is innovative as it circumvents the need for retraining models and adapts existing architectures to handle long-form translation tasks effectively. The methodology is well-structured, detailing the derivation of the proxy cross-attention matrix and the mechanisms for managing audio and textual histories during streaming inference.
Results & evaluation
The experiments are robust, utilizing well-known benchmarks (Phi4-Multimodal and Qwen3-Omni) and comparing the performance of the DOA policy against established baselines like StreamAtt. The results demonstrate that DOA achieves competitive translation quality and latency, indicating its effectiveness in real-world applications. The evaluation metrics used (COMET and LongYAAL) are appropriate for assessing both quality and latency, providing a comprehensive view of the model's performance.
Reproducibility
The paper includes sufficient details about the experimental setup, including model configurations, hyperparameters, and data sources, which enhances reproducibility. The release of code under the Apache 2.0 License further supports this, allowing other researchers to replicate the findings.
Limitations
The authors acknowledge limitations in their evaluation, particularly the focus on English source speech and the lack of multilingual or cross-domain testing. Additionally, the potential generalization of the proposed method to languages with different scripts or tokenization characteristics remains untested, which could affect its applicability in diverse linguistic contexts.
Broader impact
The proposed DOA policy has significant implications for real-time translation applications, potentially improving user experiences in scenarios requiring immediate translation, such as live events or meetings. By enabling off-the-shelf SpeechLLMs to perform effectively in SimulST, this work could contribute to advancements in multilingual communication technologies. The main contribution of this paper is the introduction of a training-free, decoder-only attention policy for long-form simultaneous translation, demonstrating that self-attention can effectively support streaming decisions in SpeechLLMs. This work significantly advances the field by providing a practical solution that enhances the capabilities of existing models without the need for extensive retraining, thus promoting broader accessibility and application in real-time translation tasks.
Author of this paper? Send a correction.