CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation
Abstract
Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper proposes CRESiST, an end-to-end interleaved read-write framework for simultaneous speech translation that leverages an omni-modal LLM and an Actor-Critic self-enhancement pipeline to optimize streaming policies without rigid offline alignments. This work represents a strong technical contribution to the field of LLM-integrated audio processing, addressing key bottlenecks in latency and cache efficiency for infinite-length inference. While it does not reach the level of a paradigm-shifting landmark like the Transformer or BERT, it offers a significant architectural improvement over existing prefix-incremental methods, establishing new state-of-the-art Pareto frontiers for quality and latency. The integration of reinforcement learning (Actor-Critic) for policy distillation in a speech translation context is a novel and meaningful advance, likely to be cited by researchers working on streaming speech models and real-time translation systems.
Author of this paper? Send a correction.