STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity
Abstract
Speech-to-speech translation (S2ST) should preserve not only lexical meaning, but also expressive attributes: emotion, scenario style (e.g., news reporting vs. dramatic dialogue), and nonverbal vocalizations (NVs). Moreover, collecting cross-lingual target speech that is both translation-faithful and expressively aligned with the source is difficult at scale, making reference-based evaluation impractical. We introduce STEB (Speech-to-Speech Translation Expressiveness Benchmark), a 32.6-hour Chinese--English benchmark that evaluates both standard dimensions (translation fidelity, speaker similarity, duration alignment) and expressiveness dimensions (emotion, scenario style, NV preservation). For expressiveness evaluation, STEB uses a caption-then-summarize framework that converts speech into structured expressive attributes and compares source and hypothesis attributes with an LLM judge. Human validation shows statistically significant correlations with listener judgments across all expressive dimensions. We evaluate six S2ST systems covering cascaded systems, end-to-end models, and speech large language models. Many systems, especially cascaded ones, achieve strong translation fidelity, but they still struggle with emotion preservation (best: 3.82/5) and NV preservation (best: 2.31/5). These results reveal a gap between semantic transfer and expressive transfer, identifying expressiveness preservation as an open challenge for S2ST. Audio samples are available at https://cmots.github.io/steb.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant contribution to the field of Speech-to-Speech Translation by introducing STEB, a comprehensive benchmark that evaluates not just translation accuracy but also the preservation of expressive attributes such as emotion, scenario style, and nonverbal vocalizations. The proposed "caption-then-summarize" LLM-based evaluation framework provides a scalable and reference-free solution to a previously intractable problem, validated by strong human correlation. The empirical results reveal a critical gap between semantic transfer and expressive transfer in current S2ST systems, offering valuable insights for future model development and establishing a new standard for evaluating expressive speech technologies.
Methodology
The paper introduces a novel evaluation framework for Speech-to-Speech Translation (S2ST) that moves beyond semantic fidelity to assess expressive attributes (emotion, scenario style, nonverbal vocalizations). The core methodological contribution is the "caption-then-summarize" pipeline, which leverages multimodal LLMs to convert audio into structured textual descriptions of expressiveness, enabling reference-free comparison via an LLM-as-a-judge. This approach addresses the critical lack of parallel expressive S2ST references. The data curation pipeline is rigorous, involving source separation, speaker diarization, and multi-stage quality filtering using both automatic metrics (DNSMOS, BEATs) and human validation. The methodology is sound and addresses a significant gap in the field, although it relies heavily on the capabilities of current multimodal LLMs for annotation and judging.
Results & evaluation
The experimental setup is comprehensive, evaluating six diverse S2ST systems (cascaded, end-to-end, and speech LLMs) on a 32.6-hour Chinese-English benchmark. The results clearly demonstrate the decoupling of translation fidelity and expressiveness preservation, with cascaded systems excelling in BLEU but failing in emotion/NV preservation, while end-to-end models show better expressiveness but lower translation accuracy. The human correlation study provides strong validation for the LLM judge, showing statistically significant agreement with human raters, particularly for emotion and NVs. The analysis of why explicit NV markers help cascaded systems but not end-to-end models offers valuable insights into system design.
Reproducibility
The paper provides detailed descriptions of the data curation pipeline, including specific models used (BS-Roformer, Silero VAM, pyannote, Qwen3 variants) and hyperparameters. The inclusion of prompts for the LLM judges and the annotation pipeline enhances reproducibility. The release of code and audio samples (or metadata for copyrighted audio) further supports reproducibility. The strict quality control steps are well-documented, allowing other researchers to replicate the benchmark construction.
Limitations
The benchmark is currently limited to Chinese-English pairs, restricting its generalizability to other language pairs. The reliance on LLM-based evaluation introduces potential biases inherent in the judge models, although human correlation mitigates this to some extent. The scenario style dimension remains subjective and shows lower correlation with human judgments compared to emotion and NVs. The benchmark size, while substantial, may not cover the full diversity of real-world speech scenarios, particularly rare or highly specialized contexts.
Broader impact
This work significantly advances the field of S2ST by highlighting the importance of expressiveness preservation, which is crucial for applications like dubbing, virtual assistants, and cross-lingual communication. By providing a standardized benchmark and evaluation metric, it enables fair comparison of future S2ST systems and drives research towards more human-like and expressive translation. The findings suggest that current systems are not yet ready for high-quality expressive dubbing, setting a clear direction for future improvements. This paper presents a significant contribution to the field of Speech-to-Speech Translation by introducing STEB, a comprehensive benchmark that evaluates not just translation accuracy but also the preservation of expressive attributes such as emotion, scenario style, and nonverbal vocalizations. The proposed "caption-then-summarize" LLM-based evaluation framework provides a scalable and reference-free solution to a previously intractable problem, validated by strong human correlation. The empirical results reveal a critical gap between semantic transfer and expressive transfer in current S2ST systems, offering valuable insights for future model development and establishing a new standard for evaluating expressive speech technologies.
Author of this paper? Send a correction.