This week’s shortlist
Recent generative techniques have facilitated the rapid proliferation of speech enhancement approaches built upon learnable compact latent representations and large generative backbones. Nevertheless, it remains inherently challenging to balance between acoustic fidelity…
Abstract
Recent generative techniques have facilitated the rapid proliferation of speech enhancement approaches built upon learnable compact latent representations and large generative backbones. Nevertheless, it remains inherently challenging to balance between acoustic fidelity, perceptual quality, and processing efficiency. In this paper, we revisit the Mel spectrum as a structured acoustic representation and propose BAME, a flow-matching-based framework that exploits the complementary priors for generative speech enhancement task. First, leveraging the explicit hierarchical frequency distribution of Mel spectrum, we elaborately devise a subband-aware Transformer (SaT) for efficient generative modeling. Using only 2.9% of the parameters, the proposed SaT achieves superior enhancement performance compared with a 331 M DiT backbone. Second, motivated by range-null decomposition theory of Mel filtering in the magnitude-spectral domain, we introduce a phase-aware neural vocoder that incorporates degraded-phase prior, together with a quality-guided dual-phase strategy to adaptively mitigate the effects of unreliable phase information. Third, we devise a simple yet effective one-step fidelity-perception alignment strategy that fine-tunes the flow model with complementary constraints from fidelity and perception domains. Extensive experiments are conducted on DNS-Challenge, EARS, GSR, and URGENT benchmarks, and both objective and subjective results demonstrate promising performance on various intrusive and non-intrusive evaluation metrics. These results further advocate the potential of exploiting structured acoustic representations for reliable and efficient generative speech enhancement. Audio samples and codes are available at https://github.com/AndongLi-speech/BAME.
AI-generated analysis. Check claims against the paper.
Summary
The paper proposes BAME, a flow-matching-based generative speech enhancement framework that leverages the structural priors of the Mel spectrum through a subband-aware Transformer and a phase-aware vocoder, achieving efficient one-step generation with balanced fidelity and perceptual quality. The technical contribution is significant due to the efficient backbone design, the rigorous exploitation of Mel's mathematical properties (RND), and the effective one-step fine-tuning strategy that addresses the fidelity-perception trade-off.
Methodology
The paper proposes BAME, a flow-matching-based framework for generative speech enhancement that revisits the Mel spectrum as a structured acoustic representation. The core novelty lies in the Subband-aware Transformer (SaT), which exploits the hierarchical frequency distribution of Mel bands to achieve efficient generative modeling with significantly fewer parameters (2.9% of a DiT baseline) while maintaining superior performance. The methodology also introduces a phase-aware neural vocoder that leverages the Range-Null Decomposition (RND) theory to recover magnitude information from Mel spectra and incorporates degraded phase priors. A key contribution is the Quality-guided Dual-Phase (QDV) strategy, which adaptively selects between zero-phase and degraded-phase vocoding based on non-intrusive quality scores to mitigate the instability of degraded phase in reverberant conditions. Finally, the One-Step Fidelity-Perception Alignment (OS-FPA) strategy fine-tunes the flow model to balance intrusive fidelity and non-intrusive perceptual quality in a single step, addressing the trade-off typically observed in iterative sampling methods.
Results & evaluation
The experiments are extensive, covering four major benchmarks: DNS-Challenge 2020, EARS, GSR, and URGENT. The evaluation includes a comprehensive suite of intrusive metrics (PESQ, eSTOI, LPS, SpkSim, SBS, WER) and non-intrusive metrics (DNSMOS, UTMOS, DiMOS, WVMOS, SCOREQ). Subjective A/B preference tests were conducted with 22 participants. Ablation studies thoroughly investigate the impact of the subband structure, the choice of backbone modules (GRU vs LSTM vs Conv), the trajectory type (FM vs Schrödinger Bridge), and the specific components of the OS-FPA strategy. The results demonstrate that the proposed SaT backbone outperforms larger DiT backbones and that the QDV strategy effectively improves robustness against phase artifacts.
Reproducibility
The paper provides a GitHub link with code and audio samples. Detailed hyperparameters, dataset configurations, and training procedures are described in the appendix and main text. The use of standard components (GRU, MHSA, WavLM) and open-source datasets enhances reproducibility. The specific implementation of the RND-based vocoder and the OS-FPA fine-tuning procedure are clearly defined.
Limitations
The method relies on a pre-trained WavLM model for semantic conditioning, which adds computational overhead during inference if not optimized. The QDV strategy requires running the vocoder twice and evaluating quality scores, which may increase latency compared to single-pass vocoding. The performance gains in some non-intrusive metrics are marginal compared to state-of-the-art generative methods, suggesting diminishing returns in perceptual quality. The reliance on Mel spectra may still limit the recovery of very fine-grained high-frequency details compared to complex spectrum modeling, although the phase-aware vocoder mitigates this.
Broader impact
This work advocates for the use of structured, handcrafted representations (Mel) over learned latents in generative speech enhancement, offering a more efficient and interpretable alternative. The subband-aware modeling approach could be extended to other audio processing tasks. The one-step generation strategy is particularly relevant for real-time applications where latency is critical. The integration of fidelity and perception alignment provides a framework for balancing objective reconstruction error with subjective quality, which is applicable to other generative audio tasks. The paper proposes BAME, a flow-matching-based generative speech enhancement framework that leverages the structural priors of the Mel spectrum through a subband-aware Transformer and a phase-aware vocoder, achieving efficient one-step generation with balanced fidelity and perceptual quality. The technical contribution is significant due to the efficient backbone design, the rigorous exploitation of Mel's mathematical properties (RND), and the effective one-step fine-tuning strategy that addresses the fidelity-perception trade-off.
Recent advances in large audio-language models (LALMs) have broadened audio understanding across speech, sound, and music, with tool-based agents further supporting these tasks by invoking predefined audio tools. However, a persistent gap remains between perceiving acoustic…
Abstract
Recent advances in large audio-language models (LALMs) have broadened audio understanding across speech, sound, and music, with tool-based agents further supporting these tasks by invoking predefined audio tools. However, a persistent gap remains between perceiving acoustic content and using it as evidence for reasoning: recognizing what is audible does not ensure that a model can identify which cues resolve an uncertain decision. To bridge this gap, we propose Think2Listen, an evidence-centric learning framework for agentic audio understanding. Guided by evidence before commitment, Think2Listen couples iterative reasoning with a targeted LALM interaction to actively seek acoustic cues that distinguish competing hypotheses and revise candidate judgments based on the observations returned. It learns what to ask, how to use evidence, and when to stop, using observation-induced changes in candidate-state to inform training credit. Extensive evaluations on two benchmarks spanning seven audio categories demonstrate that Think2Listen achieves (I) enhanced multi-step audio reasoning, with gains in overall accuracy of 5.08%--17.67% over competing audio agents; and (II) improved audio understanding and reasoning across speech, sound, and music, boosting overall accuracy by 2.29%--13.10% over the corresponding baselines. The source code is available at https://github.com/JinWang79/Think2Listen.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces an evidence-centric learning framework for agentic audio understanding that couples iterative reasoning with targeted LALM interactions, using observation-induced candidate-state changes to inform policy training. This approach effectively bridges the gap between acoustic perception and discriminative reasoning, demonstrating significant performance gains over existing audio agents by learning what to ask, how to use evidence, and when to stop.
Methodology
The paper proposes "Think2Listen," a framework that integrates an "Adaptive Audio Evidence Loop" with a novel training signal called "Transition-as-Credit." The methodology addresses the gap between acoustic perception and evidence-based reasoning by allowing an agent to iteratively query a Large Audio-Language Model (LALM) for specific acoustic cues before committing to an answer. The core technical contribution is the "Transition-as-Credit" mechanism, which uses a semantic extractor to reconstruct how observations change the agent's candidate state (eliminating or confirming hypotheses). This process-level feedback is used to calibrate rewards in a Group Relative Policy Optimization (GRPO) style update, distinguishing between useful evidence-seeking and misleading or redundant queries. The approach is theoretically sound, leveraging intermediate state changes to provide denser supervision signals than terminal accuracy alone.
Results & evaluation
The authors evaluate the framework on two benchmarks spanning seven audio categories (speech, sound, music). The results show significant improvements over competing audio agents (5.08%–17.67% accuracy gain) and baselines (2.29%–13.10% gain). The inclusion of ablation studies on the "Transition-as-Credit" component and the analysis of untrained interaction patterns (Figure 2) provides strong empirical support for the necessity of the proposed learning strategy. The experiments effectively demonstrate that learning *what* to ask and *when* to stop is crucial for agentic audio understanding.
Reproducibility
The paper provides a GitHub repository link, which is a positive indicator for reproducibility. The methodology is described with sufficient detail regarding the state formulation, the semantic extractor's role, and the reward calibration formulas. However, specific hyperparameters for the LALM and the actor policy are likely in the appendix (truncated in the provided text), which is standard. The reliance on a specific LALM for the "listener" role may limit immediate reproducibility if that model is not open-source or easily accessible, but the framework itself is clearly defined.
Limitations
The framework relies on the quality of the underlying LALM for generating observations; if the LALM hallucinates acoustic features, the agent's reasoning will be flawed, although the "misleading trajectory" flag attempts to mitigate this. The computational cost of iterative querying and the semantic extraction step for every turn may be high compared to single-shot inference. Additionally, the method is tested on specific benchmarks; generalization to open-ended audio tasks without predefined candidate sets $C$ is not explicitly addressed.
Broader impact
This work contributes to the emerging field of agentic multimodal AI, specifically in the audio domain. By formalizing "evidence-centric" learning, it offers a template for other modalities where perception and reasoning are decoupled. The technique of using state-transition changes for credit assignment is applicable to other tool-using agents. It advances the capability of AI systems to handle complex, multi-step audio reasoning tasks that require active information gathering rather than passive listening. The paper introduces an evidence-centric learning framework for agentic audio understanding that couples iterative reasoning with targeted LALM interactions, using observation-induced candidate-state changes to inform policy training. This approach effectively bridges the gap between acoustic perception and discriminative reasoning, demonstrating significant performance gains over existing audio agents by learning what to ask, how to use evidence, and when to stop.
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate…
Abstract
Zero-shot voice cloning typically emphasizes preserving the timbre and prosodic characteristics of a reference speaker. However, in real-world use, speech should adapt its delivery to the target text rather than remain tied to a fixed speaking style. In this work, we formulate this problem as text-adaptive prosody synthesis for zero-shot voice cloning and study how post-training can endow a cloning model with this capability. We develop a three-stage framework. First, Direct Preference Optimization (DPO) provides a cold start by aligning the model with text-appropriate expressive speech. Second, Group Sequence Policy Optimization (GSPO) jointly optimizes prosodic expressiveness, acoustic quality, intelligibility, and speaker similarity. Finally, context-conditioned On-Policy Self-Distillation (OPSD) enables self-improvement by transferring expressive behavior from strong-reference to weak-reference conditions. On Qwen3-TTS, our framework improves Emotion from 18.5 to 81.2 and Paralinguistics from 38.4 to 73.9. Extensive ablations validate the role of each stage.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a three-stage post-training framework (DPO, GSPO, OPSD) to achieve text-adaptive prosody in zero-shot TTS, significantly improving expressiveness while maintaining speaker identity. The technical contribution is strong, with a well-designed methodology that leverages modern RLHF techniques and LLM-based data pipelines, supported by extensive experiments and ablations that validate the effectiveness of each stage.
Methodology
The paper proposes a three-stage post-training framework (DPO, GSPO, OPSD) to enable text-adaptive prosody in zero-shot TTS. The methodology is sound, leveraging established RLHF techniques (DPO, GSPO) and a novel application of self-distillation (OPSD) to decouple speaker identity from reference prosody. The use of LLMs (Gemini) for automated preference data construction and reward modeling is a strong, modern approach that addresses the scarcity of high-quality expressive speech data. The formulation of the problem as "text-adaptive prosody synthesis" rather than just "voice cloning" is a meaningful conceptual shift.
Results & evaluation
Experiments are extensive, covering multiple benchmarks (Emergent TTS-Eval, Seed-TTS-Eval) and subjective evaluations (MOS, ABX). The ablation studies are thorough, isolating the contributions of each stage and hyperparameter choices (e.g., speaker diversity, reward weights). The results show significant improvements in emotion and paralinguistics scores, though there is a noted trade-off with speaker similarity, which is honestly reported. The use of human ABX tests to validate reduced reliance on reference expressiveness is a strong experimental design choice.
Reproducibility
Reproducibility is moderate. While the framework is well-described, reliance on proprietary models (Gemini 3.1, Qwen3-TTS) and specific datasets (Emilia, SeedVC) limits open reproduction. The automated data pipeline is detailed, but the exact prompts and filtering thresholds for Gemini-based verification are not fully specified. The code is not explicitly linked, though a demo page is provided.
Limitations
The primary limitation is the trade-off between expressiveness and speaker similarity, which the authors acknowledge but do not fully resolve. The method relies heavily on the quality of the LLM-based reward model and data generation, introducing potential biases. The evaluation is primarily on English and Chinese, with limited cross-lingual generalization analysis. The computational cost of the multi-stage training and LLM-in-the-loop data generation is not discussed.
Broader impact
This work has significant implications for the development of more natural and expressive TTS systems, moving beyond static voice cloning towards dynamic, context-aware speech generation. The framework could be adapted to other speech tasks and languages. The use of LLMs for data augmentation and reward modeling sets a precedent for future work in speech AI. The paper introduces a three-stage post-training framework (DPO, GSPO, OPSD) to achieve text-adaptive prosody in zero-shot TTS, significantly improving expressiveness while maintaining speaker identity. The technical contribution is strong, with a well-designed methodology that leverages modern RLHF techniques and LLM-based data pipelines, supported by extensive experiments and ablations that validate the effectiveness of each stage.
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present…
Abstract
Audio phase relations encode source locations, yet existing large audio-language models (LALM) reduce audio to single-channel content to stabilise general audio understanding, leaving spatial information for direction, distance, and source-relation questions unused. We present SpheraAudio, which accepts recordings with any channel count -- mono, stereo, First-Order Ambisonics (FOA), or higher -- and maps them into a single token stream. A spatial encoder turns complex multichannel spectra into spatial tokens interleaved with audio tokens at corresponding temporal positions. Spatial Multimodal RoPE (SMRoPE) makes each token's modality identity and temporal slot visible to rotary attention without altering the language model's original RoPE. Training runs in three stages -- spatial-encoder pretraining, projector alignment, and joint supervised-and-reinforcement learning -- progressing from geometric supervision to language-based spatial decisions, with answers trained as explicit spatial chain-of-thought: step-by-step geometric reasoning over the heard scene. Current benchmarks tie questions to a fixed channel-count format and score answers without spatial reasoning evidence. We introduce SpheraQA, which spans mono, stereo, and FOA recordings and organises questions hierarchically from single-attribute perception through cross-source relations to multi-step reasoning, scored programmatically against simulated scene geometry and real-recording annotations with step-by-step solution references. Across SpheraQA and public spatial-audio benchmarks, SpheraAudio sets a new performance standard for spatial audio understanding. It supports high-precision localisation, tracking, and reasoning over static and moving sound sources while preserving strong general-audio capabilities. We will release code, SpheraQA, and evaluation metrics to support reproducibility.
AI-generated analysis. Check claims against the paper.
Summary
The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.
Methodology
The paper proposes SpheraAudio, a framework integrating spatial audio understanding into Large Audio-Language Models (LALMs). The core methodological contributions are the "Spatial AuT" encoder, which processes complex multichannel spectra (FOA, stereo, mono) into spatial tokens, and "SMRoPE" (Spatial Multimodal RoPE), a novel rotary position embedding scheme that encodes modality identity and temporal alignment without altering the base LLM's RoPE. The training pipeline is rigorous, involving geometric pretraining of the spatial encoder, projector alignment, and joint SFT with explicit spatial Chain-of-Thought (CoT) supervision, followed by GRPO reinforcement learning. The approach effectively bridges the gap between raw multichannel signal processing and high-level language reasoning by explicitly pairing audio and spatial tokens in time.
Results & evaluation
The authors introduce SpheraQA, a substantial benchmark (20,492 questions) covering perception, relations, and reasoning across mono, stereo, and FOA formats. Experimental results show SpheraAudio outperforming strong baselines like Spatial-Omni-7B and Gemini 3.1 Pro on spatial tasks while retaining or improving general audio capabilities. Ablation studies convincingly demonstrate the necessity of token interleaving (vs. addition) and the specific SMRoPE design for temporal correspondence. The inclusion of stereo-specific experiments and general audio retention tests provides a comprehensive evaluation of the model's versatility.
Reproducibility
The paper provides detailed implementation specifications, including architecture dimensions, loss functions, hyperparameters for SFT and RL, and precise evaluation protocols for SpheraQA. The commitment to release code, the benchmark, and evaluation metrics enhances reproducibility. The use of GPT-4o for answer extraction in free-form responses is a practical but potentially biased evaluation choice, though mitigated by programmatic scoring for structured answers.
Limitations
The model relies on a 4B parameter backbone (Qwen3-Omni), which may limit its performance on extremely complex reasoning compared to larger LLMs. The spatial encoder is pretrained on simulated FOA data; while real recordings are included in evaluation, the generalization to uncalibrated, noisy real-world microphone arrays beyond the tested LOCATA/STARSS23 datasets is not fully explored. The "Motion direction" task shows a negative delta compared to the baseline, indicating a weakness in specific dynamic spatial reasoning.
Broader impact
This work significantly advances the field of spatial audio AI by providing a unified framework for handling arbitrary channel counts. It enables embodied agents to understand sound scenes with precision, supporting applications in navigation, robotics, and immersive media. The introduction of SpheraQA provides a much-needed standardized benchmark for evaluating spatial reasoning in LALMs, moving beyond simple localization to complex geometric reasoning. The paper presents SpheraAudio, a novel LALM framework that integrates spatial audio understanding via a dedicated spatial encoder and SMRoPE, achieving state-of-the-art performance on spatial reasoning tasks while preserving general audio capabilities. The contribution is significant due to its comprehensive handling of multichannel inputs, the introduction of the SpheraQA benchmark, and the rigorous training pipeline combining geometric pretraining, CoT supervision, and reinforcement learning, effectively bridging the gap between signal processing and language-based spatial reasoning.
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive…
Abstract
Rapid progress in text-to-audio (TTA) generation demands reliable and explainable automatic evaluation. However, existing metrics fail to simultaneously deliver high correlation with human judgments and meaningful, audio-grounded explanations, forcing reliance on labor-intensive human evaluation that bottlenecks model development iteration. To address this, we propose CoTTAGE, a reasoning-based evaluator for text relevance (REL) and audio quality (OVL). Following a reason-then-predict paradigm, CoTTAGE first generates an audio-grounded rationale to diagnose errors before predicting scores. Supported by over 10K newly collected human OVL annotations, CoTTAGE is first trained via supervised fine-tuning and then refined with reinforcement learning using a Thurstone-based preference objective that converts sampled score distributions into pairwise preference probabilities and rewards agreement with rankings derived from human judgments. Across six benchmarks spanning REL and OVL, CoTTAGE achieves state-of-the-art correlation with human judgments while producing human-preferred explanations to accelerate TTA development.
AI-generated analysis. Check claims against the paper.
Summary
CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Methodology
The paper proposes CoTTAGE, a reasoning-based evaluator for text-to-audio (TTA) generation that assesses both text relevance (REL) and overall audio quality (OVL). The core methodological contribution is the "reason-then-predict" paradigm, where the model generates an audio-grounded rationale before predicting a score. This is implemented via a two-stage training pipeline: (1) Supervised Fine-Tuning (SFT) on reasoning traces generated by a proprietary LLM (Gemini-3.1 Pro) conditioned on ground-truth MOS scores, and (2) Reinforcement Learning (RL) using a Thurstone-based pairwise preference objective. The RL stage is particularly interesting as it leverages the variance of sampled scores to create uncertainty-aware preference probabilities, rewarding the model for aligning with human rankings rather than just pointwise scores. The use of a Bhattacharyya coefficient for the reward signal is a solid technical choice for handling probabilistic preferences.
Results & evaluation
The experimental setup is robust, utilizing six benchmarks (RELATE, XACLE, PAM, Clotho) to test both in-domain and out-of-domain generalization. The collection of 10K+ new human OVL annotations is a significant contribution, as OVL data was previously scarce compared to REL data. The results show CoTTAGE outperforming strong baselines, including embedding-based metrics (CLAPScore, ELSA) and general-purpose LALMs (Gemini-3.1 Pro, Qwen2.5-Omni). The ablation study clearly demonstrates the benefit of the RL stage over SFT alone. The human preference evaluation of the rationales further validates the explainability claim, showing that CoTTAGE's explanations are preferred over those of the base model and even the proprietary teacher model in some cases.
Reproducibility
The paper provides high reproducibility. The code and data are publicly available on GitHub. Detailed hyperparameters, prompts, and annotation protocols are included in the appendices. The use of standard models (Qwen2.5-Omni-7B) and LoRA fine-tuning makes the method accessible to the community. The detailed description of the MTurk annotation process, including quality control measures, adds to the transparency of the dataset creation.
Limitations
The primary limitation is the reliance on a proprietary model (Gemini-3.1 Pro) for generating the reasoning traces during SFT. This introduces a dependency on a closed-source system and potential biases from that specific model. Additionally, the evaluation is limited to short audio clips (<30s), and the model does not provide temporal grounding (timestamps) for errors, which limits its utility for fine-grained debugging of long-form audio. The domain coverage is also limited to environmental sounds, with no evaluation on music or speech-centric generation.
Broader impact
This work addresses a critical bottleneck in TTA development: the lack of scalable, explainable automatic evaluation. By providing a model that not only scores audio but also explains *why* it gave that score, CoTTAGE enables more efficient iteration cycles for TTA model developers. The release of the 10K OVL annotations is a valuable resource for the community. The approach of using RL with preference objectives for audio evaluation is a promising direction that could be extended to other audio tasks. CoTTAGE introduces a reasoning-based evaluator for text-to-audio generation that achieves state-of-the-art correlation with human judgments by combining supervised fine-tuning with a novel Thurstone-based reinforcement learning objective, while also providing human-preferred, audio-grounded explanations for its scores.
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized…
Abstract
Speech tokenizers convert audio into discrete units, enabling language models to process and generate speech. The goal is to learn compact, text-like representations while preserving enough detail to reconstruct a speaker's voice and delivery. We propose Facto: a factorized tokenizer that disentangles time-varying information (content and prosody) from static components (speaker identity and recording conditions). We build on a recent linear factorization method for self-supervised features, extending it to handle unseen speakers. Then, we cluster the factorized space and train a lightweight decoder to reconstruct audio from the resulting tokens. By removing the static components before clustering, our tokens are more robust to small noise perturbations. We validate this by testing token stability under various noise conditions. Next, we theoretically analyze Facto's convergence and scaling properties, quantifying how the factorization improves with the number of training speakers. Finally, we evaluate Facto on a range of downstream tasks. We show that greater token stability translates into improved text-to-speech performance and competitive results in voice conversion and spoken language modeling. Code and checkpoints at https://github.com/ubisoft/ubisoft-laforge-facto/ and demo page at https://ubisoft-laforge.github.io/speech/facto/.
AI-generated analysis. Check claims against the paper.
Summary
Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.
Methodology
The paper proposes "Facto," a factorized speech tokenizer that disentangles time-varying content/prosody from static speaker/channel information. The core method builds on a linear factorization of self-supervised features (WavLM), extending it to handle unseen speakers via a linear content estimator. The methodology is theoretically grounded, providing convergence analysis for the factorization and a bias-variance trade-off analysis for the content estimator. The approach uses PCA and k-means clustering on the factorized content space, followed by a lightweight HiFiGAN-based decoder with FiLM conditioning for speaker identity reintroduction. The theoretical contributions (Theorem 1 and 2, Corollary 1) are a strong differentiator, offering rigorous bounds on error scaling with the number of training speakers.
Results & evaluation
The experiments validate the theoretical predictions, showing that the factorization stabilizes as the number of speakers increases (O(1/K) decay). The paper demonstrates improved token stability under noise perturbations compared to baselines. Downstream tasks include text-to-speech, voice conversion, and spoken language modeling. The results show competitive performance, with clear improvements in TTS due to token stability. The use of LibriSpeech and standard metrics provides a solid empirical foundation.
Reproducibility
The paper provides code and checkpoints on GitHub and a demo page. The methodology is described in sufficient detail, including hyperparameters (e.g., rank R=12, WavLM layer 6) and training setups. The theoretical derivations are provided in the main text or appendices, enhancing reproducibility.
Limitations
The method relies on the quality of the self-supervised features (WavLM) and the linear factorization assumption, which may not capture all non-linear interactions. The content estimator is linear, which might limit its ability to handle highly complex speaker variations. The paper focuses on English speech (LibriSpeech), and generalization to other languages or noisy real-world environments beyond small perturbations is not extensively tested.
Broader impact
The work contributes to the field of speech tokenization by providing a principled, theoretically sound approach to disentangling content and speaker identity. This can benefit downstream tasks like TTS, voice conversion, and speech language modeling by providing more robust and compact representations. The emphasis on stability and theoretical guarantees sets a high standard for future work in speech representation learning. Facto proposes a factorized speech tokenizer that disentangles content from speaker identity using linear factorization of self-supervised features, providing theoretical guarantees on convergence and stability. The paper makes a significant contribution by extending linear factorization to unseen speakers and rigorously analyzing the scaling properties with the number of training speakers, resulting in more robust tokens for downstream tasks like TTS and voice conversion.
In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what…
Abstract
In this report, we introduce the Reciprocal World Model (RWM) and present Duo v1, a full-duplex world model that continuously perceives audiovisual inputs while responding to users in real time. The RWM treats intelligence and physical reality as equally fundamental to what constitutes a world. Existing world models largely focus on simulating the latter, leaving reasoning and decision-making as external controls rather than intrinsic parts of the world itself. In RWM, we identify two equally fundamental capabilities: Intent, which provides the cognition underlying world interactions, and Realization, which generates coherent audio-visual behavior. To realize this formulation, Duo v1 introduces three key technical designs: (1) Leveraging autoregressive modeling for continuous interaction reasoning and participation decisions, rather than restricting it to instruction understanding for diffusion models. (2) Constructing chunk-specific conditions and synchronize tokens across Intent and Realization on a unified timeline. Module-specific attention masks enforce causal dependencies both within and across chunks, preventing access to future inputs. (3) Mitigating distribution shift in both modules through dual rollout distillation, training on autoregressively generated Intent tokens and synthesized audiovisual histories from Realization. Both modules continuously perceive real-time audio, video, and text, enabling full-duplex interaction with spontaneous conversations and interruptions. Beyond conversation, users can share real objects, reshape virtual scenes, guide assistants, or play audiovisual games without mouse or keyboard controls. The release of Duo v1 makes our first step, with the next version coming soon. We look forward to broader exploration of this emerging field.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a novel Reciprocal World Model architecture that tightly couples autoregressive intent reasoning with causal audiovisual generation to enable full-duplex, real-time interaction. The technical contribution is significant, particularly the dual rollout distillation and unified timeline synchronization, which address critical challenges in streaming multimodal generation. While the experimental section is truncated, the architectural innovation and open-source release suggest a high-impact contribution to the field of interactive world models.
Methodology
The paper proposes the Reciprocal World Model (RWM), a framework that integrates an autoregressive "Intent" module (based on a multimodal LLM) with a causal diffusion "Realization" module (based on a DiT). The core methodological contribution is the tight coupling of these two modules on a unified timeline using chunk-specific conditions and synchronize tokens. The Intent module handles high-level reasoning and participation decisions (speak/wait), while the Realization module generates synchronized audio and video. A key technical innovation is the "dual rollout distillation" strategy, which mitigates distribution shift by training the Realization module on autoregressively generated Intent tokens and synthesized audiovisual histories, ensuring that the inference-time conditioning matches the training distribution. The use of module-specific attention masks to enforce strict causality within and across chunks is a robust design choice for real-time streaming.
Results & evaluation
The provided text is truncated before the detailed experimental results section. However, the abstract and introduction claim that the system enables full-duplex interaction, spontaneous conversations, and interruptions. The paper mentions the release of "Duo v1" and provides links to a homepage and Hugging Face collection, suggesting that qualitative results and potentially some quantitative metrics are available externally. Without the full experimental section, it is difficult to rigorously assess the quantitative performance (e.g., latency, FID, CLAP scores, human preference studies) compared to baselines like Moshi or Qwen-Omni. The reliance on external links for primary results is a minor weakness for a standalone paper evaluation.
Reproducibility
The paper provides links to the GitHub repository and Hugging Face model weights, which significantly enhances reproducibility. The architectural details, such as the VAE compression ratios (40 latent vectors/sec for audio, 17 frames to 5 latent frames for video) and the specific attention mask strategies, are described in sufficient detail for implementation. The training data construction pipelines are mentioned but not fully detailed in the truncated text.
Limitations
The primary limitation is the lack of visible quantitative evaluation in the provided text. The system relies on a modular approach (LLM + Diffusion), which may introduce latency or synchronization challenges compared to unified end-to-end models, although the paper claims to address this with the unified timeline. The "dual rollout distillation" is computationally expensive as it requires generating rollouts during training. The paper is a "report" rather than a full conference paper, which may mean some theoretical justifications or ablation studies are omitted.
Broader impact
This work has significant potential impact on the field of human-computer interaction and embodied AI. By treating intelligence and physical reality as equally fundamental, it moves beyond passive video generation to active, reciprocal interaction. This could lead to more natural and engaging AI companions, virtual assistants, and interactive media. The full-duplex capability is a major step towards human-like conversational agents that can listen and speak simultaneously. The paper introduces a novel Reciprocal World Model architecture that tightly couples autoregressive intent reasoning with causal audiovisual generation to enable full-duplex, real-time interaction. The technical contribution is significant, particularly the dual rollout distillation and unified timeline synchronization, which address critical challenges in streaming multimodal generation. While the experimental section is truncated, the architectural innovation and open-source release suggest a high-impact contribution to the field of interactive world models.
Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the…
Abstract
Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the semantic level: small perturbations can disrupt their semantic encoding process and consequently cause severe degradation in downstream ASR and LALM performance. To study this vulnerability, we propose T-SemAttack, a transferable semantic attack based on a semantic encoder ensemble. By jointly drifting the representation spaces of multiple semantic encoders, T-SemAttack effectively disrupts the semantic content of speech while preserving perceptual quality, thereby inducing error features. We further analyze layer-wise representation drift and cross-model gradient alignment. Our findings show that perturbations can induce substantial shifts in deep semantic representations and that the input gradients of different surrogate encoders exhibit directional alignment. These observations suggest that shared adversarially sensitive directions in the input space may be a key factor underlying cross-model attack transferability. Building on these, we introduce ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation to improve reconstruction fidelity and downstream SpeechLLM robustness. Extensive experiments on multiple datasets, 9 open-source and black-box ASR systems, and 4 LALMs demonstrate that T-SemAttack achieves strong transferable attack performance, while the proposed Robust Speech Tokenizer exhibits robustness under high-fidelity reconstruction and downstream tasks. Our code and demo are available at https://wxzyd123.github.io/t-semattack.
AI-generated analysis. Check claims against the paper.
Summary
The paper identifies the speech tokenizer as a critical security bottleneck and proposes a transferable semantic attack (T-SemAttack) along with a robust tokenizer (ROSETok) to mitigate these vulnerabilities. The technical contribution is significant, combining a novel attack mechanism based on multi-encoder representation drift with a principled defense strategy involving semantic distillation, supported by extensive experiments across multiple ASR and LALM systems and rigorous human perceptual evaluations.
Methodology
The paper proposes T-SemAttack, a transferable semantic attack targeting the front-end speech tokenizer rather than the entire ASR/LALM pipeline. The core innovation is the use of a semantic encoder ensemble (S3Tokenizer, HuBERT, Whisper, MFCC) to jointly optimize perturbations that drift the representation space while preserving perceptual quality via a psychoacoustic masking constraint. This approach is methodologically sound, leveraging the shared structural bottlenecks (e.g., Mel-spectrogram inputs) of different encoders to achieve cross-model transferability. The proposed defense, ROSETok, employs a two-stage training strategy: base pretraining with noisy inputs and semantic distillation from a Whisper teacher, followed by attack-aware fine-tuning. The choice to target encoder-level semantic preservation rather than token-level invariance is well-justified by the analysis of how semantic drift, rather than mere token flipping, causes downstream failure.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors test the attack on 9 ASR systems (including black-box commercial APIs like GPT-4o-mini and Gemini) and 4 LALMs, demonstrating strong transferability. The inclusion of a human study (MOS evaluation) to verify perceptual quality is a significant strength, confirming that the attacks remain "harmless" to human listeners while being devastating to models. The evaluation of ROSETok shows clear improvements in reconstruction fidelity (STOI) and downstream semantic robustness (FSC scores) compared to baselines like DAC and StableToken. The ablation studies effectively isolate the contributions of the semantic loss, psychoacoustic constraint, and distillation components.
Reproducibility
The paper provides high reproducibility. Code and a demo are available. Detailed hyperparameters for both the attack (SGD steps, learning rate, perturbation budget) and the tokenizer training (batch size, learning rate schedules, loss weights) are specified. The datasets used (LibriTTS, Seed-TTS, Emilia) are standard or publicly available. The specific configuration of the surrogate encoders and the psychoacoustic model implementation details are described sufficiently for replication.
Limitations
The primary limitation is the computational cost of the attack, with an RTF of ~21.8, which is high for real-time applications, though the authors argue for an offline threat model. The defense (ROSETok) is evaluated primarily on Gaussian noise and the specific T-SemAttack; its robustness against other types of adversarial attacks or distribution shifts is not fully explored. The human study, while valuable, uses a small sample size (20 participants) and a limited set of utterances (4 speakers), which may not generalize to all demographic groups or acoustic conditions.
Broader impact
This work has significant implications for the safety and security of deployed speech AI systems. By identifying the tokenizer as a critical vulnerability point, it shifts the focus of security research from end-to-end model hardening to front-end robustness. The dual-use nature of the attack (potential for evading transcription or moderation) is acknowledged, but the defensive contribution (ROSETok) provides a constructive path forward. The findings are directly applicable to industry systems relying on speech tokenizers for LLM integration, urging a re-evaluation of their security posture. The paper identifies the speech tokenizer as a critical security bottleneck and proposes a transferable semantic attack (T-SemAttack) along with a robust tokenizer (ROSETok) to mitigate these vulnerabilities. The technical contribution is significant, combining a novel attack mechanism based on multi-encoder representation drift with a principled defense strategy involving semantic distillation, supported by extensive experiments across multiple ASR and LALM systems and rigorous human perceptual evaluations.
Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its…
Abstract
Can a speech generator also serve as an effective voice encoder? We show that a pretrained autoregressive text-to-speech (TTS) backbone can be repurposed for reference compression, improving speaker similarity in subsequent synthesis. Jointly encoding reference speech and its transcript further strengthens this capability. Controlled ablations and activation interventions trace the text benefit through intermediate acoustic representations into the compressed vectors that condition generation. Building on these findings, we introduce RSE-TTS, whose Reconstructive Speaker Encoder (RSE) shares the generation backbone and compresses paired reference text and speech into eight continuous vectors. Sharing reduces encoder-generator parameters by 48% relative to independent pretrained encoding. Large-scale English training yields a 118M-parameter generator that improves both word error rate and speaker similarity over the corresponding official MOSS-TTS-Nano cloning modes on all three English benchmarks. Combined with acoustic continuation, RSE-TTS approaches the speaker similarity of substantially larger systems while achieving lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local models. Audio samples are available at https://voxnexusai.github.io/RSE-TTS/.
AI-generated analysis. Check claims against the paper.
Summary
The paper demonstrates that a pretrained TTS backbone can be effectively repurposed as a text-aware reference encoder, significantly improving speaker similarity in compact models. By sharing the generation backbone and jointly encoding reference text and speech, RSE-TTS achieves a 48% reduction in parameters while outperforming baseline compact systems and approaching the quality of much larger models, supported by rigorous mechanistic analyses of the encoding process.
Methodology
The paper proposes RSE-TTS, a compact text-to-speech system that repurposes a pretrained autoregressive TTS backbone as a reference encoder. The core methodological contribution is the "Reconstructive Speaker Encoder" (RSE), which shares weights with the generation backbone to compress paired reference text and speech into eight continuous vectors. This parameter-sharing strategy reduces the encoder-generator parameter count by 48% compared to independent encoders. The authors provide a rigorous mechanistic analysis using activation patching and gradient interventions to demonstrate that the reference text influences intermediate acoustic representations, which are then captured by the query vectors. The approach effectively bridges the gap between generative modeling and speaker encoding, leveraging the text-speech alignment learned during pretraining to improve voice cloning fidelity in compact models.
Results & evaluation
The experimental evaluation is comprehensive, utilizing three major English benchmarks (Seed-TTS, CV3-eval, MiniMax). The authors conduct controlled ablations to isolate the effects of pretrained initialization, reference text inclusion, and parameter sharing. Results show that RSE-TTS outperforms the official MOSS-TTS-Nano in both Word Error Rate (WER) and Speaker Similarity (SIM). Furthermore, when combined with acoustic continuation, the system approaches the performance of much larger models like CosyVoice 3 and MOSS-TTS-Local while maintaining a significantly smaller footprint (129M vs. 859M+ parameters). The inclusion of mechanistic studies (e.g., reversing transcript order, patching activations) adds depth to the empirical validation, confirming the causal role of text-aware encoding in voice reconstruction.
Reproducibility
The paper provides detailed descriptions of the model architecture, training objectives, and evaluation protocols. However, the primary training data is an "in-house English speech corpus," which limits full reproducibility. The authors do not release the code or the specific checkpoint used for the final results, though they reference the base MOSS-TTS-Nano model. The detailed hyperparameters and loss weighting schemes in the appendix aid in partial reproduction, but the lack of open-source code and proprietary data remains a barrier.
Limitations
The system requires a reference transcript for optimal performance, which may not be available in all real-world scenarios. The paper focuses exclusively on English, leaving generalizability to other languages unverified. Additionally, the reliance on an in-house dataset for the final large-scale training limits the ability for the community to verify the scaling claims independently. The "acoustic continuation" mode, while effective, increases inference latency and context window requirements, partially negating the efficiency gains of the compact encoder.
Broader impact
This work contributes to the development of efficient, high-fidelity voice cloning systems suitable for edge devices or real-time applications. By demonstrating that generative capabilities can be transferred to encoding tasks, it offers a new paradigm for designing compact multimodal systems. The mechanistic insights into how text influences speaker representations could inform future architectures for zero-shot TTS and voice conversion. The paper demonstrates that a pretrained TTS backbone can be effectively repurposed as a text-aware reference encoder, significantly improving speaker similarity in compact models. By sharing the generation backbone and jointly encoding reference text and speech, RSE-TTS achieves a 48% reduction in parameters while outperforming baseline compact systems and approaching the quality of much larger models, supported by rigorous mechanistic analyses of the encoding process.