Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.
Primary: Meta AI
All Institutions: Meta AI
SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
The paper proposes a unified framework for song generation and editing based on "reconstructive pretraining." The core idea is to train a latent flow matching model (DiT-based) to reconstruct audio from a set of interpretable conditions. By varying the density of these conditions (sparse for generation, dense for editing), the model learns both to generate diverse songs and to perform fine-grained edits by modifying specific attributes while keeping others fixed. Key technical contributions include: (1) Word-level phoneme alignment, which fuses text and phoneme embeddings at word boundaries to improve pronunciation learning and convergence; (2) Beat conditioning, providing explicit beat/downbeat positions to improve musicality; and (3) Extending Representation Alignment (REPA) to the VAE latent space, aligning VAE latents with MusicFM features to create semantically meaningful latent representations. The editing mechanism relies on a residual encoder with a bottleneck capacity that controls the trade-off between reconstruction fidelity and editability. This approach avoids the need for paired editing data or noise injection/inversion techniques common in prior work.
The experiments are comprehensive, covering generation quality (WER, A4, SongEval), subjective evaluation (MOS), and editing capabilities. SongCraft achieves the lowest Word Error Rate (0.133) among baselines, significantly outperforming DiffRhythm (0.588) and YuE (0.414). It also achieves competitive perceptual quality scores, ranking second only to Levo (which uses DPO). The subjective evaluation confirms these trends, with SongCraft scoring highest in 5 out of 6 dimensions. The study of the residual encoder bottleneck provides valuable insights into the reconstruction-editability trade-off, showing non-monotonic behavior where moderate bottleneck sizes (M=15) yield the best balance. Ablations validate the contribution of each proposed component (phoneme alignment, beat conditioning, REPA).
The paper provides detailed implementation specifics, including model architecture (4.2B parameters, 36 layers), training hyperparameters (learning rate, batch size, GPU count), and data preprocessing steps. However, the use of an "internal dataset" limits full reproducibility for external researchers. The reliance on specific open-source tools for feature extraction (Whisper, Phonemizer, Beat This, BTC, SWIFT-F0, 3D-Speaker) is clearly documented, which aids in replicating the conditioning pipeline. The code and model weights are not explicitly linked in the provided text, though the institution (Meta AI) often releases such assets.
The primary limitation is the use of a proprietary internal dataset, which hinders independent verification and benchmarking. The model is large (4.2B parameters), requiring significant computational resources for training and inference. The editing capabilities, while novel, are limited to the specific attributes extracted (lyrics, melody, beats, speaker identity, chords) and may not generalize to arbitrary semantic edits without further conditioning. The non-monotonic behavior of the residual encoder bottleneck suggests that tuning this parameter is critical and potentially non-trivial for new datasets or domains.
This work advances the field of controllable music generation by unifying generation and editing in a single model without requiring paired data. The reconstructive pretraining paradigm could be applied to other audio domains (speech, sound effects) for controllable synthesis and editing. The integration of semantic alignment (REPA) into the VAE latent space offers a generalizable technique for improving the quality of latent diffusion/flow matching models. The ability to perform fine-grained edits (e.g., changing lyrics while preserving melody and timbre) has significant applications in music production, prototyping, and personalized content creation. SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Singapore Institute of Technology
The paper proposes a modular framework that augments frozen Large Audio-Language Models with a dedicated frame-level grounding model to achieve precise temporal localization. By decoupling semantic query representation from acoustic event detection and using cross-attention for frame-level alignment, the method significantly outperforms existing generative and specialized baselines across multiple benchmarks, while also enhancing downstream temporal reasoning capabilities when used as an external tool.
The paper proposes a modular framework that decouples semantic query understanding from fine-grained temporal localization. Instead of post-training the Large Audio-Language Model (LALM) to generate timestamp tokens (which is error-prone and lacks explicit acoustic grounding), the authors freeze the LALM and use its hidden states as semantic queries. A separate, lightweight grounding model uses cross-attention to align these semantic queries with frame-level audio features from a specialized encoder (ATST-Frame) and a CNN branch. This allows the model to predict frame-level activity scores directly, which are then converted to temporal intervals. The methodology is sound, leveraging the strengths of LALMs (semantic reasoning) and specialized audio encoders (temporal precision) while mitigating their respective weaknesses. The addition of a "tool-use" mechanism for downstream reasoning is a clever architectural choice that enhances the utility of the grounding model.
The experimental evaluation is comprehensive, covering five diverse benchmarks (AudioGrounding, DESED, UnAV-100, TACOS, Clotho-Moment). The results show consistent and significant improvements over strong baselines, including specialized grounding models (SpotSound, AM-DETR) and recent LALMs (Qwen3-Omni, Audio Flamingo 3). The gains are particularly notable at stricter IoU thresholds ([email protected]), validating the claim of improved precision. The ablation studies effectively isolate the contributions of audio context in the LALM, the choice of audio encoder, and the frame-level alignment mechanism. The downstream reasoning experiments on DCASE 2025 Task 5 further demonstrate the practical value of the grounding model as a tool for temporal reasoning.
The paper provides sufficient implementation details for reproducibility, including specific hyperparameters (learning rate, batch size, number of epochs, attention heads), architectural choices (Conformer decoder, CNN branch), and preprocessing steps (10s windows, 50Hz upsampling). The use of publicly available checkpoints for baselines and open-source models for the backbone (Qwen3-Omni, Audio Flamingo 3) facilitates reproduction. However, the specific code repository is not linked in the provided text, which is a minor drawback.
The framework relies on a frozen LALM, which means it cannot adapt to new semantic concepts or out-of-distribution queries without retraining the grounding model's alignment. The 10-second windowing strategy may limit performance on very long-form audio or events spanning multiple windows. The evaluation is limited to English audio and specific benchmarks; generalization to other languages or more complex, multi-event scenarios is not fully explored. The "tool-use" reasoning approach adds inference latency and complexity.
This work addresses a critical bottleneck in audio AI: the lack of precise temporal grounding in large multimodal models. By providing a modular solution that can be attached to existing LALMs, it offers a practical path for enhancing temporal perception in audio understanding systems. This has implications for applications requiring precise event localization, such as video editing, audio surveillance, and interactive audio assistants. The demonstrated improvement in downstream reasoning suggests that explicit temporal evidence can significantly boost the reliability of LALM-based reasoning tasks. The paper proposes a modular framework that augments frozen Large Audio-Language Models with a dedicated frame-level grounding model to achieve precise temporal localization. By decoupling semantic query representation from acoustic event detection and using cross-attention for frame-level alignment, the method significantly outperforms existing generative and specialized baselines across multiple benchmarks, while also enhancing downstream temporal reasoning capabilities when used as an external tool.
What did an ancestral bird species sound like? Existing ancestral state reconstruction methods can infer low-dimensional traits such as morphological characters at internal nodes of a phylogenetic tree, but no one has tried to produce rich perceptual signals such as audio. Some of the challenges include inferred representations that are either too low-dimensional to decode or lie in non-generative feature spaces, so no method to date can produce ancestral audio. We introduce the first framework that generates plausible ancestral vocalizations. Our pipeline encodes bird recordings into a VAE latent space, learns a low-dimensional trait projection aligned with phylogenetic distances, performs ancestral inference in this trait space, and recovers decodable latents through an anchored inverse lift before emitting novel waveforms for each ancestral node. Because the entire pipeline stays within a decodable latent space, every internal node receives a genuinely new audio output representing plausible intermediate ancestral sounds unavailable to retrieval-based alternatives. Experiments on two phylogenetically distant bird clades, 21-species Tyrannidae and 19-species Paridae, show that our method is the only approach that simultaneously achieves genuine generation, phylogenetic consistency, and naturalistic audio quality across both datasets.
Primary: University of Wisconsin-Madison
All Institutions: University of Wisconsin-Madison, Massachusetts Institute of Technology
The paper introduces the first framework for generating plausible ancestral vocalizations by combining phylogenetic inference with generative audio models. It effectively addresses the challenge of reconstructing high-dimensional perceptual signals on phylogenies by using a learned trait subspace and an anchored inverse lift to ensure decodable, naturalistic audio outputs that are phylogenetically consistent.
The paper proposes a novel four-stage pipeline for ancestral sound reconstruction: encoding audio into a VAE latent space, learning a low-dimensional tree-metric trait projection, performing Brownian motion ancestral inference in this trait space, and recovering decodable latents via an anchored inverse lift. The key technical contribution is the "anchored inverse lift," which resolves the underdetermined inverse problem of mapping from a low-dimensional trait space back to the high-dimensional VAE latent space by preserving the nullspace component (acoustic texture) from a nearby observed latent. This is mathematically grounded with proofs for exact trait preservation and nullspace preservation. The use of Cholesky reparametrization for the Stiefel manifold constraint is a solid optimization choice.
The experiments are conducted on two distinct bird clades (Tyrannidae and Paridae) to test generalization across suboscine (innate) and oscine (learned) vocalizations. The evaluation metrics are well-chosen for a task without ground truth, focusing on phylogenetic plausibility (Tree-Embedding Correlation, Edge Monotonicity) and audio quality (FAD, Silence Rate). The results clearly demonstrate that the proposed method outperforms naive latent-space BM in audio quality and retrieval baselines in novelty (Unique Rate). The ablation study effectively isolates the contribution of the learned projection and the anchoring strategy.
The paper provides detailed algorithmic descriptions, hyperparameters, and dataset curation steps. It specifies the use of pretrained Make-An-Audio 2 VAE and BigVGAN vocoder, which are publicly available. However, the specific code for the trait learning and ancestral inference pipeline is not linked in the provided text, which slightly hinders immediate reproducibility, though the mathematical formulation is clear enough for re-implementation.
The primary limitation is the lack of ground truth for ancestral sounds, making evaluation inherently indirect. The method relies on the assumption that Brownian motion is a suitable model for vocal evolution, which may not hold for all traits or clades, although the paper does test alternative models (OU, Pagel's lambda) in the appendix. The anchor dependence means that fine-grained acoustic texture is copied from extant species rather than inferred, which could introduce bias if the nearest anchor is not representative.
This work bridges evolutionary biology and machine learning, offering a tool to visualize and analyze ancestral traits in a perceptual modality. It could impact comparative biology, conservation efforts (reconstructing extinct species' calls), and the understanding of vocal evolution and speciation. The framework is generalizable to other high-dimensional traits where phylogenetic signal exists but direct generation is challenging. The paper introduces the first framework for generating plausible ancestral vocalizations by combining phylogenetic inference with generative audio models. It effectively addresses the challenge of reconstructing high-dimensional perceptual signals on phylogenies by using a learned trait subspace and an anchored inverse lift to ensure decodable, naturalistic audio outputs that are phylogenetically consistent.
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
Primary: University of Maryland
All Institutions: Adobe Research, University of Maryland, OpenAI
[One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
The paper proposes a structured taxonomy of 22 paralinguistic and acoustic characteristics, which is a significant conceptual contribution to the field. The methodology relies heavily on a data simulation engine that augments clean speech with Room Impulse Responses (RIRs) and noise to create controlled acoustic environments. This is a standard but effective approach for acoustic tasks. The core innovation lies in the two-stage curriculum learning strategy: Stage 1 focuses on atomic single-attribute QA pairs to build foundational knowledge, while Stage 2 uses LLM-generated multi-attribute QA pairs to foster joint reasoning. The use of Qwen2.5-7B for generating training data and Mistral-Small-3.2 for generating benchmark questions to avoid self-referential bias is a thoughtful methodological choice. However, the reliance on synthetic data for acoustic characteristics and existing datasets for speaker traits limits the generalizability to real-world, uncontrolled acoustic scenes. The model is fine-tuned from Qwen2-Audio-7B-Instruct using LoRA, which is a standard and efficient approach.
The experimental setup is robust, introducing ParA-Bench, a 6,000-question multiple-choice benchmark. The evaluation compares ParA-LLM against state-of-the-art models like GPT-4o-Audio, Qwen2-Audio, and specialized reasoning models. The results show a 7.5% improvement over GPT-4o-Audio on the proposed benchmark, which is a strong result. The ablation study on the curriculum learning shows consistent gains, validating the training strategy. The inclusion of downstream applications, such as Text-to-Impulse Response (Text2IR) generation, demonstrates the practical utility of the learned representations. However, the evaluation is primarily limited to the proposed benchmark and a few standard audio benchmarks (MMAU-Pro, MMAR). There is a lack of evaluation on other established paralinguistic benchmarks (e.g., IEMOCAP for emotion, or specific speaker verification datasets) to fully validate the model's capabilities across different paralinguistic tasks.
The paper provides detailed information on the data sources (EARS, Emilia, Expresso, VoxCeleb, MIT IR Survey, etc.), the simulation parameters, and the training hyperparameters (LoRA rank, learning rates, batch sizes). The release of the dataset, model, and benchmark significantly enhances reproducibility. The use of specific models for data generation (Qwen2.5-7B, Mistral-Small-3.2) is clearly stated. However, the exact prompts used for LLM-based QA generation are not fully detailed in the text, which could be a minor barrier to exact reproduction of the data pipeline.
The primary limitation is the reliance on synthetic acoustic data for the acoustic characteristics, which may not fully capture the complexity and variability of real-world acoustic environments. The speaker and speech characteristics are derived from existing datasets, which may have inherent biases or limited diversity. The model is based on a 7B parameter LLM, which may limit its scalability to more complex reasoning tasks compared to larger models. The benchmark, while novel, is relatively small (6K questions) and may not fully represent the breadth of paralinguistic understanding. The paper does not extensively discuss the computational cost of the data generation pipeline or the potential biases introduced by the LLMs used for data augmentation.
This work has significant potential impact on the field of audio AI by addressing a critical gap in paralinguistic understanding. The release of a large-scale dataset and a comprehensive benchmark will facilitate future research in this area. The demonstrated downstream applications, such as TTS control and acoustic space captioning, highlight the practical value of the model. The framework could be extended to other modalities or languages, and the curriculum learning approach could be applied to other multi-attribute understanding tasks. The work also contributes to the broader goal of building more human-like audio AI systems that can understand not just what is said, but how it is said. [One sentence main contribution]. The paper introduces ParA-LLM, a unified audio LLM framework that achieves state-of-the-art performance in paralinguistic and acoustic understanding through a novel taxonomy of 22 characteristics, a large-scale synthetic-augmented dataset, and a two-stage curriculum learning strategy, while releasing a new benchmark (ParA-Bench) to evaluate this underexplored dimension of speech AI.
Large Audio Language Models (LALMs) utilize either continuous features or discrete tokens, yet the optimal representation paradigm for general audio understanding remains debated. Existing benchmarks often focus on narrow domains or evaluate encoders outside LALM contexts. To address these gaps, we systematically evaluate continuous and discrete representations across speech, sound and music. Utilizing our UniARC framework with dual evaluation strategies across model scales from SmolLM2-135M to Llama-3-8B, we analyze the dynamic relationships of data volume, model capacity, and computational efficiency. Our results reveal the pivotal role of semantic constraints in tokenization for audio understanding and demonstrate that scaling backbones fail to compensate for information loss in audio representation, especially in data-limited tasks. These findings offer practical guidance for balancing semantic density, fidelity, and efficiency in future LALMs.
Primary: Tsinghua University
All Institutions: Tsinghua University
The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
The paper proposes the UniARC framework, a systematic evaluation methodology for comparing continuous (SSL-based) and discrete (clustering/codec-based) audio representations within Large Audio Language Models (LALMs). The methodology is sound, utilizing two distinct evaluation strategies: parameter-efficient fine-tuning (LoRA) on smaller backbones (SmolLM2) and frozen-backbone probing on larger backbones (Llama-3). This dual approach effectively isolates the impact of representation quality from the capacity of the language model. The choice of encoders is comprehensive, covering major SSL models (HuBERT, WavLM, Whisper) and discrete tokenizers (K-means, DAC, WavTokenizer, SpeechTokenizer). The standardization of feature processing (projectors, temporal concatenation) ensures fair comparison.
The experiments span three domains (speech, sound, music) and multiple tasks (ASR, emotion, captioning, classification). The results provide clear insights: semantic alignment is more critical than raw fidelity for understanding tasks. The finding that scaling the LLM backbone does not compensate for poor audio representation quality is a significant and counter-intuitive result that adds value. The analysis of data diversity and computational efficiency further strengthens the contribution. However, the sample size of encoders is limited, and the specific hyperparameters for the K-means clustering (1000 centroids) might not be optimal for all scales, though the authors justify this choice.
The paper provides sufficient details for reproduction, including model names, dataset names, and general training strategies (LoRA rank, learning rates). However, specific details on the projector architecture (beyond "two-layer MLP") and the exact prompt templates used for instruction tuning are not fully detailed in the provided text. The reliance on open-source models (SmolLM2, Llama-3, XARES-LLM base) aids reproducibility.
The study is limited to specific model scales (up to 8B) and a specific set of encoders. It does not explore hybrid approaches in depth, only comparing pure continuous vs. pure discrete. The "inverse scaling" phenomenon observed might be confounded by the simple MLP projector, as the authors acknowledge. The lack of ablation studies on the projector design is a minor weakness.
The findings have practical implications for the design of future LALMs, suggesting that investment in semantically rich audio encoders is more valuable than simply scaling up the LLM backbone for understanding tasks. This guides resource allocation in the field. The emphasis on multi-domain pre-training for encoders is also a valuable takeaway. The paper provides a rigorous and systematic comparison of continuous and discrete audio representations in LALMs, revealing that semantic compatibility is the primary driver of performance and that LLM scaling cannot compensate for poor audio encoding. This comprehensive benchmarking study offers critical guidance for the development of efficient and effective multimodal audio models.
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Primary: Alibaba Group
All Institutions: The Chinese University of Hong Kong, Alibaba Group, Shanghai Jiao Tong University, Shanghai Innovation Institute, Zhejiang University
The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
The paper introduces a coherent framework for "native" audio-visual dialogue, addressing the critical gap in data availability for models that process raw audio/video without ASR intermediaries. The core methodological contribution is OmniVChat-Studio, a multi-agent synthesis engine. The design is robust, utilizing distinct agents (Director, Renderer, Reviewer, Validator) to handle the complex pipeline of script generation, video rendering, and quality verification. The separation of "Fixed" and "Flexible" modules allows for extensibility to new subcategories without re-engineering the core pipeline. The proposed OmniVChat-RL reward design is particularly strong, moving beyond simple correctness to incorporate efficiency (conciseness) and style (naturalness), which are crucial for conversational agents. The use of GSPO (Group Sequence Policy Optimization) with LoRA is a practical and effective training strategy for large multimodal models.
The experimental setup is rigorous, featuring a synthetic benchmark (OmniVChat-Bench) and a human-recorded transfer probe (OmniVChat-Bench-Human). The evaluation metrics are well-defined, using LLM-based grading with tiered rubrics to handle the open-ended nature of dialogue. The results show significant improvements in both correctness and efficiency (reply length reduction) when using the proposed RL method. The comparison against state-of-the-art models (Gemini, Qwen, etc.) provides a strong baseline. The ablation studies effectively isolate the contributions of the efficiency and style rewards. The transfer from synthetic to human data is a key validation point, and the results support the claim that synthetic data can effectively train models for real-world scenarios.
The paper provides detailed descriptions of the agent roles, sampling strategies, and reward functions. The code and data are promised to be released, which is standard for such contributions. The specific prompts for the Director and Reviewer agents are referenced in appendices, which aids reproducibility. However, the reliance on specific proprietary video generation models (implied by the "Renderer" agent) and LLMs for grading may limit full reproducibility for external groups without access to these specific tools.
The primary limitation is the reliance on synthetic data for training and evaluation. While the paper argues for the validity of this approach, the "reality gap" between synthesized and real-world audio-visual noise, lighting, and human behavior remains a concern, even with the human-recorded probe. The evaluation relies heavily on LLM-as-a-judge, which can introduce biases or inconsistencies, although the paper attempts to mitigate this with detailed rubrics and agreement checks. The multi-turn evaluation is limited to single-turn probes in the human dataset, potentially underestimating the model's performance in long, complex real-world conversations.
This work has significant implications for the development of real-time, low-latency multimodal assistants. By demonstrating that synthetic data can effectively train models for native audio-visual dialogue, it lowers the barrier to entry for developing such systems, which are currently hindered by the scarcity of high-quality, annotated real-world data. The focus on efficiency and style in the reward model aligns with user expectations for natural, concise interactions, making the research highly relevant to industry applications in smart devices and virtual assistants. The paper defines the OmniVChat task and presents a comprehensive pipeline including a multi-agent synthesis engine, a tiered-rubric benchmark, and an RL training strategy that significantly improves the efficiency and correctness of native audio-visual dialogue models. The rigorous validation through both synthetic and human-recorded benchmarks, along with the detailed ablation of reward components, establishes a strong foundation for future research in low-latency multimodal interaction.
Audio large language models (AudioLLMs) operate reactively, responding only when queried. We introduce proactive audio assistance, where an AudioLLM monitors an audio stream and autonomously decides when to alert the user from a single natural-language intent, motivated by wearable applications for Deaf and Hard of Hearing users. We propose Interrupt and Silent Modeling (ISM), a model-agnostic paradigm that embeds proactive decisions into LLM decoding via two special tokens: \texttt{
Primary: Meta Reality Labs
All Institutions: Meta Reality Labs
The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
The paper proposes Interrupt and Silent Modeling (ISM), a model-agnostic framework that enables proactive audio assistance in AudioLLMs. The core method involves extending the vocabulary of a standard AudioLLM (Qwen2-Audio-7B) with two special tokens,
The experiments are conducted on ESC-50 (training and evaluation) and Epic-Sounds (zero-shot evaluation). The results on ESC-50 are exceptionally high (99.6% F1), which is somewhat suspiciously perfect for a real-world audio task, suggesting potential overfitting or a very easy evaluation setup (e.g., clean, isolated clips). The zero-shot transfer to Epic-Sounds is a strong point, showing the model generalizes to noisy, egocentric kitchen audio without domain-specific training. The streaming evaluation confirms real-time viability with 3.5s latency. However, the lack of comparison with other proactive or streaming audio models (if any exist) and the limited number of baselines (Zero-Shot, Reactive SFT) weakens the comparative analysis. The "perfect" de-duplication recall is also a red flag that warrants scrutiny of the evaluation protocol.
The paper provides sufficient details on the architecture (Qwen2-Audio-7B, LoRA rank 8), training procedure (two-stage SFT), and data construction for the four states. However, the exact prompts used for the "watch-out intent" and the specific construction of the streaming evaluation set are not fully detailed in the provided text. The code and data are not explicitly linked, which hinders immediate reproducibility.
The primary limitation is the computational cost of using a 7B LLM for a binary decision task, which may be prohibitive for battery-powered wearable devices despite the latency claim. The near-perfect scores on ESC-50 may not reflect real-world performance in highly complex, overlapping acoustic scenes. The paper focuses only on explicit intents, leaving implicit and semantic assistance for future work. The lack of user studies with DHH participants limits the validation of the system's practical utility.
This work has significant potential for assistive technology, particularly for Deaf and Hard of Hearing (DHH) users who rely on sound awareness. By enabling proactive, intent-based alerts with de-duplication, it addresses key pain points in current wearable sound awareness systems (notification fatigue, lack of personalization). The model-agnostic nature of ISM makes it a portable capability that can be integrated into future AudioLLMs, potentially driving a shift from reactive to proactive audio AI. The paper introduces a novel proactive audio assistance task and a simple, effective method (ISM) to enable it in AudioLLMs. While the technical contribution is solid and the application is highly relevant, the experimental results on ESC-50 are suspiciously high, and the computational efficiency for wearable deployment remains a concern.
The success of Large Audio Language Models has driven the development of massive multimodal networks exceeding billions of parameters. However, the demand for privacy-preserving, low-latency processing has shifted focus toward Small Audio Language Models (SALMs) capable of on-device execution. In this paper, we introduce Samsone, a family of SALMs designed for edge computing. Our core model, Samsone-134M, establishes a new state-of-the-art for its size class across multiple benchmarks. We further explore the scaling laws of SALMs by introducing Samsone-99M and Samsone-356M. Despite their compact footprint, the Samsone family delivers performance competitive with models orders of magnitude larger. To foster open research and reproducibility, we train Samsone on publicly available data. We release the training code, model weights, mobile-optimized checkpoints and provide an open-source Android application to demonstrate real-time on-device inference of Samsone.
Primary: Samsung R&D Institute Poland
All Institutions: Samsung R&D Institute Poland, AGH University of Krakรณw
The paper introduces Samsone, a family of small audio language models (SALMs) that achieve state-of-the-art performance for their size class by applying vocabulary reduction and depth pruning to standard encoder-decoder architectures. While the architectural components are not novel, the rigorous engineering of these models for on-device inference, combined with strong benchmark results that challenge the necessity of massive parameter counts for audio understanding, represents a significant practical contribution to the field of efficient multimodal AI.
The methodology relies on a standard "encoder-projector-decoder" architecture, which is not novel in itself. The core technical contribution lies in the systematic application of size optimization techniquesโspecifically vocabulary reduction and depth pruningโto existing open-source components (Whisper-Tiny encoder and SmolLM2 backbone). The approach is pragmatic rather than architecturally groundbreaking. The use of a SEP token to handle multiple audio inputs is a sensible design choice for a general-purpose audio LM, but it is not a significant theoretical advance. The training procedure is straightforward single-stage fine-tuning, lacking the complex multi-stage pre-training or alignment strategies seen in larger LALMs.
The experimental evaluation is the strongest aspect of the paper. The authors provide a comprehensive comparison against both Large Audio Language Models (LALMs) and existing Small Audio Language Models (SALMs) on rigorous benchmarks like MMAU and MMAU-Pro. The results are compelling: Samsone-134M significantly outperforms previous SALMs (Mellow, Pengi) and approaches the performance of models 60x larger (Qwen2-Audio). The inclusion of on-device latency benchmarks on a specific hardware (Samsung Galaxy S25 Ultra) adds practical value, validating the "on-device" claim. However, the ablation study is somewhat limited, testing only a few alternative components without deep diving into the impact of specific training data mixes or the exact trade-offs of the pruning strategies beyond the final performance numbers.
High. The authors explicitly state they release training code, model weights, and an Android application. The use of publicly available datasets (ReasonAQA, AudioSkillsXL) and open-source base models (Whisper, SmolLM2) ensures that the work can be easily reproduced by the community. The detailed description of the training setup (optimizer, learning rate, epochs) further supports reproducibility.
The paper acknowledges that the models suffer from a loss of general-purpose linguistic capabilities due to extensive AQA fine-tuning. This is a common issue in domain-specific fine-tuning but limits the model's versatility. Additionally, the evaluation is heavily skewed towards multiple-choice and short-answer tasks (AQA); performance on open-ended generation or complex reasoning tasks beyond the benchmark scope is less clear. The lack of hardware-specific optimizations (NPU/GPU) means the reported latency may not represent the best possible performance on modern mobile devices.
This work is highly relevant to the growing field of edge AI and privacy-preserving audio processing. By demonstrating that sub-100M parameter models can achieve competitive performance with billion-parameter models, it lowers the barrier to entry for deploying advanced audio understanding on consumer devices. This has implications for real-time applications in healthcare, smart home assistants, and mobile interfaces where data privacy and latency are critical. The paper introduces Samsone, a family of small audio language models (SALMs) that achieve state-of-the-art performance for their size class by applying vocabulary reduction and depth pruning to standard encoder-decoder architectures. While the architectural components are not novel, the rigorous engineering of these models for on-device inference, combined with strong benchmark results that challenge the necessity of massive parameter counts for audio understanding, represents a significant practical contribution to the field of efficient multimodal AI.
Computational models of auditory physiology commonly target specific responses or stages of the auditory pathway, limiting their ability to integrate findings across experimental paradigms and neural timescales. We present a foundation model of human auditory electrophysiology: a causal neural network trained to map binaural acoustic waveforms directly to high-sample-rate EEG. The model was trained on approximately 250 hours of EEG data from 92 subjects, with varied electrode montages and stimuli spanning tonebursts, speech, and music. We tested whether the model recovered effects of stimulus rate, frequency, and presentation method on auditory brainstem responses (ABRs); subcortical and cortical temporal response functions (TRFs) to continuous speech; and the click-evoked binaural interaction component (BIC). Predicted ABRs and TRFs reproduced established response morphology and stimulus-dependent effects, with model-grand-average correlations falling within the corresponding subject-level human distributions. The model-predicted BIC metrics closely resembled the values reported in the literature. These findings demonstrate that a single audio-to-EEG model can capture auditory physiology across paradigms and timescales, supporting future in silico experimentation and hearing technology applications.
Primary: University of Michigan
All Institutions: University of Michigan, Kresge Hearing Research Institute, Department of Otolaryngology--Head and Neck Surgery
The paper presents a causal deep neural network that maps binaural audio to high-sample-rate EEG, demonstrating the ability to reproduce established auditory physiological phenomena across multiple paradigms. The technical contribution is significant in its integration of diverse stimuli and subjects into a single model, offering a promising direction for in silico experimentation in auditory neuroscience and hearing technology development.
The authors propose a "foundation model" for auditory electrophysiology, utilizing a causal encoder-decoder architecture (WaveNet-based) to map binaural acoustic waveforms directly to high-sample-rate EEG. The methodology is robust, employing a 24-band filterbank, log-compression, and a 9-layer causal WaveNet encoder with dilated convolutions. A key technical contribution is the use of a Short-Time Fourier Transform (STFT) domain loss function that penalizes phase and amplitude errors, which is more suitable for EEG signal reconstruction than standard MSE. The inclusion of a parallel linear pathway to model stimulus-locked electromagnetic artifacts is a sophisticated engineering choice that improves training stability. The model is conditioned on subject identity and demographics, allowing for potential personalization, though the primary evaluation uses a "default" subject embedding.
The evaluation is comprehensive, testing the model on three distinct paradigms: parallel auditory brainstem responses (pABR), speech temporal response functions (TRFs), and binaural interaction components (BIC). The dataset is substantial (250 hours, 92 subjects), covering diverse stimuli (speech, music, tonebursts) and electrode montages. The results show that the model reproduces established physiological phenomena, such as frequency-dependent ABR morphology and rate-dependent latency/amplitude changes. Statistical comparisons (Crawford-Howell tests) indicate that model predictions fall within the distribution of human subject-level correlations, validating the model's biological plausibility. However, the lack of a direct comparison with state-of-the-art non-foundation models (e.g., specific ABR predictors or TRF estimators) limits the clarity of its relative performance advantage.
The paper provides detailed architectural specifications (kernel sizes, dilations, filterbank parameters) and training hyperparameters (learning rate, batch size, loss function details). The description of the artifact pathway and the STFT loss is sufficient for replication. However, the code and pre-trained weights are not explicitly linked in the provided text, and the specific implementation of the "default subject" zero-shot mechanism could benefit from further clarification. The data curation process (clock drift correction, resampling) is described, which is crucial for high-sample-rate EEG work.
The model is trained on a relatively young cohort (mean age 22.6 years), limiting generalizability to older populations or those with hearing loss. The evaluation relies heavily on correlation with grand averages and published statistics rather than direct prediction accuracy metrics (like SNR or RMSE) against held-out individual subjects in a standardized way. The "foundation model" claim is somewhat strong given the single-modality input (audio) and output (EEG) and the specific domain of auditory physiology; it is more accurately described as a specialized domain-specific model. The paper does not discuss computational cost or real-time inference capabilities, which are critical for the proposed hearing aid applications.
This work has significant potential for advancing computational auditory neuroscience by providing a unified framework for simulating brain responses to sound. It could accelerate the development of hearing technologies by enabling in silico testing of hearing aid algorithms. The causal nature of the model makes it suitable for real-time applications. However, the impact is currently limited to the auditory domain and specific electrophysiological signals, and broader impact would require validation on larger, more diverse datasets and integration with other neural signals. The paper presents a causal deep neural network that maps binaural audio to high-sample-rate EEG, demonstrating the ability to reproduce established auditory physiological phenomena across multiple paradigms. The technical contribution is significant in its integration of diverse stimuli and subjects into a single model, offering a promising direction for in silico experimentation in auditory neuroscience and hearing technology development.
Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy which traditional MOS predictors fail to capture. While proprietary Large Language Models (LLMs) like Gemini can evaluate these aspects, they are too costly for massive inference and reinforcement learning feedback. To address this, we first introduce Live-ProsodyJudge (LPJ), a cost-effective pairwise evaluator distilled from Gemini into Qwen3-Omni. However, we identify a critical flaw in standard multi-dimensional evaluation: verdict coupling. The judge tends to lazily align all individual dimension scores with its overall preference, collapsing a rich multi-dimensional rubric into a single preference bit. To resolve this, we further propose Decoupled-Live-ProsodyJudge (D-LPJ). D-LPJ eliminates the overall verdict target to prevent blind following, masks uncertain pair-dimensions during Supervised Fine-Tuning(SFT), and introduces a novel span-local GRPO strategy that applies normalized advantages strictly to their corresponding rationale spans. Evaluated on highly curated human-annotated test sets, 10 sample balanced-order LPJ achieves higher point accuracy than a single Gemini call, while D-LPJ successfully produces independent,decoupled dimension judgments. Furthermore, in a Best-of-8 TTS candidate selection tournament, the LPJ-selected utterance falls within the human top-3 in 85.29% of high-confidence cases, demonstrating its efficacy for fine-grained TTS preference optimization.
Primary: The Chinese University of Hong Kong
All Institutions: The Chinese University of Hong Kong, TaoLive-AIGC Team, Taobao & Tmall Group of Alibaba
The paper introduces a decoupled multi-dimensional prosody judge for live-streaming TTS, mitigating verdict coupling through span-local GRPO and achieving higher accuracy than its proprietary teacher model. This work provides a robust, cost-effective framework for fine-grained speech evaluation, enabling scalable preference optimization and offering a novel solution to the problem of correlated judgments in multi-dimensional LLM-based evaluators.
The paper proposes a two-stage approach to evaluating live-streaming TTS. First, it distills a proprietary LLM (Gemini) into an open-weight model (Qwen3-Omni) to create a cost-effective pairwise judge (LPJ). The core methodological contribution is the identification and mitigation of "verdict coupling," where judges lazily align all dimension scores with a single overall preference. To solve this, the authors propose D-LPJ, which removes the overall verdict target, masks uncertain dimensions during SFT, and introduces a "span-local GRPO" strategy. This reinforcement learning technique applies normalized advantages strictly to the rationale spans of specific dimensions, preventing cross-dimension reward cancellation. The methodology is sound, leveraging standard distillation techniques but applying them to a specific failure mode in multi-dimensional evaluation. The span-local GRPO is a clever application of RLHF principles to structured output generation.
The experiments are rigorous for an industry-adjacent paper. The authors construct a curated dataset of 1,043 human-annotated pairs, controlling for lexical confounding by using matched transcripts. They evaluate on multiple test sets (T1-T5) covering different TTS models and human recordings. Key results show that the distilled model (LPJ) outperforms a single call of the teacher model (Gemini) in point accuracy, which is a strong claim. The decoupled model (D-LPJ) successfully produces varied dimension judgments, with 86.10% agreement on high-confidence sets. The Best-of-8 selection experiment demonstrates practical utility, with the judge's top choice falling within the human top-3 in 85.29% of cases. The inclusion of position bias diagnostics and ablation studies (e.g., removing stage-1 SFT) adds depth to the evaluation.
Reproducibility is moderate. The paper provides detailed hyperparameters (LoRA ranks, learning rates, batch sizes) and describes the data construction process. However, the reliance on proprietary models (Gemini for distillation, Qwen3-Omni as base) and internal datasets (TaoLive data) limits full external reproducibility. The code is not explicitly linked, and the specific "span-local GRPO" implementation details, while described mathematically, would require significant effort to replicate without the exact codebase. The use of internal human annotations for the test sets is a strength for validity but a barrier for independent verification.
The primary limitation is the dependence on proprietary teacher models for distillation, which may not be accessible to all researchers. The evaluation is heavily focused on live-streaming TTS, which is a specific niche; generalizability to other speech domains (e.g., audiobooks, conversational agents) is not tested. The "span-local GRPO" is a novel idea but its computational cost and stability compared to standard GRPO are not deeply analyzed. The paper acknowledges that the decoupled model is currently limited to core dimensions (C1-C4) and does not fully address the sparse conditional dimensions (L1-L3) in the decoupled setting.
This work has significant impact on the field of speech synthesis evaluation. By providing a scalable, cost-effective alternative to expensive LLM-based judges, it enables more frequent and fine-grained evaluation of TTS systems, which is crucial for RLHF-based optimization. The concept of "verdict coupling" and its mitigation via span-local rewards could be applied to other multi-dimensional evaluation tasks in NLP and audio. The Best-of-N selection framework demonstrated here is directly applicable to production TTS pipelines. The paper introduces a decoupled multi-dimensional prosody judge for live-streaming TTS, mitigating verdict coupling through span-local GRPO and achieving higher accuracy than its proprietary teacher model. This work provides a robust, cost-effective framework for fine-grained speech evaluation, enabling scalable preference optimization and offering a novel solution to the problem of correlated judgments in multi-dimensional LLM-based evaluators.
Streaming automatic speech recognition (ASR) must be judged jointly on what it transcribes and on how quickly it commits each word. Delayed streams modeling (DSM) has become the dominant paradigm for streaming large audio-language models, exposing a structural delay $ฯ$ that bounds the decoder's lookahead. We show that $ฯ$ is a poor proxy for user-perceived latency, and that the alignment-based supervision of DSM leaves latency on the table: the same forced-aligned transcript is used at every $ฯ$, forcing the model to withhold words it could already commit. We introduce AWED, a word-level emission-delay metric defined relative to the acoustic end of each word, and post-train a DSM recognizer with GRPO under a reward that scores transcription accuracy and measured delay jointly. Trained at a single operating point ($ฯ=6$ frames), our model dominates both its supervised fine-tuning initialization and the Voxtral Realtime backbone across all evaluated lookahead budgets: it cuts WER by 30.8\% relative at an 80\,ms structural delay, and by 5.7\% relative at 480\,ms while lowering median AWED from 1.17\,s to 1.04\,s. Latency-rewarded post-training thus advances the accuracy--latency Pareto frontier of streaming ASR without architectural change.
Primary: University of Illinois Urbana-Champaign
All Institutions: University of Illinois Urbana-Champaign
The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
The paper proposes a two-stage training recipe for streaming ASR using Delayed Streams Modeling (DSM). Stage 1 is standard Supervised Fine-Tuning (SFT) to reproduce the Voxtral Realtime baseline. Stage 2 introduces a Reinforcement Learning (RL) post-training phase using Group Relative Policy Optimization (GRPO). The core methodological contribution is the definition of a new latency metric, AWED (Aligned Word Emission Delay), which measures the time difference between the acoustic end of a word and its emission by the model. This metric is used to construct a joint reward function that balances transcription accuracy (WER) and latency. The use of GRPO is appropriate for the discrete, non-differentiable nature of emission timing decisions. The method effectively decouples the optimization of "what to say" and "when to say it" by leveraging the existing token prediction head, avoiding architectural changes.
The experiments are conducted on a large-scale dataset of ~175k hours of English speech. Evaluation is performed on eight diverse benchmarks from the HuggingFace Open ASR Leaderboard. The results demonstrate consistent improvements in the accuracy-latency Pareto frontier compared to the SFT baseline and the original Voxtral Realtime model. Specifically, the model achieves a 30.8% relative WER reduction at tight latency budgets (80ms) and maintains accuracy gains at higher latency budgets while reducing median AWED. Ablation studies on training duration and reward weight provide insight into the hyperparameter sensitivity, confirming the robustness of the chosen configuration.
The paper provides sufficient detail for reproducibility, including the specific base model (Voxtral-Mini-4B-Realtime), the use of LoRA for parameter efficiency, the data sources, and the evaluation protocol. The definition of AWED is clearly specified. However, the specific implementation details of the GRPO loop (e.g., group size, clipping epsilon) are not fully detailed in the text provided, though standard GRPO practices are implied. The reliance on a specific proprietary checkpoint (Voxtral) may limit reproducibility for those without access to that specific model version, but the method is generalizable.
The study is limited to English ASR. The evaluation relies on a specific base model (Voxtral), so the generalizability of the AWED+GRPO approach to other streaming ASR architectures (e.g., RNN-T based or other LALMs) is not tested. The latency metric AWED, while novel, is specific to the DSM paradigm and may not directly apply to other streaming approaches. The paper does not discuss computational overhead of the RL training phase compared to SFT.
This work addresses a critical practical issue in streaming ASR: the gap between nominal structural delay and perceived user latency. By providing a metric (AWED) and an optimization method (GRPO with joint reward) to close this gap, the paper offers a path to more responsive voice assistants and real-time transcription systems. The approach of using RL to optimize non-differentiable aspects of generation (timing) is a valuable technique that could be extended to other tasks involving sequential decision-making with latency constraints. The paper introduces AWED, a word-level emission-delay metric, and applies GRPO with a joint accuracy-latency reward to post-train a streaming ASR model, significantly improving the accuracy-latency Pareto frontier. This work effectively bridges the gap between nominal structural delay and user-perceived latency in Delayed Streams Modeling, demonstrating that reinforcement learning can optimize discrete emission timing without architectural changes, resulting in substantial WER reductions at tight latency budgets.
Speech Large Language Models (Speech-LLMs), typically built from a pre-trained speech encoder, a modality projector, and an LLM fine-tuned with Low-Rank Adapters (LoRA), have shown strong Automatic Speech Recognition (ASR) performance on general-domain speech. However, adapting them to domain-shifted speech, such as child or dialectal speech, remains challenging under limited target-domain data. Given the dominant role of the LLM in Speech-LLMs, with cross-entropy loss applied only at the LLM output, the speech encoder may receive insufficient adaptation to new acoustic conditions. In this paper, we propose Encoder Awakening via Adapters (EAVA), a simple yet effective domain-adaptive fine-tuning method for Speech-LLM-based ASR. First, lightweight adapters are inserted into each encoder layer and trained exclusively, enabling target-domain acoustic knowledge to be incorporated into the encoder while preserving its pre-trained knowledge. Second, the full model is jointly fine-tuned on the target domain with LoRA applied to the LLM. Experiments on three domain-shifted ASR datasets, covering child and dialectal speech, show that EAVA consistently outperforms vanilla fine-tuning and other baselines, achieving new state-of-the-art performance.
Primary: University of California Los Angeles
All Institutions: University of California Los Angeles
Proposes a two-stage adapter-based fine-tuning method that effectively adapts the speech encoder in Speech-LLMs to domain-shifted data. The paper demonstrates that targeted "awakening" of the encoder via lightweight adapters, followed by joint fine-tuning, significantly outperforms standard fine-tuning strategies, achieving new state-of-the-art results on child and dialectal speech benchmarks while preserving pre-trained knowledge.
The paper proposes EAVA, a two-stage fine-tuning strategy for Speech-LLMs. The core insight is that standard fine-tuning often fails to adequately adapt the speech encoder to new acoustic domains because the LLM dominates the gradient flow. Stage 1 ("Encoder Awakening") freezes the entire pre-trained model and trains only lightweight adapters (Residual Adapters) inserted into the encoder layers. Stage 2 jointly fine-tunes the encoder, adapters, projector, and LLM LoRA. The methodology is sound, leveraging the parameter efficiency of adapters to preserve pre-trained knowledge while injecting domain-specific acoustic features. The use of CKA analysis to visualize layer-wise representation shifts provides a strong theoretical grounding for why this approach works (preserving lower layers, adapting upper layers).
The experiments are comprehensive, covering three distinct domain-shifted datasets (OGI Kids, MyST, CORAAL) and two different Speech-LLM backbones (Canary-Qwen, Phi-4-Multimodal). The paper reports state-of-the-art results, outperforming vanilla fine-tuning, multi-stage alignment, and previous SOTA methods. Ablation studies effectively isolate the contribution of the two-stage process and the choice of adapter architecture. The inclusion of cross-dataset transfer analysis and loss function variations (CTC vs. CE) adds significant depth to the evaluation.
The paper provides a public GitHub repository link. It specifies model architectures, dataset splits, hyperparameters (learning rates, adapter dimensions), and training schedules. The use of open-source backbones (Canary-Qwen, Phi-4) and standard datasets enhances reproducibility.
The method relies on the availability of a pre-trained Speech-LLM backbone. The gains, while consistent, are moderate in some cases (e.g., MyST). The paper does not extensively explore the computational overhead of the two-stage training compared to single-stage fine-tuning, though the parameter efficiency is highlighted. The "awakening" stage requires a separate training phase, which may be a barrier for rapid deployment scenarios.
This work addresses a critical bottleneck in deploying Speech-LLMs in real-world, diverse acoustic environments. By providing a parameter-efficient and effective domain adaptation method, it lowers the barrier for adapting large speech models to specific populations (children, dialect speakers) or noisy environments. The insights into encoder representation shifts via CKA are valuable for the broader community working on multimodal model adaptation. Proposes a two-stage adapter-based fine-tuning method that effectively adapts the speech encoder in Speech-LLMs to domain-shifted data. The paper demonstrates that targeted "awakening" of the encoder via lightweight adapters, followed by joint fine-tuning, significantly outperforms standard fine-tuning strategies, achieving new state-of-the-art results on child and dialectal speech benchmarks while preserving pre-trained knowledge.
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4\% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
Primary: Argmax, Inc.
All Institutions: Argmax, Inc., University of Virginia, Bilkent University
LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
The paper proposes Localized Attention-Constrained Inference (LACI), an inference-only technique to address reliability issues in long-form autoregressive TTS. The core insight is identifying "emergent alignment heads" within the model's attention mechanism that track the mapping between input text tokens and generated audio tokens. The authors define two specific failure modes: "skips" (deletions) and "hallucinations" (insertions/substitutions). LACI monitors the attention weights of these specific heads in real-time. If the attention pattern deviates from the expected monotonic progress (e.g., skipping a bin of text tokens or stalling), the system detects the error, rolls back the generation state to the error onset, and applies a temporary hard attention mask (guardrail) to force correct alignment during regeneration. This is a clever, low-overhead approach that leverages internal model states rather than external ASR feedback for detection, which is a significant methodological improvement over standard best-of-N sampling.
The experiments are rigorous and well-targeted. The authors use the AppTek Call Center dataset, a real-world long-form TTS use case, which is more relevant than standard short-utterance benchmarks for this specific problem. They evaluate on multiple model scales (Qwen3-TTS-0.6B, 1.7B) and families (VoxCPM2), demonstrating generalizability. The metrics include standard WER and SIM, but the introduction of wSIM (sliding-window speaker similarity) is a valuable contribution that exposes failure patterns hidden by global averaging. The results show dramatic improvements in worst-of-N WER (35.2% to 3.4%) and wSIM (0.01 to 0.47), effectively closing the gap between short-form and long-form reliability. The comparison against ACI (always-on guardrails) clearly demonstrates the superiority of the detection-and-recovery approach.
The paper states that code will be open-sourced upon publication, which is standard for arXiv preprints. However, the specific details of the "instability measure" and detector settings are referenced as being in the code, which is a slight barrier to immediate reproducibility without access to the repository. The use of standard open-weights models (Qwen3-TTS, VoxCPM2) and public datasets (AppTek) aids reproducibility. The manual verification of wSIM failures adds a layer of qualitative validation that is hard to automate but important for trust.
The method relies on the existence of "emergent alignment heads," which may not be present or stable in all TTS architectures, particularly non-autoregressive or diffusion-based models. The paper focuses on autoregressive neural codec language models. The computational overhead, while described as "negligible," involves rollback and regeneration, which could be significant for very long texts if failures are frequent. The evaluation is limited to English (implied by the datasets and models used). The "worst-of-N" metric, while useful for reliability, does not reflect the average user experience if N is large, though the paper argues for the importance of reliability in professional settings.
This work has high practical impact for deploying TTS in real-world applications like audiobooks, customer support agents, and long-form voice cloning. The ability to reliably generate long-form speech with consistent speaker identity is a critical bottleneck for current TTS systems. The proposed wSIM metric is a valuable tool for the community to better evaluate speaker consistency in long-form generation. The inference-only nature of the solution makes it easily adoptable by users of existing open-weights models without retraining. LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.
Primary: Meta AI
All Institutions: Meta AI
SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
The paper proposes a unified framework for song generation and editing based on "reconstructive pretraining." The core idea is to train a latent flow matching model (DiT-based) to reconstruct audio from a set of interpretable conditions. By varying the density of these conditions (sparse for generation, dense for editing), the model learns both to generate diverse songs and to perform fine-grained edits by modifying specific attributes while keeping others fixed. Key technical contributions include: (1) Word-level phoneme alignment, which fuses text and phoneme embeddings at word boundaries to improve pronunciation learning and convergence; (2) Beat conditioning, providing explicit beat/downbeat positions to improve musicality; and (3) Extending Representation Alignment (REPA) to the VAE latent space, aligning VAE latents with MusicFM features to create semantically meaningful latent representations. The editing mechanism relies on a residual encoder with a bottleneck capacity that controls the trade-off between reconstruction fidelity and editability. This approach avoids the need for paired editing data or noise injection/inversion techniques common in prior work.
The experiments are comprehensive, covering generation quality (WER, A4, SongEval), subjective evaluation (MOS), and editing capabilities. SongCraft achieves the lowest Word Error Rate (0.133) among baselines, significantly outperforming DiffRhythm (0.588) and YuE (0.414). It also achieves competitive perceptual quality scores, ranking second only to Levo (which uses DPO). The subjective evaluation confirms these trends, with SongCraft scoring highest in 5 out of 6 dimensions. The study of the residual encoder bottleneck provides valuable insights into the reconstruction-editability trade-off, showing non-monotonic behavior where moderate bottleneck sizes (M=15) yield the best balance. Ablations validate the contribution of each proposed component (phoneme alignment, beat conditioning, REPA).
The paper provides detailed implementation specifics, including model architecture (4.2B parameters, 36 layers), training hyperparameters (learning rate, batch size, GPU count), and data preprocessing steps. However, the use of an "internal dataset" limits full reproducibility for external researchers. The reliance on specific open-source tools for feature extraction (Whisper, Phonemizer, Beat This, BTC, SWIFT-F0, 3D-Speaker) is clearly documented, which aids in replicating the conditioning pipeline. The code and model weights are not explicitly linked in the provided text, though the institution (Meta AI) often releases such assets.
The primary limitation is the use of a proprietary internal dataset, which hinders independent verification and benchmarking. The model is large (4.2B parameters), requiring significant computational resources for training and inference. The editing capabilities, while novel, are limited to the specific attributes extracted (lyrics, melody, beats, speaker identity, chords) and may not generalize to arbitrary semantic edits without further conditioning. The non-monotonic behavior of the residual encoder bottleneck suggests that tuning this parameter is critical and potentially non-trivial for new datasets or domains.
This work advances the field of controllable music generation by unifying generation and editing in a single model without requiring paired data. The reconstructive pretraining paradigm could be applied to other audio domains (speech, sound effects) for controllable synthesis and editing. The integration of semantic alignment (REPA) into the VAE latent space offers a generalizable technique for improving the quality of latent diffusion/flow matching models. The ability to perform fine-grained edits (e.g., changing lyrics while preserving melody and timbre) has significant applications in music production, prototyping, and personalized content creation. SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Singapore Institute of Technology
The paper proposes a modular framework that augments frozen Large Audio-Language Models with a dedicated frame-level grounding model to achieve precise temporal localization. By decoupling semantic query representation from acoustic event detection and using cross-attention for frame-level alignment, the method significantly outperforms existing generative and specialized baselines across multiple benchmarks, while also enhancing downstream temporal reasoning capabilities when used as an external tool.
The paper proposes a modular framework that decouples semantic query understanding from fine-grained temporal localization. Instead of post-training the Large Audio-Language Model (LALM) to generate timestamp tokens (which is error-prone and lacks explicit acoustic grounding), the authors freeze the LALM and use its hidden states as semantic queries. A separate, lightweight grounding model uses cross-attention to align these semantic queries with frame-level audio features from a specialized encoder (ATST-Frame) and a CNN branch. This allows the model to predict frame-level activity scores directly, which are then converted to temporal intervals. The methodology is sound, leveraging the strengths of LALMs (semantic reasoning) and specialized audio encoders (temporal precision) while mitigating their respective weaknesses. The addition of a "tool-use" mechanism for downstream reasoning is a clever architectural choice that enhances the utility of the grounding model.
The experimental evaluation is comprehensive, covering five diverse benchmarks (AudioGrounding, DESED, UnAV-100, TACOS, Clotho-Moment). The results show consistent and significant improvements over strong baselines, including specialized grounding models (SpotSound, AM-DETR) and recent LALMs (Qwen3-Omni, Audio Flamingo 3). The gains are particularly notable at stricter IoU thresholds ([email protected]), validating the claim of improved precision. The ablation studies effectively isolate the contributions of audio context in the LALM, the choice of audio encoder, and the frame-level alignment mechanism. The downstream reasoning experiments on DCASE 2025 Task 5 further demonstrate the practical value of the grounding model as a tool for temporal reasoning.
The paper provides sufficient implementation details for reproducibility, including specific hyperparameters (learning rate, batch size, number of epochs, attention heads), architectural choices (Conformer decoder, CNN branch), and preprocessing steps (10s windows, 50Hz upsampling). The use of publicly available checkpoints for baselines and open-source models for the backbone (Qwen3-Omni, Audio Flamingo 3) facilitates reproduction. However, the specific code repository is not linked in the provided text, which is a minor drawback.
The framework relies on a frozen LALM, which means it cannot adapt to new semantic concepts or out-of-distribution queries without retraining the grounding model's alignment. The 10-second windowing strategy may limit performance on very long-form audio or events spanning multiple windows. The evaluation is limited to English audio and specific benchmarks; generalization to other languages or more complex, multi-event scenarios is not fully explored. The "tool-use" reasoning approach adds inference latency and complexity.
This work addresses a critical bottleneck in audio AI: the lack of precise temporal grounding in large multimodal models. By providing a modular solution that can be attached to existing LALMs, it offers a practical path for enhancing temporal perception in audio understanding systems. This has implications for applications requiring precise event localization, such as video editing, audio surveillance, and interactive audio assistants. The demonstrated improvement in downstream reasoning suggests that explicit temporal evidence can significantly boost the reliability of LALM-based reasoning tasks. The paper proposes a modular framework that augments frozen Large Audio-Language Models with a dedicated frame-level grounding model to achieve precise temporal localization. By decoupling semantic query representation from acoustic event detection and using cross-attention for frame-level alignment, the method significantly outperforms existing generative and specialized baselines across multiple benchmarks, while also enhancing downstream temporal reasoning capabilities when used as an external tool.
What did an ancestral bird species sound like? Existing ancestral state reconstruction methods can infer low-dimensional traits such as morphological characters at internal nodes of a phylogenetic tree, but no one has tried to produce rich perceptual signals such as audio. Some of the challenges include inferred representations that are either too low-dimensional to decode or lie in non-generative feature spaces, so no method to date can produce ancestral audio. We introduce the first framework that generates plausible ancestral vocalizations. Our pipeline encodes bird recordings into a VAE latent space, learns a low-dimensional trait projection aligned with phylogenetic distances, performs ancestral inference in this trait space, and recovers decodable latents through an anchored inverse lift before emitting novel waveforms for each ancestral node. Because the entire pipeline stays within a decodable latent space, every internal node receives a genuinely new audio output representing plausible intermediate ancestral sounds unavailable to retrieval-based alternatives. Experiments on two phylogenetically distant bird clades, 21-species Tyrannidae and 19-species Paridae, show that our method is the only approach that simultaneously achieves genuine generation, phylogenetic consistency, and naturalistic audio quality across both datasets.
Primary: University of Wisconsin-Madison
All Institutions: University of Wisconsin-Madison, Massachusetts Institute of Technology
The paper introduces the first framework for generating plausible ancestral vocalizations by combining phylogenetic inference with generative audio models. It effectively addresses the challenge of reconstructing high-dimensional perceptual signals on phylogenies by using a learned trait subspace and an anchored inverse lift to ensure decodable, naturalistic audio outputs that are phylogenetically consistent.
The paper proposes a novel four-stage pipeline for ancestral sound reconstruction: encoding audio into a VAE latent space, learning a low-dimensional tree-metric trait projection, performing Brownian motion ancestral inference in this trait space, and recovering decodable latents via an anchored inverse lift. The key technical contribution is the "anchored inverse lift," which resolves the underdetermined inverse problem of mapping from a low-dimensional trait space back to the high-dimensional VAE latent space by preserving the nullspace component (acoustic texture) from a nearby observed latent. This is mathematically grounded with proofs for exact trait preservation and nullspace preservation. The use of Cholesky reparametrization for the Stiefel manifold constraint is a solid optimization choice.
The experiments are conducted on two distinct bird clades (Tyrannidae and Paridae) to test generalization across suboscine (innate) and oscine (learned) vocalizations. The evaluation metrics are well-chosen for a task without ground truth, focusing on phylogenetic plausibility (Tree-Embedding Correlation, Edge Monotonicity) and audio quality (FAD, Silence Rate). The results clearly demonstrate that the proposed method outperforms naive latent-space BM in audio quality and retrieval baselines in novelty (Unique Rate). The ablation study effectively isolates the contribution of the learned projection and the anchoring strategy.
The paper provides detailed algorithmic descriptions, hyperparameters, and dataset curation steps. It specifies the use of pretrained Make-An-Audio 2 VAE and BigVGAN vocoder, which are publicly available. However, the specific code for the trait learning and ancestral inference pipeline is not linked in the provided text, which slightly hinders immediate reproducibility, though the mathematical formulation is clear enough for re-implementation.
The primary limitation is the lack of ground truth for ancestral sounds, making evaluation inherently indirect. The method relies on the assumption that Brownian motion is a suitable model for vocal evolution, which may not hold for all traits or clades, although the paper does test alternative models (OU, Pagel's lambda) in the appendix. The anchor dependence means that fine-grained acoustic texture is copied from extant species rather than inferred, which could introduce bias if the nearest anchor is not representative.
This work bridges evolutionary biology and machine learning, offering a tool to visualize and analyze ancestral traits in a perceptual modality. It could impact comparative biology, conservation efforts (reconstructing extinct species' calls), and the understanding of vocal evolution and speciation. The framework is generalizable to other high-dimensional traits where phylogenetic signal exists but direct generation is challenging. The paper introduces the first framework for generating plausible ancestral vocalizations by combining phylogenetic inference with generative audio models. It effectively addresses the challenge of reconstructing high-dimensional perceptual signals on phylogenies by using a learned trait subspace and an anchored inverse lift to ensure decodable, naturalistic audio outputs that are phylogenetically consistent.
Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models. Although this design enables streaming generation with explicit textual guidance, generated acoustic tokens become part of the context for subsequent text predictions. Given identical speech inputs, we observe markedly lower answer accuracy for the internal text generated in speech-to-text-and-speech (S2TS) mode than for speech-to-text (S2T) responses. We term this discrepancy the \emph{output-mode gap} (OMG). To reduce OMG, we propose \emph{Joint-Output On-Policy Distillation} (JO-OPD), which distills the model's stronger S2T policy into joint generation using student-generated S2TS trajectories. At each text position, the S2T teacher provides soft targets from a text-only projection of the student's preceding outputs, while the student predicts from the corresponding full interleaved history. A preservation objective further regularizes native non-text predictions. Experiments on Step-Audio-2-mini and Baichuan-Audio-Instruct reveal OMG across two interleaved generation architectures. On Step-Audio-2-mini, JO-OPD reduces OMG from 42.87 to 16.26 percentage points on Spoken-MQA and from 29.72 to 13.04 points on speech-rendered GSM8K, with little change in S2T accuracy and substantially larger reductions than matched SFT baselines. ASR-based evaluation further shows a 7.49-point improvement in spoken-answer accuracy on Spoken-MQA.
Primary: Huawei (AI Lab, Leibniz Research Center)
All Institutions: AI Lab, Leibniz Research Center, Huawei, The Chinese University of Hong Kong, City University of Hong Kong
The paper identifies and quantifies the "Output-Mode Gap" in interleaved speech LLMs and proposes Joint-Output On-Policy Distillation (JO-OPD) to mitigate it by distilling the model's stronger text-only policy into the joint generation process. The technical contribution is significant as it provides a principled way to improve text reasoning accuracy in speech-output modes without sacrificing speech generation quality, supported by rigorous experiments on two major architectures and ablation studies validating the key components of the method.
The paper introduces Joint-Output On-Policy Distillation (JO-OPD) to address the "Output-Mode Gap" (OMG) in interleaved speech-language models. The core insight is that when a model generates both text and audio tokens, the acoustic tokens in the context degrade the quality of subsequent text predictions compared to a pure text-only mode. JO-OPD leverages the model's own stronger Speech-to-Text (S2T) policy as a teacher to supervise the Speech-to-Text-and-Speech (S2TS) student. A key technical contribution is the "text-only projection" of the student's history for the teacher, allowing the teacher to provide soft targets based on the semantic content without being confused by the acoustic noise in the context. Additionally, a preservation objective regularizes the non-text (acoustic) predictions to prevent the distillation process from degrading speech generation quality. The method is theoretically sound, leveraging on-policy distillation to align the student's distribution with the teacher's on the student's own trajectories, which is a robust approach for reducing exposure bias in autoregressive generation.
The experiments are conducted on two distinct architectures: Step-Audio-2-mini and Baichuan-Audio-Instruct. The evaluation metrics focus on answer accuracy for Spoken-MQA and GSM8K, comparing S2T accuracy vs. S2TS internal text accuracy. The results show a significant reduction in OMG (e.g., from 42.87 to 16.26 points on Spoken-MQA for Step-Audio-2). The paper also includes ablations on soft vs. hard targets, the importance of non-text preservation, and the effect of training scale. The inclusion of ASR-based evaluation for spoken answer accuracy provides a holistic view of the impact on final user-facing output. The comparison against Self-SFT and Response-SFT baselines is appropriate and demonstrates the superiority of the proposed on-policy distillation approach.
The paper provides detailed hyperparameters (learning rate, batch size, top-k values) and describes the training setup clearly. However, the specific implementation details of the "text-only projection" and the exact mechanism for the preservation loss in different architectures (Step-Audio vs. Baichuan) could benefit from more pseudocode or explicit mathematical definitions in the main text. The use of synthesized speech (flite) for training is noted, which is a practical detail for reproducibility, though it may limit generalization to diverse speaker characteristics.
The primary limitation is the reliance on the model's own S2T capability as the teacher; if the base model has weak S2T performance, the distillation benefit may be limited. The evaluation is primarily on question-answering and reasoning tasks, so the impact on open-ended conversational quality or prosody is less clear, although the preservation loss aims to mitigate this. The training data size (27k prompts) is relatively small, and the long-term stability of the model after such fine-tuning is not extensively explored.
This work addresses a critical issue in the deployment of speech LLMs: the degradation of reasoning capabilities when speech output is enabled. By providing a method to close this gap, the paper contributes to the development of more reliable and capable voice assistants. The technique of using internal modality policies as teachers is a generalizable insight that could be applied to other multimodal interleaved generation tasks. The paper identifies and quantifies the "Output-Mode Gap" in interleaved speech LLMs and proposes Joint-Output On-Policy Distillation (JO-OPD) to mitigate it by distilling the model's stronger text-only policy into the joint generation process. The technical contribution is significant as it provides a principled way to improve text reasoning accuracy in speech-output modes without sacrificing speech generation quality, supported by rigorous experiments on two major architectures and ablation studies validating the key components of the method.
When reinforcement learning (RL) is used for post-training automatic speech recognition (ASR), the reward almost always lives in the text space: it compares a hypothesis with the reference and never checks whether the hypothesis is supported by the audio. On highly regular speech this licenses a shortcut - guessing from a strong language prior rather than listening. Once the acoustics degrade, the shortcut runs unchecked and emits fluent but ungrounded words, i.e., insertion errors. We propose an acoustic-fidelity reward: a GRPO reward augmented with a separately pretrained, permanently frozen, non-autoregressive character-level wav2vec2-CTC acoustic judge, used strictly at training and absent at inference, where a single model decodes greedily. Trained on LibriSpeech and evaluated across a six-tier difficulty gradient including real AMI meeting speech (33,282 utterance-condition instances), the method reduces insertion errors by 28.3% on close-talking AMI-IHM and 22.3% on far-field AMI-SDM, while lowering WER on AMI-SDM from 35.89% to 34.71% and showing no detectable WER difference on the other five tiers, against a schedule-matched WER-GRPO baseline. The insertion reduction holds under a meeting-level clustered bootstrap. Four prespecified analyses support content-conditioned insertion calibration: output collapses 85-90% on unintelligible audio that preserves energy and voice activity; the gain is not recovered by the evaluated 32-best CTC rescoring configuration, yet RL internalizes it into a single greedy decoding run; and policy-only confidence yields lower insertion-AURC in all four evaluated settings. We frame this as a mechanism paper, demonstrated in one instantiation: a 7B speech LLM with a 0.3B CTC judge.
Primary: Tencent
All Institutions: Tencent
The paper introduces a frozen acoustic judge to reinforce ASR policies against insertion hallucinations. It demonstrates that incorporating acoustic fidelity into the RL reward function effectively reduces ungrounded insertions in noisy and far-field conditions without compromising overall WER, offering a robust solution for LLM-based speech recognition.
The paper proposes a reinforcement learning (RL) post-training framework for ASR that augments the standard Word Error Rate (WER) reward with an "acoustic-fidelity" reward. This reward is generated by a frozen, non-autoregressive wav2vec2-CTC model that scores the acoustic consistency of the policy's generated transcriptions against the input audio. The core innovation is the use of an independent, frozen judge to penalize hallucinations (specifically insertions) during training, thereby forcing the policy to ground its outputs in acoustic evidence rather than relying solely on language priors. The method is applied to a 7B speech LLM (Qwen2-Audio) using Group Relative Policy Optimization (GRPO). The approach is conceptually sound, addressing a known failure mode in LLM-based ASR where models generate fluent but ungrounded text when acoustic signals are weak. The use of a character-level CTC judge is a practical choice, avoiding the need for a complex language model in the reward function.
The experimental setup is rigorous, featuring a six-tier difficulty gradient ranging from clean LibriSpeech to noisy conditions and real-world AMI meeting speech (both close-talking and far-field). The evaluation focuses on insertion errors, which is the specific target of the proposed reward. The results show a significant reduction in insertion errors (28.3% on AMI-IHM, 22.3% on AMI-SDM) without a significant increase in WER on other tiers. The paper includes four prespecified analyses: a counterfactual probe with unintelligible audio, a comparison with CTC rescoring, a judge-free selective ASR analysis, and a cost-sensitive WER analysis. These analyses provide strong evidence that the model is learning to suppress insertions based on acoustic content rather than just energy detection. The use of paired bootstrap confidence intervals adds statistical rigor.
The paper provides detailed implementation specifics, including the exact models used (Qwen2-Audio-7B, wav2vec2-large-960h), hyperparameters (LoRA rank, learning rate, batch size), and the specific reward formulation. The data splits and augmentation strategies are clearly described. However, the code is not explicitly linked in the provided text, and the specific version of vLLM and the LoRA merging process are mentioned as implementation details that might be tricky to reproduce exactly without the authors' code. The "frozen judge" setup is straightforward to implement given the open-source nature of the components.
The primary limitation is the scope of the evaluation, which is limited to one policy model (Qwen2-Audio-7B) and one judge model (wav2vec2-CTC). The paper acknowledges that judge swaps (e.g., with HuBERT or Whisper) were not performed. Additionally, the method does not address deletion errors, which can increase as a trade-off for reducing insertions, although the paper notes this cost is low. The quadratic normalization in the CTC score is an implementation detail that the authors acknowledge as a potential source of bias, though they argue it is mitigated by group normalization. The lack of a no-RL baseline in the counterfactual analysis limits the ability to attribute all grounding behavior to the RL process itself.
This work has significant implications for the deployment of LLM-based ASR systems in real-world, noisy environments. By providing a training-time mechanism to curb hallucinations without adding inference-time latency or complexity, it offers a practical path to more reliable speech recognition. The concept of using an independent acoustic judge for RL rewards could be extended to other multimodal tasks where grounding in sensory input is critical. It also highlights the importance of reward design in RL for generative models, moving beyond simple text-matching metrics to include modality-specific fidelity checks. The paper introduces a frozen acoustic judge to reinforce ASR policies against insertion hallucinations. It demonstrates that incorporating acoustic fidelity into the RL reward function effectively reduces ungrounded insertions in noisy and far-field conditions without compromising overall WER, offering a robust solution for LLM-based speech recognition.
Instruction-guided music editors typically process each request independently, limiting their ability to support workflows in which users progressively refine a track. We introduce AURA, a unified multimodal framework for conversational music editing. AURA uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens. A concept-to-audio module injects these tokens and frame-aligned reference features into a frozen MusicGen backbone, enabling precise edits while preserving unaffected content. AURA optimizes only 91M parameters while retaining 1.9B frozen backbone parameters. Experiments on Slakh2100 and MoisesDB demonstrate substantial improvements in edit correctness and content preservation over existing instruction-guided methods, including a 4-5 times reduction in FAD for out-of-domain addition and removal.
Primary: Aalto University
All Institutions: Aalto University, Technical University of Denmark, University of South Dakota, OpenRB Lab
AURA introduces a unified multimodal framework for conversational music editing that uses compact concept tokens from a frozen MLLM to condition a frozen MusicGen backbone, achieving significant improvements in edit correctness and content preservation while maintaining high parameter efficiency.
The paper proposes AURA, a framework that effectively decouples multimodal intent understanding from audio generation. The core innovation lies in the "Concept-to-Audio" (C2A) module and the use of compact concept tokens generated by a frozen Multimodal LLM (Qwen2.5-Omni). By projecting these tokens into the conditioning space of a frozen MusicGen backbone, the authors avoid the instability and high computational cost of fine-tuning large generative models. The use of a Bidirectional Feature Alignment Module (BiFAM) with FiLM modulation to inject frame-aligned reference audio features is a technically sound approach to preserving unedited content, as it allows the decoder to attend to the reference structure without appending tokens to the autoregressive sequence. The parameter efficiency (91M trainable vs 1.9B frozen) is a significant practical advantage.
The experiments are comprehensive, covering both in-domain (Slakh2100) and out-of-domain (MoisesDB) benchmarks. The metrics chosen are appropriate for this task, including FAD, CLAP, KL divergence, SSIM, and SI-SDR. The results show substantial improvements in content preservation (SSIM and SI-SDR) over baselines like M2UGen, Instruct-MusicGen, and LeVo. Specifically, the multi-turn editing experiments demonstrate that AURA handles conversational context better than single-turn baselines, which suffer from error accumulation. The ablation study confirms the utility of concept tokens over full-text conditioning. However, the paper lacks subjective human evaluation (MOS/ABX), which is standard in audio generation research to validate objective metrics.
The paper provides sufficient architectural details, including the specific MLLM used, the number of concept tokens, LoRA ranks, and training hyperparameters. The use of standard open-source components (MusicGen, EnCodec, Qwen2.5-Omni) enhances reproducibility. However, the code repository is not explicitly linked in the text provided (only a demo page), which may hinder immediate reproduction.
The primary limitation acknowledged is the performance in stem extraction tasks, where the generative approach lags behind dedicated separation models. Additionally, the reliance on a specific MLLM (Qwen2.5-Omni) may introduce biases or limitations inherent to that model's audio understanding capabilities. The lack of subjective evaluation is a notable gap.
This work contributes to the field of controllable audio generation by introducing a scalable, parameter-efficient method for conversational editing. It bridges the gap between high-level language understanding and low-level audio synthesis, offering a viable path for interactive music production tools. The framework's design could be extended to other audio editing tasks or modalities. AURA introduces a unified multimodal framework for conversational music editing that uses compact concept tokens from a frozen MLLM to condition a frozen MusicGen backbone, achieving significant improvements in edit correctness and content preservation while maintaining high parameter efficiency.