Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.
Primary: Meta AI
All Institutions: Meta AI
SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
The paper proposes a unified framework for song generation and editing based on "reconstructive pretraining." The core idea is to train a latent flow matching model (DiT-based) to reconstruct audio from a set of interpretable conditions. By varying the density of these conditions (sparse for generation, dense for editing), the model learns both to generate diverse songs and to perform fine-grained edits by modifying specific attributes while keeping others fixed. Key technical contributions include: (1) Word-level phoneme alignment, which fuses text and phoneme embeddings at word boundaries to improve pronunciation learning and convergence; (2) Beat conditioning, providing explicit beat/downbeat positions to improve musicality; and (3) Extending Representation Alignment (REPA) to the VAE latent space, aligning VAE latents with MusicFM features to create semantically meaningful latent representations. The editing mechanism relies on a residual encoder with a bottleneck capacity that controls the trade-off between reconstruction fidelity and editability. This approach avoids the need for paired editing data or noise injection/inversion techniques common in prior work.
The experiments are comprehensive, covering generation quality (WER, A4, SongEval), subjective evaluation (MOS), and editing capabilities. SongCraft achieves the lowest Word Error Rate (0.133) among baselines, significantly outperforming DiffRhythm (0.588) and YuE (0.414). It also achieves competitive perceptual quality scores, ranking second only to Levo (which uses DPO). The subjective evaluation confirms these trends, with SongCraft scoring highest in 5 out of 6 dimensions. The study of the residual encoder bottleneck provides valuable insights into the reconstruction-editability trade-off, showing non-monotonic behavior where moderate bottleneck sizes (M=15) yield the best balance. Ablations validate the contribution of each proposed component (phoneme alignment, beat conditioning, REPA).
The paper provides detailed implementation specifics, including model architecture (4.2B parameters, 36 layers), training hyperparameters (learning rate, batch size, GPU count), and data preprocessing steps. However, the use of an "internal dataset" limits full reproducibility for external researchers. The reliance on specific open-source tools for feature extraction (Whisper, Phonemizer, Beat This, BTC, SWIFT-F0, 3D-Speaker) is clearly documented, which aids in replicating the conditioning pipeline. The code and model weights are not explicitly linked in the provided text, though the institution (Meta AI) often releases such assets.
The primary limitation is the use of a proprietary internal dataset, which hinders independent verification and benchmarking. The model is large (4.2B parameters), requiring significant computational resources for training and inference. The editing capabilities, while novel, are limited to the specific attributes extracted (lyrics, melody, beats, speaker identity, chords) and may not generalize to arbitrary semantic edits without further conditioning. The non-monotonic behavior of the residual encoder bottleneck suggests that tuning this parameter is critical and potentially non-trivial for new datasets or domains.
This work advances the field of controllable music generation by unifying generation and editing in a single model without requiring paired data. The reconstructive pretraining paradigm could be applied to other audio domains (speech, sound effects) for controllable synthesis and editing. The integration of semantic alignment (REPA) into the VAE latent space offers a generalizable technique for improving the quality of latent diffusion/flow matching models. The ability to perform fine-grained edits (e.g., changing lyrics while preserving melody and timbre) has significant applications in music production, prototyping, and personalized content creation. SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Primary: StepFun
All Institutions: StepFun
The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
The paper proposes a unified discrete autoregressive framework for general audio generation, departing from the dominant diffusion/flow-matching paradigm. The core architectural contribution is the "time-depth" decomposition: the LLM backbone predicts the first (semantic) RVQ codebook along the time axis, while a lightweight causal Transformer predicts the remaining 15 residual codebooks along the depth axis. This design effectively isolates the heavy acoustic modeling from the long-range linguistic planning, preventing the acoustic loss from overwhelming the LLM's text capabilities. The introduction of the "RVQ Adaptor," a zero-initialized residual module, is a clever engineering solution to align multi-codebook audio embeddings with the pretrained text embedding space without disrupting the backbone. The four-stage progressive pretraining curriculum, specifically the use of stop-gradient detachment for the residual predictor during the main generation phase, is a sophisticated approach to mitigating catastrophic forgetting and interference.
The evaluation is robust, covering both component-level ablations (RVQ Adaptor, progressive pretraining) and end-to-end generation quality. The ablation studies convincingly demonstrate that the RVQ Adaptor significantly improves audio understanding metrics (ASR, Translation) and that the progressive pretraining preserves text capabilities (MMLU, HumanEval) far better than a baseline. The TTS evaluation uses an Arena-style pairwise comparison with Elo ratings, showing strong performance against commercial systems. The Voice Design evaluation utilizes the InstructTTSEval benchmark with LLM-based judging (Gemini) and human preference testing, demonstrating state-of-the-art instruction following. The inclusion of subjective human evaluation for "human-likeness" is a strong point, as objective metrics often fail to capture prosodic naturalness.
The paper provides high-level architectural details and training recipes (learning rates, batch sizes, data mixtures) but lacks specific implementation details for the tokenizer training (e.g., specific GAN hyperparameters, discriminator architectures beyond "multi-period") and the exact composition of the 2.7T token pretraining corpus. The reliance on proprietary internal models (StepAudio R1.5, StepAudio Music Model) for data generation and evaluation limits full external reproducibility. However, the conceptual framework is clear enough for replication in principle.
The paper focuses heavily on TTS and Voice Design in its quantitative evaluation. While it claims capabilities in music, sound effects, and vocals, the rigorous benchmarking for these domains is less detailed compared to speech. The "Vibe Speech" and complex scene generation capabilities are demonstrated qualitatively but lack objective metrics. The model's performance on long-form generation (beyond the 44-minute context window mentioned) is not extensively tested. Additionally, the reliance on LLMs for reward modeling in RL (GRPO) introduces potential biases from the judge model.
This work significantly advances the field of unified audio generation by proving that discrete autoregressive models can compete with or surpass continuous diffusion models in speech tasks while offering better integration with LLMs. The "interference-aware" training strategy is a valuable lesson for the broader community of multimodal LLMs, providing a blueprint for adding new modalities without degrading existing capabilities. The unified instruction format (ROLE/DIRECTOR/SCRIPT) offers a practical standard for controlling complex audio scenes, which could influence future API designs and application development. The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
Primary: Alibaba Group
All Institutions: Alibaba Group, Shanghai Jiao Tong University
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
The paper introduces DiffSynth-Music, a framework for controllable music generation built upon a frozen Diffusion Transformer (DiT) backbone (ACE-Step-1.5-XL-SFT). The core methodology involves injecting layer-wise key-value (KV) representations derived from audio conditioning signals into the generation branch. This approach is conceptually related to adapter-based methods like ControlNet but is tailored for audio and uses a shared VAE to map diverse conditioning waveforms into a common latent space. A key strength is the design of three template models (Control, Prosody, Reference) that support five distinct control types: beats, vocals, accompaniment, prosody, and reference audio. The detailed construction of these control signals from raw music recordings (e.g., beat extraction, source separation, vocal resynthesis for prosody, loudest excerpt for reference) is well-described and practical. The prosody generation method, which uses a sinusoidal carrier with estimated pitch and envelope to reduce phonetic/timbral cues, is a clever way to achieve melodic control without enforcing specific vocal identity. The KV-injection mechanism, where control keys and values are concatenated with the generation branch's keys and values, allows for composable conditioning. The efficiency gain from computing each control cache once at a fixed template timestep (clean-data endpoint) and reusing it throughout sampling is a significant practical advantage, reducing computational overhead. The training formulation uses conditional flow matching, which is a suitable choice for learning continuous-time vector fields, and the adapter-only optimization ensures the pretrained backbone remains frozen, preserving its general generation capabilities. The overall architecture is sound, well-motivated, and addresses a clear need in controllable music generation.
The experimental evaluation is comprehensive and well-structured. The authors evaluate DiffSynth-Music on both Mandarin and English songs, demonstrating its applicability across different languages, which is a strong point. The comparison against several state-of-the-art music generation systems (ACE-Step-1.5-XL-SFT, DiffRhythm-2, HeartMuLa-3B, MiniMax-Music3, LeVo-2-Large) provides a robust baseline. Critically, the backbone model (ACE-Step-1.5-XL-SFT) is included as a baseline to specifically highlight the improvements gained by adding the audio-conditioned templates. The evaluation metrics are appropriate and cover both control adherence and general music quality. For control adherence, metrics like Beat-F1, Cemgil (beats), V-MSE, PER (vocals), A-MSE (accompaniment), Pitch_50 (prosody), and MuLan-A (reference) are used. The results clearly show that DiffSynth-Music significantly outperforms all baselines across all seven control adherence metrics, often by a large margin (e.g., Beat-F1 from ~0.3 to ~0.85, Pitch_50 from ~0.04 to ~0.47). This strongly validates the effectiveness of the proposed conditioning framework. To assess general music quality, AudioBox-Aesthetics scores (CE, CU, PC, PQ) and MuLan-T (text-music alignment) are used. The results indicate that DiffSynth-Music maintains "broadly comparable" scores to the evaluated base models, with some metric-specific trade-offs. While some scores are slightly lower than the backbone, the overall message is that the added controllability does not severely degrade the general generation quality, which is a reasonable and expected outcome for such an adapter-based system. The experimental setup and results convincingly support the paper's claims.
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
Long-form text-to-speech (TTS) enables multi-turn conversations with consistent prosody and higher quality voice cloning from longer reference audio. Recent open-weights autoregressive TTS models such as Qwen3-TTS and VoxCPM2 attain state-of-the-art word error rate (WER) and speaker similarity (SIM) on short-form prompts but significantly deteriorate when used with long-form prompts. We propose Localized Attention-Constrained Inference (LACI), an inference-only method to detect TTS errors in near real-time, roll back to the error onset and regenerate with temporary guardrails, adding negligible computational overhead. Using LACI, we improve worst-of-N WER across 10 RNG seeds for Qwen3-TTS-0.6B from 35.2% to 3.4% on prompts longer than 1500 words, even surpassing its short-form reliability of 5.4\% on prompts with fewer than 500 words. To demonstrate the efficacy of LACI on voice cloning reliability, we propose a sliding-window version of the SIM metric that we call wSIM. wSIM exposes several novel failure patterns that are not captured by SIM. LACI improves worst-of-N wSIM from 0.01 to 0.47 on 120 seconds of reference audio while reducing the rate of catastrophic generations with WER above 30% from 26% to below 1%
Primary: Argmax, Inc.
All Institutions: Argmax, Inc., University of Virginia, Bilkent University
LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
The paper proposes Localized Attention-Constrained Inference (LACI), an inference-only technique to address reliability issues in long-form autoregressive TTS. The core insight is identifying "emergent alignment heads" within the model's attention mechanism that track the mapping between input text tokens and generated audio tokens. The authors define two specific failure modes: "skips" (deletions) and "hallucinations" (insertions/substitutions). LACI monitors the attention weights of these specific heads in real-time. If the attention pattern deviates from the expected monotonic progress (e.g., skipping a bin of text tokens or stalling), the system detects the error, rolls back the generation state to the error onset, and applies a temporary hard attention mask (guardrail) to force correct alignment during regeneration. This is a clever, low-overhead approach that leverages internal model states rather than external ASR feedback for detection, which is a significant methodological improvement over standard best-of-N sampling.
The experiments are rigorous and well-targeted. The authors use the AppTek Call Center dataset, a real-world long-form TTS use case, which is more relevant than standard short-utterance benchmarks for this specific problem. They evaluate on multiple model scales (Qwen3-TTS-0.6B, 1.7B) and families (VoxCPM2), demonstrating generalizability. The metrics include standard WER and SIM, but the introduction of wSIM (sliding-window speaker similarity) is a valuable contribution that exposes failure patterns hidden by global averaging. The results show dramatic improvements in worst-of-N WER (35.2% to 3.4%) and wSIM (0.01 to 0.47), effectively closing the gap between short-form and long-form reliability. The comparison against ACI (always-on guardrails) clearly demonstrates the superiority of the detection-and-recovery approach.
The paper states that code will be open-sourced upon publication, which is standard for arXiv preprints. However, the specific details of the "instability measure" and detector settings are referenced as being in the code, which is a slight barrier to immediate reproducibility without access to the repository. The use of standard open-weights models (Qwen3-TTS, VoxCPM2) and public datasets (AppTek) aids reproducibility. The manual verification of wSIM failures adds a layer of qualitative validation that is hard to automate but important for trust.
The method relies on the existence of "emergent alignment heads," which may not be present or stable in all TTS architectures, particularly non-autoregressive or diffusion-based models. The paper focuses on autoregressive neural codec language models. The computational overhead, while described as "negligible," involves rollback and regeneration, which could be significant for very long texts if failures are frequent. The evaluation is limited to English (implied by the datasets and models used). The "worst-of-N" metric, while useful for reliability, does not reflect the average user experience if N is large, though the paper argues for the importance of reliability in professional settings.
This work has high practical impact for deploying TTS in real-world applications like audiobooks, customer support agents, and long-form voice cloning. The ability to reliably generate long-form speech with consistent speaker identity is a critical bottleneck for current TTS systems. The proposed wSIM metric is a valuable tool for the community to better evaluate speaker consistency in long-form generation. The inference-only nature of the solution makes it easily adoptable by users of existing open-weights models without retraining. LACI is an inference-only method that detects and repairs long-form TTS errors by monitoring emergent attention alignment heads, significantly improving reliability and voice cloning quality. The paper provides a rigorous analysis of failure modes in autoregressive TTS and offers a practical, low-overhead solution that generalizes across model sizes and families, with the introduction of wSIM as a more sensitive metric for speaker consistency.
Song generation and editing have mostly been treated as separate tasks. Existing editing methods often require noise injection and regeneration or curated paired training data. We propose a unified approach for song generation and editing based on reconstructive pretraining, in which a model is trained to reconstruct audio from varying numbers of interpretable conditions. With conditions such as text and lyrics, the model learns to generate diverse songs. With dense conditions specifying fine-grained music attributes, the model learns to reconstruct the target and enables editing by modifying any single attribute while keeping others fixed. This leads to SongCraft, a latent flow matching based model trained for both generation and fine-grained editing. To improve song generation quality, we further introduce word-level phoneme alignment that improves pronunciation learning and accelerates convergence, beat conditioning that improves general musicality, and representation alignment on VAE latent space that produces semantically meaningful latents for improved generation quality. Experiments show that SongCraft achieves the lowest word error rate among evaluated song generation baselines while maintaining competitive audio quality. We further show that a single model can support editing of lyrics, vocal melody, beats, and singer identity, and we also study the trade-off between reconstruction quality and editability.
Primary: Meta AI
All Institutions: Meta AI
SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
The paper proposes a unified framework for song generation and editing based on "reconstructive pretraining." The core idea is to train a latent flow matching model (DiT-based) to reconstruct audio from a set of interpretable conditions. By varying the density of these conditions (sparse for generation, dense for editing), the model learns both to generate diverse songs and to perform fine-grained edits by modifying specific attributes while keeping others fixed. Key technical contributions include: (1) Word-level phoneme alignment, which fuses text and phoneme embeddings at word boundaries to improve pronunciation learning and convergence; (2) Beat conditioning, providing explicit beat/downbeat positions to improve musicality; and (3) Extending Representation Alignment (REPA) to the VAE latent space, aligning VAE latents with MusicFM features to create semantically meaningful latent representations. The editing mechanism relies on a residual encoder with a bottleneck capacity that controls the trade-off between reconstruction fidelity and editability. This approach avoids the need for paired editing data or noise injection/inversion techniques common in prior work.
The experiments are comprehensive, covering generation quality (WER, A4, SongEval), subjective evaluation (MOS), and editing capabilities. SongCraft achieves the lowest Word Error Rate (0.133) among baselines, significantly outperforming DiffRhythm (0.588) and YuE (0.414). It also achieves competitive perceptual quality scores, ranking second only to Levo (which uses DPO). The subjective evaluation confirms these trends, with SongCraft scoring highest in 5 out of 6 dimensions. The study of the residual encoder bottleneck provides valuable insights into the reconstruction-editability trade-off, showing non-monotonic behavior where moderate bottleneck sizes (M=15) yield the best balance. Ablations validate the contribution of each proposed component (phoneme alignment, beat conditioning, REPA).
The paper provides detailed implementation specifics, including model architecture (4.2B parameters, 36 layers), training hyperparameters (learning rate, batch size, GPU count), and data preprocessing steps. However, the use of an "internal dataset" limits full reproducibility for external researchers. The reliance on specific open-source tools for feature extraction (Whisper, Phonemizer, Beat This, BTC, SWIFT-F0, 3D-Speaker) is clearly documented, which aids in replicating the conditioning pipeline. The code and model weights are not explicitly linked in the provided text, though the institution (Meta AI) often releases such assets.
The primary limitation is the use of a proprietary internal dataset, which hinders independent verification and benchmarking. The model is large (4.2B parameters), requiring significant computational resources for training and inference. The editing capabilities, while novel, are limited to the specific attributes extracted (lyrics, melody, beats, speaker identity, chords) and may not generalize to arbitrary semantic edits without further conditioning. The non-monotonic behavior of the residual encoder bottleneck suggests that tuning this parameter is critical and potentially non-trivial for new datasets or domains.
This work advances the field of controllable music generation by unifying generation and editing in a single model without requiring paired data. The reconstructive pretraining paradigm could be applied to other audio domains (speech, sound effects) for controllable synthesis and editing. The integration of semantic alignment (REPA) into the VAE latent space offers a generalizable technique for improving the quality of latent diffusion/flow matching models. The ability to perform fine-grained edits (e.g., changing lyrics while preserving melody and timbre) has significant applications in music production, prototyping, and personalized content creation. SongCraft unifies song generation and fine-grained editing through reconstructive pretraining with a latent flow matching model. The paper introduces a novel conditioning framework using sparse conditions for generation and dense interpretable attributes for editing, eliminating the need for paired data or noise injection. Technical innovations include word-level phoneme alignment for improved intelligibility, beat conditioning for musicality, and extending REPA to the VAE latent space for better semantic structure. Experiments demonstrate state-of-the-art word error rate and competitive perceptual quality, along with a systematic analysis of the reconstruction-editability trade-off via residual encoder bottleneck capacity.
Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception, particularly in precise event localization. Existing approaches primarily post-train LALMs to predict event boundaries as timestamp tokens. However, this generative formulation lacks explicit correspondence between the timestamp predictions and fine-grained acoustic evidence, limiting the precision and reliability of temporal localization. To address this issue, we augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query. Specifically, the frozen LALM encodes the event query with audio as context, and the grounding model combines these query representations with fine-grained audio features to localize the target event at the frame level. Extensive experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods. Further evaluation shows that the grounding model can provide temporal evidence to support downstream reasoning.
Primary: University of Science and Technology of China
All Institutions: University of Science and Technology of China, Singapore Institute of Technology
The paper proposes a modular framework that augments frozen Large Audio-Language Models with a dedicated frame-level grounding model to achieve precise temporal localization. By decoupling semantic query representation from acoustic event detection and using cross-attention for frame-level alignment, the method significantly outperforms existing generative and specialized baselines across multiple benchmarks, while also enhancing downstream temporal reasoning capabilities when used as an external tool.
The paper proposes a modular framework that decouples semantic query understanding from fine-grained temporal localization. Instead of post-training the Large Audio-Language Model (LALM) to generate timestamp tokens (which is error-prone and lacks explicit acoustic grounding), the authors freeze the LALM and use its hidden states as semantic queries. A separate, lightweight grounding model uses cross-attention to align these semantic queries with frame-level audio features from a specialized encoder (ATST-Frame) and a CNN branch. This allows the model to predict frame-level activity scores directly, which are then converted to temporal intervals. The methodology is sound, leveraging the strengths of LALMs (semantic reasoning) and specialized audio encoders (temporal precision) while mitigating their respective weaknesses. The addition of a "tool-use" mechanism for downstream reasoning is a clever architectural choice that enhances the utility of the grounding model.
The experimental evaluation is comprehensive, covering five diverse benchmarks (AudioGrounding, DESED, UnAV-100, TACOS, Clotho-Moment). The results show consistent and significant improvements over strong baselines, including specialized grounding models (SpotSound, AM-DETR) and recent LALMs (Qwen3-Omni, Audio Flamingo 3). The gains are particularly notable at stricter IoU thresholds ([email protected]), validating the claim of improved precision. The ablation studies effectively isolate the contributions of audio context in the LALM, the choice of audio encoder, and the frame-level alignment mechanism. The downstream reasoning experiments on DCASE 2025 Task 5 further demonstrate the practical value of the grounding model as a tool for temporal reasoning.
The paper provides sufficient implementation details for reproducibility, including specific hyperparameters (learning rate, batch size, number of epochs, attention heads), architectural choices (Conformer decoder, CNN branch), and preprocessing steps (10s windows, 50Hz upsampling). The use of publicly available checkpoints for baselines and open-source models for the backbone (Qwen3-Omni, Audio Flamingo 3) facilitates reproduction. However, the specific code repository is not linked in the provided text, which is a minor drawback.
The framework relies on a frozen LALM, which means it cannot adapt to new semantic concepts or out-of-distribution queries without retraining the grounding model's alignment. The 10-second windowing strategy may limit performance on very long-form audio or events spanning multiple windows. The evaluation is limited to English audio and specific benchmarks; generalization to other languages or more complex, multi-event scenarios is not fully explored. The "tool-use" reasoning approach adds inference latency and complexity.
This work addresses a critical bottleneck in audio AI: the lack of precise temporal grounding in large multimodal models. By providing a modular solution that can be attached to existing LALMs, it offers a practical path for enhancing temporal perception in audio understanding systems. This has implications for applications requiring precise event localization, such as video editing, audio surveillance, and interactive audio assistants. The demonstrated improvement in downstream reasoning suggests that explicit temporal evidence can significantly boost the reliability of LALM-based reasoning tasks. The paper proposes a modular framework that augments frozen Large Audio-Language Models with a dedicated frame-level grounding model to achieve precise temporal localization. By decoupling semantic query representation from acoustic event detection and using cross-attention for frame-level alignment, the method significantly outperforms existing generative and specialized baselines across multiple benchmarks, while also enhancing downstream temporal reasoning capabilities when used as an external tool.
What did an ancestral bird species sound like? Existing ancestral state reconstruction methods can infer low-dimensional traits such as morphological characters at internal nodes of a phylogenetic tree, but no one has tried to produce rich perceptual signals such as audio. Some of the challenges include inferred representations that are either too low-dimensional to decode or lie in non-generative feature spaces, so no method to date can produce ancestral audio. We introduce the first framework that generates plausible ancestral vocalizations. Our pipeline encodes bird recordings into a VAE latent space, learns a low-dimensional trait projection aligned with phylogenetic distances, performs ancestral inference in this trait space, and recovers decodable latents through an anchored inverse lift before emitting novel waveforms for each ancestral node. Because the entire pipeline stays within a decodable latent space, every internal node receives a genuinely new audio output representing plausible intermediate ancestral sounds unavailable to retrieval-based alternatives. Experiments on two phylogenetically distant bird clades, 21-species Tyrannidae and 19-species Paridae, show that our method is the only approach that simultaneously achieves genuine generation, phylogenetic consistency, and naturalistic audio quality across both datasets.
Primary: University of Wisconsin-Madison
All Institutions: University of Wisconsin-Madison, Massachusetts Institute of Technology
The paper introduces the first framework for generating plausible ancestral vocalizations by combining phylogenetic inference with generative audio models. It effectively addresses the challenge of reconstructing high-dimensional perceptual signals on phylogenies by using a learned trait subspace and an anchored inverse lift to ensure decodable, naturalistic audio outputs that are phylogenetically consistent.
The paper proposes a novel four-stage pipeline for ancestral sound reconstruction: encoding audio into a VAE latent space, learning a low-dimensional tree-metric trait projection, performing Brownian motion ancestral inference in this trait space, and recovering decodable latents via an anchored inverse lift. The key technical contribution is the "anchored inverse lift," which resolves the underdetermined inverse problem of mapping from a low-dimensional trait space back to the high-dimensional VAE latent space by preserving the nullspace component (acoustic texture) from a nearby observed latent. This is mathematically grounded with proofs for exact trait preservation and nullspace preservation. The use of Cholesky reparametrization for the Stiefel manifold constraint is a solid optimization choice.
The experiments are conducted on two distinct bird clades (Tyrannidae and Paridae) to test generalization across suboscine (innate) and oscine (learned) vocalizations. The evaluation metrics are well-chosen for a task without ground truth, focusing on phylogenetic plausibility (Tree-Embedding Correlation, Edge Monotonicity) and audio quality (FAD, Silence Rate). The results clearly demonstrate that the proposed method outperforms naive latent-space BM in audio quality and retrieval baselines in novelty (Unique Rate). The ablation study effectively isolates the contribution of the learned projection and the anchoring strategy.
The paper provides detailed algorithmic descriptions, hyperparameters, and dataset curation steps. It specifies the use of pretrained Make-An-Audio 2 VAE and BigVGAN vocoder, which are publicly available. However, the specific code for the trait learning and ancestral inference pipeline is not linked in the provided text, which slightly hinders immediate reproducibility, though the mathematical formulation is clear enough for re-implementation.
The primary limitation is the lack of ground truth for ancestral sounds, making evaluation inherently indirect. The method relies on the assumption that Brownian motion is a suitable model for vocal evolution, which may not hold for all traits or clades, although the paper does test alternative models (OU, Pagel's lambda) in the appendix. The anchor dependence means that fine-grained acoustic texture is copied from extant species rather than inferred, which could introduce bias if the nearest anchor is not representative.
This work bridges evolutionary biology and machine learning, offering a tool to visualize and analyze ancestral traits in a perceptual modality. It could impact comparative biology, conservation efforts (reconstructing extinct species' calls), and the understanding of vocal evolution and speciation. The framework is generalizable to other high-dimensional traits where phylogenetic signal exists but direct generation is challenging. The paper introduces the first framework for generating plausible ancestral vocalizations by combining phylogenetic inference with generative audio models. It effectively addresses the challenge of reconstructing high-dimensional perceptual signals on phylogenies by using a learned trait subspace and an anchored inverse lift to ensure decodable, naturalistic audio outputs that are phylogenetically consistent.
Autoregressive generation of interleaved text and acoustic tokens is a common approach to spoken-response generation in speech large language models. Although this design enables streaming generation with explicit textual guidance, generated acoustic tokens become part of the context for subsequent text predictions. Given identical speech inputs, we observe markedly lower answer accuracy for the internal text generated in speech-to-text-and-speech (S2TS) mode than for speech-to-text (S2T) responses. We term this discrepancy the \emph{output-mode gap} (OMG). To reduce OMG, we propose \emph{Joint-Output On-Policy Distillation} (JO-OPD), which distills the model's stronger S2T policy into joint generation using student-generated S2TS trajectories. At each text position, the S2T teacher provides soft targets from a text-only projection of the student's preceding outputs, while the student predicts from the corresponding full interleaved history. A preservation objective further regularizes native non-text predictions. Experiments on Step-Audio-2-mini and Baichuan-Audio-Instruct reveal OMG across two interleaved generation architectures. On Step-Audio-2-mini, JO-OPD reduces OMG from 42.87 to 16.26 percentage points on Spoken-MQA and from 29.72 to 13.04 points on speech-rendered GSM8K, with little change in S2T accuracy and substantially larger reductions than matched SFT baselines. ASR-based evaluation further shows a 7.49-point improvement in spoken-answer accuracy on Spoken-MQA.
Primary: Huawei (AI Lab, Leibniz Research Center)
All Institutions: AI Lab, Leibniz Research Center, Huawei, The Chinese University of Hong Kong, City University of Hong Kong
The paper identifies and quantifies the "Output-Mode Gap" in interleaved speech LLMs and proposes Joint-Output On-Policy Distillation (JO-OPD) to mitigate it by distilling the model's stronger text-only policy into the joint generation process. The technical contribution is significant as it provides a principled way to improve text reasoning accuracy in speech-output modes without sacrificing speech generation quality, supported by rigorous experiments on two major architectures and ablation studies validating the key components of the method.
The paper introduces Joint-Output On-Policy Distillation (JO-OPD) to address the "Output-Mode Gap" (OMG) in interleaved speech-language models. The core insight is that when a model generates both text and audio tokens, the acoustic tokens in the context degrade the quality of subsequent text predictions compared to a pure text-only mode. JO-OPD leverages the model's own stronger Speech-to-Text (S2T) policy as a teacher to supervise the Speech-to-Text-and-Speech (S2TS) student. A key technical contribution is the "text-only projection" of the student's history for the teacher, allowing the teacher to provide soft targets based on the semantic content without being confused by the acoustic noise in the context. Additionally, a preservation objective regularizes the non-text (acoustic) predictions to prevent the distillation process from degrading speech generation quality. The method is theoretically sound, leveraging on-policy distillation to align the student's distribution with the teacher's on the student's own trajectories, which is a robust approach for reducing exposure bias in autoregressive generation.
The experiments are conducted on two distinct architectures: Step-Audio-2-mini and Baichuan-Audio-Instruct. The evaluation metrics focus on answer accuracy for Spoken-MQA and GSM8K, comparing S2T accuracy vs. S2TS internal text accuracy. The results show a significant reduction in OMG (e.g., from 42.87 to 16.26 points on Spoken-MQA for Step-Audio-2). The paper also includes ablations on soft vs. hard targets, the importance of non-text preservation, and the effect of training scale. The inclusion of ASR-based evaluation for spoken answer accuracy provides a holistic view of the impact on final user-facing output. The comparison against Self-SFT and Response-SFT baselines is appropriate and demonstrates the superiority of the proposed on-policy distillation approach.
The paper provides detailed hyperparameters (learning rate, batch size, top-k values) and describes the training setup clearly. However, the specific implementation details of the "text-only projection" and the exact mechanism for the preservation loss in different architectures (Step-Audio vs. Baichuan) could benefit from more pseudocode or explicit mathematical definitions in the main text. The use of synthesized speech (flite) for training is noted, which is a practical detail for reproducibility, though it may limit generalization to diverse speaker characteristics.
The primary limitation is the reliance on the model's own S2T capability as the teacher; if the base model has weak S2T performance, the distillation benefit may be limited. The evaluation is primarily on question-answering and reasoning tasks, so the impact on open-ended conversational quality or prosody is less clear, although the preservation loss aims to mitigate this. The training data size (27k prompts) is relatively small, and the long-term stability of the model after such fine-tuning is not extensively explored.
This work addresses a critical issue in the deployment of speech LLMs: the degradation of reasoning capabilities when speech output is enabled. By providing a method to close this gap, the paper contributes to the development of more reliable and capable voice assistants. The technique of using internal modality policies as teachers is a generalizable insight that could be applied to other multimodal interleaved generation tasks. The paper identifies and quantifies the "Output-Mode Gap" in interleaved speech LLMs and proposes Joint-Output On-Policy Distillation (JO-OPD) to mitigate it by distilling the model's stronger text-only policy into the joint generation process. The technical contribution is significant as it provides a principled way to improve text reasoning accuracy in speech-output modes without sacrificing speech generation quality, supported by rigorous experiments on two major architectures and ablation studies validating the key components of the method.
When reinforcement learning (RL) is used for post-training automatic speech recognition (ASR), the reward almost always lives in the text space: it compares a hypothesis with the reference and never checks whether the hypothesis is supported by the audio. On highly regular speech this licenses a shortcut - guessing from a strong language prior rather than listening. Once the acoustics degrade, the shortcut runs unchecked and emits fluent but ungrounded words, i.e., insertion errors. We propose an acoustic-fidelity reward: a GRPO reward augmented with a separately pretrained, permanently frozen, non-autoregressive character-level wav2vec2-CTC acoustic judge, used strictly at training and absent at inference, where a single model decodes greedily. Trained on LibriSpeech and evaluated across a six-tier difficulty gradient including real AMI meeting speech (33,282 utterance-condition instances), the method reduces insertion errors by 28.3% on close-talking AMI-IHM and 22.3% on far-field AMI-SDM, while lowering WER on AMI-SDM from 35.89% to 34.71% and showing no detectable WER difference on the other five tiers, against a schedule-matched WER-GRPO baseline. The insertion reduction holds under a meeting-level clustered bootstrap. Four prespecified analyses support content-conditioned insertion calibration: output collapses 85-90% on unintelligible audio that preserves energy and voice activity; the gain is not recovered by the evaluated 32-best CTC rescoring configuration, yet RL internalizes it into a single greedy decoding run; and policy-only confidence yields lower insertion-AURC in all four evaluated settings. We frame this as a mechanism paper, demonstrated in one instantiation: a 7B speech LLM with a 0.3B CTC judge.
Primary: Tencent
All Institutions: Tencent
The paper introduces a frozen acoustic judge to reinforce ASR policies against insertion hallucinations. It demonstrates that incorporating acoustic fidelity into the RL reward function effectively reduces ungrounded insertions in noisy and far-field conditions without compromising overall WER, offering a robust solution for LLM-based speech recognition.
The paper proposes a reinforcement learning (RL) post-training framework for ASR that augments the standard Word Error Rate (WER) reward with an "acoustic-fidelity" reward. This reward is generated by a frozen, non-autoregressive wav2vec2-CTC model that scores the acoustic consistency of the policy's generated transcriptions against the input audio. The core innovation is the use of an independent, frozen judge to penalize hallucinations (specifically insertions) during training, thereby forcing the policy to ground its outputs in acoustic evidence rather than relying solely on language priors. The method is applied to a 7B speech LLM (Qwen2-Audio) using Group Relative Policy Optimization (GRPO). The approach is conceptually sound, addressing a known failure mode in LLM-based ASR where models generate fluent but ungrounded text when acoustic signals are weak. The use of a character-level CTC judge is a practical choice, avoiding the need for a complex language model in the reward function.
The experimental setup is rigorous, featuring a six-tier difficulty gradient ranging from clean LibriSpeech to noisy conditions and real-world AMI meeting speech (both close-talking and far-field). The evaluation focuses on insertion errors, which is the specific target of the proposed reward. The results show a significant reduction in insertion errors (28.3% on AMI-IHM, 22.3% on AMI-SDM) without a significant increase in WER on other tiers. The paper includes four prespecified analyses: a counterfactual probe with unintelligible audio, a comparison with CTC rescoring, a judge-free selective ASR analysis, and a cost-sensitive WER analysis. These analyses provide strong evidence that the model is learning to suppress insertions based on acoustic content rather than just energy detection. The use of paired bootstrap confidence intervals adds statistical rigor.
The paper provides detailed implementation specifics, including the exact models used (Qwen2-Audio-7B, wav2vec2-large-960h), hyperparameters (LoRA rank, learning rate, batch size), and the specific reward formulation. The data splits and augmentation strategies are clearly described. However, the code is not explicitly linked in the provided text, and the specific version of vLLM and the LoRA merging process are mentioned as implementation details that might be tricky to reproduce exactly without the authors' code. The "frozen judge" setup is straightforward to implement given the open-source nature of the components.
The primary limitation is the scope of the evaluation, which is limited to one policy model (Qwen2-Audio-7B) and one judge model (wav2vec2-CTC). The paper acknowledges that judge swaps (e.g., with HuBERT or Whisper) were not performed. Additionally, the method does not address deletion errors, which can increase as a trade-off for reducing insertions, although the paper notes this cost is low. The quadratic normalization in the CTC score is an implementation detail that the authors acknowledge as a potential source of bias, though they argue it is mitigated by group normalization. The lack of a no-RL baseline in the counterfactual analysis limits the ability to attribute all grounding behavior to the RL process itself.
This work has significant implications for the deployment of LLM-based ASR systems in real-world, noisy environments. By providing a training-time mechanism to curb hallucinations without adding inference-time latency or complexity, it offers a practical path to more reliable speech recognition. The concept of using an independent acoustic judge for RL rewards could be extended to other multimodal tasks where grounding in sensory input is critical. It also highlights the importance of reward design in RL for generative models, moving beyond simple text-matching metrics to include modality-specific fidelity checks. The paper introduces a frozen acoustic judge to reinforce ASR policies against insertion hallucinations. It demonstrates that incorporating acoustic fidelity into the RL reward function effectively reduces ungrounded insertions in noisy and far-field conditions without compromising overall WER, offering a robust solution for LLM-based speech recognition.
Instruction-guided music editors typically process each request independently, limiting their ability to support workflows in which users progressively refine a track. We introduce AURA, a unified multimodal framework for conversational music editing. AURA uses a multimodal large language model to interpret the complete dialogue history, an optional image, and reference audio, distilling the editing intent into compact concept tokens. A concept-to-audio module injects these tokens and frame-aligned reference features into a frozen MusicGen backbone, enabling precise edits while preserving unaffected content. AURA optimizes only 91M parameters while retaining 1.9B frozen backbone parameters. Experiments on Slakh2100 and MoisesDB demonstrate substantial improvements in edit correctness and content preservation over existing instruction-guided methods, including a 4-5 times reduction in FAD for out-of-domain addition and removal.
Primary: Aalto University
All Institutions: Aalto University, Technical University of Denmark, University of South Dakota, OpenRB Lab
AURA introduces a unified multimodal framework for conversational music editing that uses compact concept tokens from a frozen MLLM to condition a frozen MusicGen backbone, achieving significant improvements in edit correctness and content preservation while maintaining high parameter efficiency.
The paper proposes AURA, a framework that effectively decouples multimodal intent understanding from audio generation. The core innovation lies in the "Concept-to-Audio" (C2A) module and the use of compact concept tokens generated by a frozen Multimodal LLM (Qwen2.5-Omni). By projecting these tokens into the conditioning space of a frozen MusicGen backbone, the authors avoid the instability and high computational cost of fine-tuning large generative models. The use of a Bidirectional Feature Alignment Module (BiFAM) with FiLM modulation to inject frame-aligned reference audio features is a technically sound approach to preserving unedited content, as it allows the decoder to attend to the reference structure without appending tokens to the autoregressive sequence. The parameter efficiency (91M trainable vs 1.9B frozen) is a significant practical advantage.
The experiments are comprehensive, covering both in-domain (Slakh2100) and out-of-domain (MoisesDB) benchmarks. The metrics chosen are appropriate for this task, including FAD, CLAP, KL divergence, SSIM, and SI-SDR. The results show substantial improvements in content preservation (SSIM and SI-SDR) over baselines like M2UGen, Instruct-MusicGen, and LeVo. Specifically, the multi-turn editing experiments demonstrate that AURA handles conversational context better than single-turn baselines, which suffer from error accumulation. The ablation study confirms the utility of concept tokens over full-text conditioning. However, the paper lacks subjective human evaluation (MOS/ABX), which is standard in audio generation research to validate objective metrics.
The paper provides sufficient architectural details, including the specific MLLM used, the number of concept tokens, LoRA ranks, and training hyperparameters. The use of standard open-source components (MusicGen, EnCodec, Qwen2.5-Omni) enhances reproducibility. However, the code repository is not explicitly linked in the text provided (only a demo page), which may hinder immediate reproduction.
The primary limitation acknowledged is the performance in stem extraction tasks, where the generative approach lags behind dedicated separation models. Additionally, the reliance on a specific MLLM (Qwen2.5-Omni) may introduce biases or limitations inherent to that model's audio understanding capabilities. The lack of subjective evaluation is a notable gap.
This work contributes to the field of controllable audio generation by introducing a scalable, parameter-efficient method for conversational editing. It bridges the gap between high-level language understanding and low-level audio synthesis, offering a viable path for interactive music production tools. The framework's design could be extended to other audio editing tasks or modalities. AURA introduces a unified multimodal framework for conversational music editing that uses compact concept tokens from a frozen MLLM to condition a frozen MusicGen backbone, achieving significant improvements in edit correctness and content preservation while maintaining high parameter efficiency.
Realtime spoken interaction demands deep reasoning, prompt responses, and fluid turn-taking. We present StepAudio 3 Realtime, an audio-language foundation model organized around a continuous listen-converse-think-act loop. Deep Perception captures rich acoustic cues to interpret user intent, while Seamless Duplex models synchronized audio streams to handle pauses, backchannels, and interruptions naturally. Crucially, we resolve the tension between deep deliberation and latency via Think-While-Speaking, executing private reasoning in parallel with spoken delivery. In reasoning mode, StepAudio 3 reaches a 73.0 macro average on StepAudioChat. With Think-While-Speaking, it achieves dialogue and reasoning performance comparable to dedicated reasoning models while speaking in real time. Furthermore, an integrated Voice Agent handles asynchronous tool execution without disrupting the dialogue flow. StepAudio 3 Realtime achieves top-tier performance across key dimensions: an exceptional 90.6 on the MMSU benchmark, 98.9 Overall on the Artificial Analysis Full-Duplex Bench, and a 56.0% macro task-success rate on $τ$-Voice.
Primary: StepFun
All Institutions: StepFun
StepAudio 3 Realtime introduces a "Think-While-Speaking" architecture that enables parallel reasoning and speech generation, achieving state-of-the-art performance in full-duplex interaction and audio understanding while maintaining competitive reasoning capabilities. The paper demonstrates a robust system design that balances latency, intelligence, and conversational fluidity, marking a significant milestone in the development of real-time audio-language foundation models.
The paper proposes a comprehensive architecture for real-time spoken interaction, centered on a "listen-converse-think-act" loop. The core technical contribution is the "Think-While-Speaking" mechanism, which decouples the formulation of reasoning (Formulation Brain) from the articulation of speech (Articulation Brain), allowing the model to speak while continuing to reason in the background. This is supported by "Adaptive Thinking," a routing mechanism that determines whether a turn requires explicit reasoning based on a blind judge's evaluation of the necessity of thought, and Multi-Token Prediction (MTP) for accelerating private reasoning. The system also features "Seamless Duplex" for full-duplex interaction, handling interruptions and backchannels via time-interleaved audio blocks and state tokens. The model utilizes a Mixture-of-Experts (MoE) architecture with 196B total parameters (11B active) and integrates a Voice Agent for asynchronous tool execution. The methodology is sophisticated, combining architectural innovations with specific training strategies like model merging of specialized teachers to balance audio understanding, text reasoning, and dialogue capabilities.
The experimental evaluation is extensive, covering ASR (LibriSpeech, AISHELL, WenetSpeech), audio understanding (MMSU, MMAU, MMAR, etc.), full-duplex interaction (Artificial Analysis Full-Duplex Bench), and agentic tasks ($\tau$-Voice). The model achieves state-of-the-art or competitive results in most categories, notably leading in MMSU (90.6) and Full-Duplex Bench (98.9). However, it trails competitors like Kimi K3 in pure reasoning/dialogue quality (73.0 vs 77.1 macro average) and GPT-Realtime in retail agentic tasks. The ablation studies on Adaptive Thinking and MTP provide insight into the trade-offs between latency, quality, and reasoning depth. The "Less is more" ablation on data quality is particularly strong, showing significant gains from high-quality SFT data over volume.
The paper provides high-level architectural details and training stages but lacks specific hyperparameters, exact data composition percentages, and code availability. The use of proprietary benchmarks (StepAudioChat) and internal data pipelines limits external reproducibility. The model merging weights (3:1:1:1) are specified, which aids in understanding the integration strategy, but the specific teacher training data is not public.
The model underperforms compared to top-tier LLMs (like Kimi K3) in pure reasoning and dialogue quality when not in real-time mode. The "Adaptive Thinking" mechanism shows inconsistent benefits, sometimes reducing reasoning scores compared to full thinking. The agentic performance in retail domains is notably weak (37.7%). The paper is a technical report without peer review, and some benchmarks are proprietary or newly introduced by the authors, making independent verification difficult.
This work significantly advances the field of real-time voice assistants by addressing the latency-reasoning trade-off. The "Think-While-Speaking" paradigm is a crucial step towards more intelligent, human-like voice interactions that can handle complex tasks without sacrificing responsiveness. The integration of full-duplex capabilities with agentic tool use sets a new standard for voice AI systems. StepAudio 3 Realtime introduces a "Think-While-Speaking" architecture that enables parallel reasoning and speech generation, achieving state-of-the-art performance in full-duplex interaction and audio understanding while maintaining competitive reasoning capabilities. The paper demonstrates a robust system design that balances latency, intelligence, and conversational fluidity, marking a significant milestone in the development of real-time audio-language foundation models.
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.
Primary: Ant Group
All Institutions: Ant Group, Tsinghua University
Realtime-Venus introduces a dual-loop runtime architecture that enables proactive full-duplex multimodal interaction with asynchronous tool delegation, achieving state-of-the-art results in streaming video understanding and conversational continuity metrics. The system effectively bridges the gap between real-time perception and complex background reasoning, offering a scalable framework for next-generation spoken and visual agents.
The paper proposes Realtime-Venus, a system built on the MiniCPM-o 4.5 architecture, featuring two 9B models (Omni and Audio) for full-duplex interaction. The core methodological contribution is the "Dual-Loop" runtime: a foreground loop for latency-sensitive perception and speech generation, and a background "Harness" loop for asynchronous tool execution and reasoning. The models are trained to emit private `
The evaluation is extensive, covering streaming video understanding (StreamingBench, OVO-Bench), audio understanding (MMAU, MMAU-Pro), and specifically full-duplex metrics (Full-Duplex-Bench v1.5). The results show Realtime-Venus-Omni leading on 6/8 video benchmarks and Realtime-Venus-Audio performing competitively on audio tasks. Notably, the full-duplex metrics show high continuation rates (97% under backchannels), demonstrating the system's ability to distinguish between non-interruptive and interruptive speech. However, the comparison is limited to a specific set of online models, and the "delegation" success is measured via an internal benchmark, limiting external validation of the tool-use component.
The paper provides detailed descriptions of the architecture, data pipeline, and training recipe. However, as a system paper from a large industry lab (Ant Group), specific hyperparameters, exact data construction scripts, and the code for the "Harness" runtime are not fully detailed in a way that would allow independent reproduction without significant reverse engineering. The reliance on the open-source MiniCPM-o 4.5 base model helps, but the proprietary data synthesis pipeline is a black box.
The primary limitation is the lack of open-source code and data for the specific Realtime-Venus models and the Harness framework. The evaluation of the asynchronous delegation component relies on an internal benchmark, making it difficult for the community to verify the quality of tool-use integration. Additionally, the paper does not provide ablation studies isolating the contribution of the memory module vs. the base model improvements in the main video benchmarks.
This work is significant for the development of real-time AI assistants that can handle complex, multi-step tasks while maintaining natural conversational flow. The dual-loop architecture addresses a critical bottleneck in current LLM agents: the latency of tool execution. By decoupling perception/speech from reasoning/tools, it enables more robust and human-like interaction patterns, particularly in multimodal environments. Realtime-Venus introduces a dual-loop runtime architecture that enables proactive full-duplex multimodal interaction with asynchronous tool delegation, achieving state-of-the-art results in streaming video understanding and conversational continuity metrics. The system effectively bridges the gap between real-time perception and complex background reasoning, offering a scalable framework for next-generation spoken and visual agents.
Asked to describe what one of six speakers in a recording talks about, audio language models describe the right one on 6 to 16% of trials, below the 16.7% a guess would give. Adding a fixed bias to the attention logits of a hundred heads, under a tenth of the model's and with no training, redirects the description to whichever speaker we choose, on 90.7% to 99.0% of trials. Those heads are largely not specific to audio. Rank the text-only language model an audio model was built from, or a released model of the same family, on a written version of the task, take its top hundred heads, and carry them over unchanged: they redirect the audio model on 80.8% to 95.0% of trials, with nothing about audio entering the selection. The audio and text head sets share 66 to 74 of 100 where chance would give about 20, and the shared part alone reproduces almost all of the steering. What that does not show is that sharing is what makes the heads work: an equal-sized draw from the same discovered hundred does nearly as well, and none of our three models separates the two explanations. A second finding concerns how such heads are found. Ranking heads by how much attention they place on the segment asked about, as an established score does, or by how much of their attention moves with the question, as a per-head normalised variant does, gives top hundreds that share 69, 37 and 4 heads across our three models. In Ultravox, where they share 4, the established score's heads leave output the judge cannot place on any segment on 69.7% of trials, against 40.0% with no intervention and 1.0% for the normalised variant. That is one arm of six; on the other five the established score steers above a random draw.
Primary: Cornell University
All Institutions: Cornell University
The paper demonstrates that audio language models inherit speaker-tracking capabilities from their text backbones, as evidenced by the ability of text-only ranked attention heads to effectively steer audio model outputs via simple logit biasing. This finding challenges the assumption of modality-specific mechanisms for high-level audio understanding and provides a rigorous causal framework for analyzing attention heads in multimodal systems, revealing that standard attention mass scores can be misleading compared to selectivity-based metrics.
The paper proposes a rigorous mechanistic analysis of audio language models (ALMs) using attention head interventions. The core method involves identifying "tracking heads" that attend to specific speaker segments and applying a fixed bias to their attention logits to steer the model's output. The novelty lies in the comparison between heads selected via audio-specific metrics (mass vs. selectivity) and heads selected purely from the text-only backbone. The authors demonstrate that text-derived head rankings can effectively steer audio models, suggesting that speaker tracking mechanisms are inherited from the language backbone rather than being modality-specific. The methodology is sound, utilizing causal interventions (biasing logits) rather than mere correlation, and includes careful controls for random head selection and layer matching.
The experiments are conducted on three distinct ALMs (Qwen2-Audio, Ultravox, SALMONN) using a custom dataset of 6-speaker recordings. The evaluation metrics include accuracy of speaker attribution and "unattributable rate" (abstention). The results show that the proposed intervention significantly improves attribution accuracy (from ~10% to >90%) and that text-derived heads are largely sufficient for this task. The comparison between mass and selectivity rankings reveals that the standard "mass" score can lead to unattributable outputs in some models, highlighting the importance of the selectivity metric. The inclusion of random controls and layer-matched baselines strengthens the causal claims.
The paper provides detailed descriptions of the intervention (bias magnitude, token spans) and the scoring metrics. However, the specific implementation details for the "lexical judge" and the exact prompt templates are not fully detailed in the provided text, which might hinder exact reproduction. The use of 4-bit quantization is noted as a potential threat to fidelity, which is a good transparency point. The code and data availability are not explicitly stated in the provided text, but the methodology is described with enough precision for a skilled researcher to attempt reproduction.
The primary limitation is the reliance on a lexical judge for evaluation, which may not capture all nuances of speaker attribution. The paper acknowledges that the "sharing" of heads between text and audio models does not prove causality of the sharing itself, as random draws from the same set also work. The analysis is limited to three models, and the generalizability to other architectures or tasks is not fully established. The use of a fixed bias magnitude ($B=10^4$) for comparability might not reflect natural model behavior, though $B=5$ is also tested.
This work has significant implications for the interpretability of multimodal models, suggesting that high-level reasoning tasks like speaker tracking may rely on pre-existing linguistic structures rather than modality-specific features. This could inform the design of more efficient multimodal models and provide insights into how attention mechanisms generalize across modalities. It also highlights the limitations of standard attention scoring metrics (mass) for causal analysis, advocating for more nuanced metrics like selectivity. The paper demonstrates that audio language models inherit speaker-tracking capabilities from their text backbones, as evidenced by the ability of text-only ranked attention heads to effectively steer audio model outputs via simple logit biasing. This finding challenges the assumption of modality-specific mechanisms for high-level audio understanding and provides a rigorous causal framework for analyzing attention heads in multimodal systems, revealing that standard attention mass scores can be misleading compared to selectivity-based metrics.
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
Primary: StepFun
All Institutions: StepFun
The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
The paper proposes a unified discrete autoregressive framework for general audio generation, departing from the dominant diffusion/flow-matching paradigm. The core architectural contribution is the "time-depth" decomposition: the LLM backbone predicts the first (semantic) RVQ codebook along the time axis, while a lightweight causal Transformer predicts the remaining 15 residual codebooks along the depth axis. This design effectively isolates the heavy acoustic modeling from the long-range linguistic planning, preventing the acoustic loss from overwhelming the LLM's text capabilities. The introduction of the "RVQ Adaptor," a zero-initialized residual module, is a clever engineering solution to align multi-codebook audio embeddings with the pretrained text embedding space without disrupting the backbone. The four-stage progressive pretraining curriculum, specifically the use of stop-gradient detachment for the residual predictor during the main generation phase, is a sophisticated approach to mitigating catastrophic forgetting and interference.
The evaluation is robust, covering both component-level ablations (RVQ Adaptor, progressive pretraining) and end-to-end generation quality. The ablation studies convincingly demonstrate that the RVQ Adaptor significantly improves audio understanding metrics (ASR, Translation) and that the progressive pretraining preserves text capabilities (MMLU, HumanEval) far better than a baseline. The TTS evaluation uses an Arena-style pairwise comparison with Elo ratings, showing strong performance against commercial systems. The Voice Design evaluation utilizes the InstructTTSEval benchmark with LLM-based judging (Gemini) and human preference testing, demonstrating state-of-the-art instruction following. The inclusion of subjective human evaluation for "human-likeness" is a strong point, as objective metrics often fail to capture prosodic naturalness.
The paper provides high-level architectural details and training recipes (learning rates, batch sizes, data mixtures) but lacks specific implementation details for the tokenizer training (e.g., specific GAN hyperparameters, discriminator architectures beyond "multi-period") and the exact composition of the 2.7T token pretraining corpus. The reliance on proprietary internal models (StepAudio R1.5, StepAudio Music Model) for data generation and evaluation limits full external reproducibility. However, the conceptual framework is clear enough for replication in principle.
The paper focuses heavily on TTS and Voice Design in its quantitative evaluation. While it claims capabilities in music, sound effects, and vocals, the rigorous benchmarking for these domains is less detailed compared to speech. The "Vibe Speech" and complex scene generation capabilities are demonstrated qualitatively but lack objective metrics. The model's performance on long-form generation (beyond the 44-minute context window mentioned) is not extensively tested. Additionally, the reliance on LLMs for reward modeling in RL (GRPO) introduces potential biases from the judge model.
This work significantly advances the field of unified audio generation by proving that discrete autoregressive models can compete with or surpass continuous diffusion models in speech tasks while offering better integration with LLMs. The "interference-aware" training strategy is a valuable lesson for the broader community of multimodal LLMs, providing a blueprint for adding new modalities without degrading existing capabilities. The unified instruction format (ROLE/DIRECTOR/SCRIPT) offers a practical standard for controlling complex audio scenes, which could influence future API designs and application development. The paper presents a unified discrete autoregressive audio generation model that effectively integrates speech, music, and sound effects within a single LLM framework. By employing a time-depth architecture and interference-aware progressive pretraining, the model achieves state-of-the-art performance in TTS and voice design while preserving the backbone's textual reasoning capabilities, offering a scalable alternative to diffusion-based approaches.
Text and lyrics specify broad musical characteristics and sung content but offer limited control over musical timing, melody, and reference-based style. We introduce DiffSynth-Music (https://modelscope.cn/models/DiffSynth-Studio/DiffSynth-Music), a framework that adds composable audio conditioning to a music synthesis backbone through layer-wise key-value injection. The three template models, Control, Prosody, and Reference, are initialized from the backbone diffusion transformer and trained with conditional flow matching. They support five control types: beats, vocals, accompaniment, prosody, and reference audio. A shared variational autoencoder maps conditioning waveforms into a common latent space, enabling their attention memories to be combined. With the template timestep fixed at the clean-data endpoint and other inputs held constant, each control cache is computed once and reused throughout sampling. Training pairs are derived from music recordings using beat extraction, source separation, vocal resynthesis, and reference-excerpt selection. Single-control evaluations on Mandarin and English songs demonstrate improved adherence across all five control types and better lyric fidelity under vocal conditioning relative to the backbone. Automatic music-quality and instruction-following scores remain broadly comparable to those of the evaluated base models, with metric-specific trade-offs. We release the three template models to support research and creative applications in controllable music generation.
Primary: Alibaba Group
All Institutions: Alibaba Group, Shanghai Jiao Tong University
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
The paper introduces DiffSynth-Music, a framework for controllable music generation built upon a frozen Diffusion Transformer (DiT) backbone (ACE-Step-1.5-XL-SFT). The core methodology involves injecting layer-wise key-value (KV) representations derived from audio conditioning signals into the generation branch. This approach is conceptually related to adapter-based methods like ControlNet but is tailored for audio and uses a shared VAE to map diverse conditioning waveforms into a common latent space. A key strength is the design of three template models (Control, Prosody, Reference) that support five distinct control types: beats, vocals, accompaniment, prosody, and reference audio. The detailed construction of these control signals from raw music recordings (e.g., beat extraction, source separation, vocal resynthesis for prosody, loudest excerpt for reference) is well-described and practical. The prosody generation method, which uses a sinusoidal carrier with estimated pitch and envelope to reduce phonetic/timbral cues, is a clever way to achieve melodic control without enforcing specific vocal identity. The KV-injection mechanism, where control keys and values are concatenated with the generation branch's keys and values, allows for composable conditioning. The efficiency gain from computing each control cache once at a fixed template timestep (clean-data endpoint) and reusing it throughout sampling is a significant practical advantage, reducing computational overhead. The training formulation uses conditional flow matching, which is a suitable choice for learning continuous-time vector fields, and the adapter-only optimization ensures the pretrained backbone remains frozen, preserving its general generation capabilities. The overall architecture is sound, well-motivated, and addresses a clear need in controllable music generation.
The experimental evaluation is comprehensive and well-structured. The authors evaluate DiffSynth-Music on both Mandarin and English songs, demonstrating its applicability across different languages, which is a strong point. The comparison against several state-of-the-art music generation systems (ACE-Step-1.5-XL-SFT, DiffRhythm-2, HeartMuLa-3B, MiniMax-Music3, LeVo-2-Large) provides a robust baseline. Critically, the backbone model (ACE-Step-1.5-XL-SFT) is included as a baseline to specifically highlight the improvements gained by adding the audio-conditioned templates. The evaluation metrics are appropriate and cover both control adherence and general music quality. For control adherence, metrics like Beat-F1, Cemgil (beats), V-MSE, PER (vocals), A-MSE (accompaniment), Pitch_50 (prosody), and MuLan-A (reference) are used. The results clearly show that DiffSynth-Music significantly outperforms all baselines across all seven control adherence metrics, often by a large margin (e.g., Beat-F1 from ~0.3 to ~0.85, Pitch_50 from ~0.04 to ~0.47). This strongly validates the effectiveness of the proposed conditioning framework. To assess general music quality, AudioBox-Aesthetics scores (CE, CU, PC, PQ) and MuLan-T (text-music alignment) are used. The results indicate that DiffSynth-Music maintains "broadly comparable" scores to the evaluated base models, with some metric-specific trade-offs. While some scores are slightly lower than the backbone, the overall message is that the added controllability does not severely degrade the general generation quality, which is a reasonable and expected outcome for such an adapter-based system. The experimental setup and results convincingly support the paper's claims.
The paper makes a strong commitment to reproducibility by open-sourcing the three template models (Control, Prosody, Reference). This is a significant contribution to the research community. The methodology section provides sufficient detail on the architecture, KV injection, and the construction of the five audio control signals, including specific tools used (Essentia, Demucs, pYIN). This level of detail allows researchers to understand and potentially replicate the control signal generation process. However, a limitation for full reproducibility is the use of a private dataset of approximately 60k music recordings for training and a private 50-sample Mandarin dataset for testing. While the English test set uses MUSDB18-HQ, the full training data and the Mandarin test data are not publicly available. This means that while the *models* are released, replicating the *training process* or *full evaluation* exactly would be challenging without access to similar proprietary datasets. The specific parameters for training (e.g., learning rates, batch sizes, number
We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.
Primary: StepFun
All Institutions: StepFun, ACE, The Chinese University of Hong Kong, University of California San Diego
StepAudio 3 Music introduces a robust framework for long-form music generation that integrates explicit symbolic planning (ABC-CoT) with a discrete-continuous audio pipeline, achieving competitive performance with leading commercial systems. The paper provides a thorough analysis of the trade-offs in audio tokenization for autoregressive models and demonstrates the efficacy of preference optimization in enhancing musical quality and condition adherence.
The paper proposes a hierarchical architecture for long-form music generation combining a Mixture-of-Experts (MoE) autoregressive language model with a flow-matching Diffusion Transformer (DiT) renderer. A key methodological contribution is the "ABC-CoT" (ABC Chain-of-Thought) mechanism, where the LLM generates an intermediate symbolic arrangement plan in ABC notation before predicting discrete audio tokens. This allows for explicit control over harmony, rhythm, and structure. The audio representation uses a novel 50-Hz single-codebook tokenizer (65,536 entries) trained with semantically informed self-supervised objectives, which the authors argue offers a better trade-off between acoustic fidelity and autoregressive predictability compared to multi-codebook RVQ approaches. The training pipeline is extensive, involving a progressive curriculum from large-scale pre-training to high-quality annealing, followed by SFT and DPO for preference alignment.
The evaluation is comprehensive, utilizing both objective metrics (SongBench, AudioBox-Aesthetics, MuQ-MuLan) and subjective human evaluation via the Artificial Analysis Music Arena. The model achieves top-tier results, ranking 4th in the Arena (Elo 1105) behind only Suno V5.5 and Mureka, and leading in several objective quality metrics. Ablation studies effectively demonstrate the contribution of the DPO stage and the ABC-CoT planning mechanism, showing consistent improvements in musicality and condition adherence. The comparison with commercial systems (Suno, MiniMax) provides strong external validation of the model's competitive standing.
The paper provides detailed architectural specifications, training hyperparameters, and data pipeline descriptions. However, as is common with large-scale industrial models, the specific codebase and full training data are not released. The reliance on proprietary components (StepAudio VAE, specific LLM backbone) limits full independent reproduction, though the methodology is described with sufficient clarity for understanding the design choices.
The primary limitation is the lack of open-source code and weights, which restricts the community's ability to verify results or build upon the specific implementation. The evaluation, while strong, relies heavily on the model's performance in a specific arena and a set of internal benchmarks; broader generalization to diverse musical styles or non-vocal genres is less thoroughly explored in the text. The "ABC-CoT" approach adds computational overhead during inference due to the two-pass generation process.
This work significantly advances the state of the art in controllable music generation by integrating symbolic reasoning (ABC notation) with neural audio synthesis. It demonstrates that explicit intermediate planning can improve the coherence and controllability of long-form music generation. The findings regarding the trade-offs between single-codebook VQ and RVQ for autoregressive music models provide valuable insights for future audio codec design. The high performance on human preference leaderboards indicates practical utility for creative applications. StepAudio 3 Music introduces a robust framework for long-form music generation that integrates explicit symbolic planning (ABC-CoT) with a discrete-continuous audio pipeline, achieving competitive performance with leading commercial systems. The paper provides a thorough analysis of the trade-offs in audio tokenization for autoregressive models and demonstrates the efficacy of preference optimization in enhancing musical quality and condition adherence.
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
Primary: Mila-Quebec AI Institute
All Institutions: Concordia University, Mila-Quebec AI Institute, Université Laval
ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
The paper proposes ZipCodec, a streaming neural speech codec that significantly reduces the frame rate to 6.25 Hz (160 ms per token) while maintaining a low bitrate of 0.80 kbps. The methodology is built upon the FocalCodec-Stream architecture but introduces several key modifications: (1) replacing the learned convolutional encoder with a causal log-mel frontend to simplify the pipeline; (2) introducing "ErfFormer," a transformer variant that removes RMSNorm and positional encodings, relying on DynamicErf activation and causal attention to handle long streaming contexts without positional extrapolation; (3) employing Scalar Spherical Quantization (SSQ) with a factorized bottleneck (64 dimensions, 4 levels) to achieve compact representation; and (4) a latency-aware streaming decoder that processes 8 WavLM representations jointly within the 160 ms window to relax causality constraints at the feature level. The training strategy involves large-scale distillation of WavLM layer-6 representations using ~94,000 hours of speech data, with specific noise and overlapping speech augmentations to match the teacher model's distribution.
The experimental evaluation is comprehensive, covering speech resynthesis (English and multilingual), voice conversion, and a suite of downstream discriminative and generative tasks (ASR, SI, SER, KS, IC, SE, SS). ZipCodec demonstrates superior performance compared to strong baselines like Mimi, PAST, and FocalCodec-Stream, particularly in intelligibility (dWER) and speaker similarity (Sim) at the significantly lower frame rate. The paper also provides detailed streaming efficiency metrics, showing real-time inference on consumer-grade CPUs and efficient GPU scaling. The results convincingly show that the 6.25 Hz bottleneck retains sufficient information for high-quality reconstruction and downstream tasks, narrowing the gap with non-streaming offline codecs.
The paper provides high reproducibility. It explicitly states that code and checkpoints are available. Detailed hyperparameters, training data sources (LibriLight, VoxPopuli, GigaSpeech), augmentation strategies, and architectural specifications (e.g., ErfFormer dimensions, SSQ parameters) are provided. The use of standard datasets and metrics (UTMOS, dWER, DNSMOS) allows for easy verification and comparison with other works in the field.
The primary limitation is the model size (842M parameters), which, while manageable for single-stream CPU inference, may be prohibitive for edge devices or high-concurrency server deployments compared to smaller codecs. Additionally, the evaluation is primarily focused on English and specific multilingual subsets; broader language coverage is not extensively tested. The reliance on WavLM distillation ties the codec's performance to the quality and biases of the WavLM teacher model.
This work has significant implications for speech-language modeling and real-time speech-to-speech systems. By reducing the frame rate to 6.25 Hz, the token sequence length is drastically shortened, which reduces computational costs and simplifies autoregressive modeling for speech LLMs. The low latency (160 ms) makes it suitable for highly responsive conversational AI applications. The open-sourcing of code and demos facilitates further research and adoption in the community. ZipCodec introduces a streaming neural speech codec operating at an exceptionally low frame rate of 6.25 Hz and 0.80 kbps, achieving state-of-the-art reconstruction and representation quality through a redesigned transformer architecture (ErfFormer), scalar spherical quantization, and large-scale WavLM distillation, thereby enabling efficient and responsive speech language modeling.
Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs' reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.
Primary: Imperial College London
All Institutions: Imperial College London, Technische Universität München, Mohamed bin Zayed University of Artificial Intelligence, Shanghai Jiao Tong University, Johns Hopkins University
The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
The paper proposes ARIA-Rubrics, a framework to evaluate the reasoning process of Large Audio Language Models (LALMs) rather than just final answer accuracy. The methodology relies on Chain-of-Thought (CoT) prompting to externalize reasoning into four steps: Perception, Analysis, Reasoning, and Answer. It introduces six metrics: Acoustic Grounding Score (using CLAP embeddings to verify entities against audio), Inter-step Coherence (using MPNet embeddings), Reasoning Chain Coverage (using a lightweight LLM judge for substantiveness), Reasoning Step Causality (using perplexity reduction), Reasoning Progress Score (using NLI entailment), and Audio Lexical Density (using Word2Vec similarity to AudioSet ontology). The approach is innovative in combining audio-specific perceptual checks with text-based reasoning quality metrics, addressing the gap where text-only metrics fail to detect perceptual hallucinations.
The authors evaluate 9 models across two benchmarks (MMAR and MMAU-mini). They identify three distinct reasoning modes: Reasoning, Scaffolding, and Decoration. The paper claims high correlation with human judgments, validating the automated metrics. The experimental setup is robust, covering both open-source and closed-source models, and the identification of specific failure modes (like scaffolding) provides actionable insights for model developers.
The code is available on GitHub. The paper specifies the models used for each metric (CLAP, MPNet, Qwen2.5-1.5B, RoBERTa-large-mnli, Word2Vec) and provides details on the CoT prompt structure and the AudioSet ontology filtering. This level of detail supports reproducibility, although the specific few-shot prompts for the LLM judge are referenced in appendices not fully visible in the truncated text, which is standard practice.
The framework relies on the quality of the CoT output; if a model refuses to generate CoT or generates unstructured text, the metrics may fail or be biased. The use of a small LLM (Qwen2.5-1.5B) for judging substantiveness and causality may limit the depth of evaluation compared to larger models. Additionally, the Audio Lexical Density metric is a proxy and may not capture all aspects of audio-relevant reasoning.
This work is significant for the development of reliable audio AI systems. By distinguishing between genuine reasoning and pattern matching, it provides a diagnostic tool for improving LALMs. It encourages the development of models that are not only accurate but also faithful in their reasoning, which is crucial for safety and interpretability in audio applications. The paper introduces ARIA-Rubrics, a comprehensive framework for evaluating audio reasoning in LALMs by combining perceptual grounding checks with reasoning process analysis. It effectively addresses the limitation of accuracy-only benchmarks by identifying specific reasoning failure modes, offering a valuable tool for the community to develop more robust and transparent audio language models.
We present UniStream, a fully causal 48 kHz neural audio codec for streaming speech, music, and environmental sounds. At its core is Multi-Expert Residual Vector Quantization (ME-RVQ), which replaces the single shared codebook in each residual quantization layer with four expert codebooks controlled by a deterministic Top-K router. Because routing decisions are derived solely from previously decoded quantized states, the decoder can reproduce the selected experts without transmitting expert identifiers, thereby expanding quantization capacity while adding 5.5M parameters. We further introduce an auxiliary Optimal Transport Conditional Flow Matching (OT-CFM) objective to regularize the quantized latent space during training. The flow module is removed entirely at inference and therefore incurs no runtime overhead. UniStream supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode within a causal 48 kHz encoder-decoder framework, while achieving real-time GPU inference. To complement narrow-band speech metrics, we report 48 kHz ViSQOL audio mode, ViSQOL speech mode, standard VGGish-FAD, DNSMOS P.835, and higher-rate reference comparisons with Opus and EnCodec. At 12 kbps, UniStream-Top1 achieves PESQ and UTMOS scores comparable to EnCodec while reducing speech Mel-D from 13.07 to 8.21. At 22.5 kbps, UniStream-Top2 achieves a ViSQOL speech-mode score of 4.67 and an environmental audio-mode score of 3.96, exceeding all evaluated systems operating at 12 kbps or below in the latter setting. It also comes within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode. Ablation studies confirm that ME-RVQ is the primary source of quality improvement, whereas OT-CFM provides perceptual gains on speech with a mild trade-off in spectral distortion.
Primary: Tsinghua University
All Institutions: Tsinghua Shenzhen International Graduate School, Tsinghua University
UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.
The paper proposes UniStream, a causal neural audio codec featuring Multi-Expert Residual Vector Quantization (ME-RVQ). The core innovation is replacing single codebooks in residual layers with four expert codebooks routed by a deterministic Top-K mechanism. Crucially, the routing is derived from previously decoded states, allowing the decoder to replicate expert selection without transmitting expert IDs, thus saving bandwidth. The method also introduces a training-only Optimal Transport Conditional Flow Matching (OT-CFM) regularizer to improve latent space quality without inference cost. The architecture is fully causal, suitable for streaming, and supports 12 kbps and 22.5 kbps modes.
Experiments are conducted on speech (LibriSpeech), music (MTG-Jamendo), and environmental audio (FSD50K). The evaluation is comprehensive, using both narrow-band metrics (PESQ, STOI, UTMOS) and full-band perceptual metrics (ViSQOL, VGGish-FAD, DNSMOS). Results show UniStream-Top1 outperforms EnCodec in spectral distortion (Mel-D) at 12 kbps, while UniStream-Top2 approaches Opus 24 kbps quality at a lower bitrate. Ablations confirm ME-RVQ is the primary driver of improvement, with OT-CFM providing modest perceptual gains.
The paper provides detailed architectural specifications (strides, dilation factors, codebook sizes, loss weights) and training hyperparameters. However, no code repository or demo link is provided in the text. The use of standard datasets and metrics facilitates reproduction, but the lack of open-source code limits immediate verification.
The paper lacks subjective listening tests (MOS), relying solely on objective metrics. Music domain performance (FAD) is noted as weaker compared to speech and environmental audio. The CPU inference time (RTF 1.05) is borderline for real-time deployment on standard hardware, though GPU inference is efficient. The comparison with non-causal baselines like EnCodec is somewhat unfair for streaming contexts, though acknowledged.
This work contributes to the field of low-latency neural audio coding, particularly for applications requiring full-band (48 kHz) fidelity in streaming scenarios. The deterministic routing mechanism offers a practical solution for expanding quantization capacity without signaling overhead, which could influence future codec designs. UniStream introduces a causal 48 kHz audio codec using Multi-Expert Residual Vector Quantization to expand capacity without signaling overhead, achieving competitive quality at low bitrates. The paper demonstrates a solid engineering approach to improving neural codecs for streaming, with rigorous evaluation across multiple audio domains, though it lacks subjective validation and open-source availability.