This week’s shortlist
Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending…
Abstract
Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending hearing to sounds that are inaudible to humans. More broadly, the physical world is full of waves, vibrations, and oscillations, only a small fraction of which are directly audible to humans. This invites the extension of multimodal LLMs toward a broader form of machine hearing, which remains largely underexplored. To investigate this direction, we introduce the Machine Hearing Benchmark (MHB), comprising 6,128 instances across 13 tasks and seven domains. MHB spans highly heterogeneous input regimes, with sampling rates from approximately $10^{-5}$ to $10^6$ Hz, sequence lengths from $10^1$ to $10^6$ samples, and both single- and multi-channel inputs. Evaluations of current open-source and proprietary Audio LLMs (ALLMs) reveal limited zero-shot transfer to machine hearing. Supervised adaptation on machine-hearing data alone does not close this gap, suggesting that the challenge extends beyond data exposure to a systematic mismatch between existing audio front ends and machine-hearing inputs. To address this mismatch, we introduce MachEar, a lightweight encoder extension that complements the pretrained Audio LLM for machine-hearing inputs. The results show that MachEar equips a pretrained Audio LLM with stronger machine-hearing capabilities while largely preserving its existing audio understanding performance.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces the Machine Hearing Benchmark and the MachEar encoder extension, demonstrating that current Audio LLMs struggle with non-human-audible signals and proposing a lightweight architectural fix to bridge this gap. The technical contribution is strong, combining a well-curated benchmark with a targeted architectural innovation. The methodology is sound, using parallel processing to handle heterogeneous signal regimes without disrupting the pretrained LLM's capabilities. The significance to the field is high, as it shifts the focus from human-centric audio understanding to broader physical world perception, addressing a critical limitation in current multimodal models. The rigorous evaluation against both general and domain-specific models provides a clear baseline and validates the proposed solution's effectiveness.
Methodology
The paper proposes a novel paradigm of "Machine Hearing," extending Audio LLMs (ALLMs) beyond human-audible frequencies to process signals ranging from $10^{-5}$ Hz to $10^6$ Hz. The methodology involves constructing the Machine Hearing Benchmark (MHB) with 6,128 instances across 13 tasks in 7 domains (Astronomy, Earth Science, Healthcare, etc.). The core technical contribution is MachEar, a lightweight encoder extension that processes inputs in parallel with the pretrained audio encoder. MachEar utilizes parallel FFT and 1D CNN pathways to extract spectral and temporal features, which are then processed by Transformer blocks to generate compact tokens. This design aims to address the systematic mismatch between standard audio front-ends (optimized for 16-48kHz speech/audio) and extreme machine-hearing inputs (ultra-low or ultra-high frequency, short sequences). The approach is elegant in its modularity, allowing it to complement rather than replace the existing LLM backbone.
Results & evaluation
The experimental evaluation is rigorous and comprehensive. The authors benchmark seven state-of-the-art ALLMs (including Qwen2.5-Omni, Audio-Flamingo-3, Gemini-3-Flash, GPT-Audio-Mini, and Doubao) in a zero-shot setting. The results clearly demonstrate that current ALLMs fail significantly on machine-hearing tasks, often performing below random guessing. The paper also evaluates domain-specific foundation models (e.g., SeisLM, SpectrumFM) to provide context, showing that while domain pretraining helps, it doesn't solve the generalization issue for ALLMs. The introduction of MachEar shows substantial improvements in Macro-F1 scores for Qwen2.5-Omni, validating the hypothesis that the bottleneck lies in the audio front-end processing rather than just the LLM's reasoning capabilities. The analysis of failure modes (e.g., short-input failures, unrecognized inputs) provides valuable diagnostic insights.
Reproducibility
The paper provides high reproducibility. The benchmark is released on HuggingFace, and the code for MachEar is presumably available (linked in the project URL). The experimental setup, including model versions, prompting protocols, and evaluation metrics (Macro-F1, Success Rate), is clearly defined. The use of public datasets and standard models further enhances reproducibility.
Limitations
The primary limitation is the reliance on a specific base model (Qwen2.5-Omni) for the MachEar demonstration; while the architecture is general, the specific gains might vary across different LLM backbones. Additionally, the benchmark, while diverse, is relatively small (6,128 instances) for such a broad range of domains, which may limit the statistical robustness of some conclusions. The paper focuses on classification tasks; the ability to perform generative or reasoning tasks on these signals is not deeply explored.
Broader impact
This work has significant broader impact by defining a new frontier for multimodal AI: perception beyond human sensory limits. It opens avenues for AI in scientific discovery (astronomy, geophysics), industrial monitoring, and healthcare diagnostics where signals are not human-audible. The proposed MachEar module is a practical solution that can be integrated into existing ALLMs, potentially accelerating the adoption of AI in these specialized fields. The paper introduces the Machine Hearing Benchmark and the MachEar encoder extension, demonstrating that current Audio LLMs struggle with non-human-audible signals and proposing a lightweight architectural fix to bridge this gap. The technical contribution is strong, combining a well-curated benchmark with a targeted architectural innovation. The methodology is sound, using parallel processing to handle heterogeneous signal regimes without disrupting the pretrained LLM's capabilities. The significance to the field is high, as it shifts the focus from human-centric audio understanding to broader physical world perception, addressing a critical limitation in current multimodal models. The rigorous evaluation against both general and domain-specific models provides a clear baseline and validates the proposed solution's effectiveness.
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently…
Abstract
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and optimizes its diarization and TS-ASR components separately. We address the first two. We integrate the diarization conditioning into a longform Parakeet backbone (DiCoP) and add cross-speaker attention, which lets the model attend across the speaker dimension so that speakers are no longer decoded in isolation, and an STNO mask refiner that re-estimates the diarization-derived conditioning. As conversational data with genuine multi-speaker interaction is scarce, we scale training with a pseudo-labeling pipeline that mines 5.4k hours of in-the-wild multi-speaker audio from YODAS2. Combining these with synthetic and real data yields state-of-the-art results across several meeting benchmarks. We open-source the resulting models, DiCoP and Pandemonium, and the pseudo-labeled data.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces DiCoP and Pandemonium, which integrate diarization conditioning into a long-form Parakeet backbone with cross-speaker attention to improve target-speaker ASR in multi-talker scenarios. This work addresses significant structural limitations of previous diarization-conditioned ASR systems by enabling joint optimization and cross-speaker interaction modeling, supported by a robust pseudo-labeling pipeline for data scaling, resulting in state-of-the-art performance on meeting benchmarks.
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision…
Abstract
Recent audio embedding models learn representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior self-supervised audio embedding models on clip-level classification tasks spanning environmental sound, speech, and music, and on frame-level sound event detection. All code and data are available at https://github.com/labhamlet/AudioSphere and https://huggingface.co/labhamlet.
AI-generated analysis. Check claims against the paper.
Summary
AudioSphere introduces a multi-channel masked autoencoder with joint spatiotemporal masking to learn self-supervised spatial audio representations, achieving state-of-the-art performance on both real-world SELD and monaural audio tasks. The paper makes a significant technical contribution by bridging the gap between monaural SSL and spatial audio modeling, demonstrating that spatial cues can be learned effectively without explicit supervision and that such representations transfer robustly to a wide range of downstream tasks. The introduction of the RealSELD benchmark and the open-sourcing of the model and data pipeline further enhance the paper's impact, providing the community with essential tools for advancing spatial audio research.
Methodology
The paper proposes AudioSphere, a multi-channel masked autoencoder (MAE) designed for self-supervised learning of spatial audio representations. The core innovation lies in the joint modeling of acoustic content (log-mel spectrograms) and spatial cues (intensity vectors) derived from First-Order Ambisonics (FOA). The authors introduce a "tube masking" strategy that masks time-frequency patches across all channels simultaneously, preventing the model from trivially copying information from unmasked channels at the same spatial location. This forces the encoder to learn correlations between acoustic features and directional cues. The architecture utilizes a ViT-Base encoder and a decoder with local-global attention. A significant methodological contribution is the data generation pipeline, which synthesizes 70,000 Ambisonic room impulse responses (ARIRs) using Matterport3D meshes and Monte Carlo ray tracing, paired with online mixing of AudioSet clips. This addresses the scarcity of large-scale, labeled spatial audio data. The use of Ambisonics Signal Matching (ASM) allows the model to generalize across different microphone array geometries (tetrahedral, planar, spherical) without retraining, a robust engineering choice that enhances utility.
Results & evaluation
The experimental evaluation is comprehensive, addressing both the transferability of spatial features to monaural tasks and the efficacy of the model for spatial tasks. The authors introduce RealSELD, a standardized benchmark for real-world SELD, which is a valuable contribution to the field. Results show that AudioSphere, when used as a frozen backbone with shallow MLP probes, outperforms fully supervised baselines on static SELD tasks and transfers competitively to dynamic environments. Crucially, the paper demonstrates that spatial pre-training does not degrade performance on monaural tasks; in fact, it matches or exceeds prior state-of-the-art monaural SSL models (like MW-MAE) on the HEAR benchmark suite, which includes environmental sound, speech, and music classification. The ablation studies on masking strategies (tube vs. per-channel) provide strong evidence for the necessity of the proposed joint masking approach. The comparison with MUSIC and Binaural Signal Matching pipelines on real-world data further validates the practical superiority of the learned representations.
Reproducibility
The authors provide high reproducibility by releasing all code, data, and model weights on GitHub and HuggingFace. The detailed description of the simulation pipeline, including the specific parameters for Matterport3D scenes and the online mixing process, allows other researchers to replicate the training data generation. The use of standard components (ViT, ASM) and clear architectural specifications further supports reproducibility. The release of the RealSELD benchmark kit is particularly beneficial for the community, enabling standardized evaluation of future spatial audio models.
Limitations
A primary limitation is the reliance on simulated room impulse responses for pre-training. While the authors argue that the model generalizes to real-world recordings, the gap between simulated and real-world acoustic conditions (e.g., non-linearities, complex material interactions) may still exist, though the results on RealSELD suggest this gap is manageable. The model is trained on First-Order Ambisonics, which limits the spatial resolution to 4 channels; higher-order Ambisonics could provide finer directional detail but would increase computational cost. Additionally, the pre-training corpus, while large (4.8k hours of AudioSet mixed with simulations), is still smaller than the massive corpora used for some monaural models (e.g., Dasheng's 272k hours), which might limit the ceiling of performance on certain acoustic tasks.
Broader impact
This work has significant implications for the field of spatial audio understanding. By demonstrating that self-supervised learning can effectively capture joint acoustic-spatial features, it opens the door to scalable, label-free training for spatial audio models. The release of RealSELD provides a much-needed standardized benchmark for evaluating real-world SELD, addressing a long-standing issue of fragmented evaluation in this subfield. The ability to use a frozen backbone for both monaural and spatial tasks suggests that spatial pre-training can be a general-purpose enhancement for audio representation learning, potentially benefiting applications in augmented reality, virtual reality, and assistive technologies for the deaf and hard of hearing. AudioSphere introduces a multi-channel masked autoencoder with joint spatiotemporal masking to learn self-supervised spatial audio representations, achieving state-of-the-art performance on both real-world SELD and monaural audio tasks. The paper makes a significant technical contribution by bridging the gap between monaural SSL and spatial audio modeling, demonstrating that spatial cues can be learned effectively without explicit supervision and that such representations transfer robustly to a wide range of downstream tasks. The introduction of the RealSELD benchmark and the open-sourcing of the model and data pipeline further enhance the paper's impact, providing the community with essential tools for advancing spatial audio research.
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text…
Abstract
Speech retrieval finds relevant spoken documents given a spoken query. The spoken query alone may not reveal the retrieval goal, such as finding an answer or matching speaker identity. Instruction-aware speech retrieval expresses this goal through a natural-language text instruction. Large audio-language models (LALMs) effectively process speech and text, and our analysis shows that different retrieval requirements favor hidden states from different audio-encoder layers. We introduce IRIS-Embed, which adapts LALMs into embedding models through contrastive training and instruction-aware audio layer weighting. We construct about 900K training samples from real data and synthetic data, with instructions expressing requirements for one or more speech attributes. On INSPIRE, an instruction-aware speech retrieval benchmark, IRIS-Embed consistently outperforms the corresponding LALM baselines. The best model achieves 62.92% macro-average Recall@50, surpassing the strongest baseline at 45.15%. Controlled comparisons demonstrate instruction awareness and the benefits of audio layer weighting. On MAEB, a massive audio embedding benchmark, macro-average gains of 9.4-19.8 points over the corresponding baselines show generalization beyond speech retrieval.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces IRIS-Embed, a novel method for instruction-aware speech retrieval that adapts Large Audio-Language Models by learning instruction-conditioned weights to fuse hidden states from different audio encoder layers. This approach significantly outperforms existing baselines on the INSPIRE benchmark and demonstrates strong generalization to broader audio embedding tasks, offering a scalable and effective framework for context-aware audio retrieval.
Methodology
The paper proposes IRIS-Embed, a framework for adapting Large Audio-Language Models (LALMs) into embedding models for instruction-aware speech retrieval. The core methodological contribution is "instruction-aware audio layer weighting." The authors observe that different retrieval tasks (e.g., speaker matching vs. content retrieval) benefit from different layers of the frozen audio encoder. To exploit this, they introduce a lightweight module that takes the text instruction as input and predicts softmax weights to combine hidden states from all layers of the audio encoder. This weighted representation is then projected and fed into the language model (adapted via LoRA) to generate the final embedding. The training objective is InfoNCE contrastive loss, with careful handling of false negatives by masking potentially relevant documents from the batch pool. The data construction pipeline is robust, combining real data from various corpora (VoxCeleb, UrbanSound8K, etc.) with synthetic data generated via TTS to control specific attributes like speaker identity and style.
Results & evaluation
The experiments are extensive and well-controlled. The primary evaluation is on the INSPIRE benchmark, which is specifically designed for instruction-aware speech retrieval. The model significantly outperforms strong baselines, including specialized encoders (ECAPA-TDNN, emotion2vec) and other LALM-based embeddings (LCO-Embedding). The best model achieves 62.92% Recall@50, surpassing the strongest baseline (45.15%) by a large margin. Ablation studies confirm the effectiveness of both instruction awareness and the audio layer weighting mechanism. Furthermore, the model's generalization is tested on the MAEB benchmark, showing consistent gains across 15 diverse audio tasks, indicating that the learned embeddings are not overfitted to the specific retrieval task but capture broad audio-linguistic alignment.
Reproducibility
The authors provide a public GitHub repository containing code and data construction scripts. The paper details the training hyperparameters, data sources, and specific model architectures (Qwen2.5-Omni-3B, Gemma-4-E2B-it). The use of standard components (LoRA, InfoNCE, frozen encoders) and open-source models enhances reproducibility. The detailed description of the data construction pipeline, including how synthetic data is generated and how instructions are formulated, allows other researchers to replicate the dataset.
Limitations
One potential limitation is the reliance on a specific set of LALMs (Qwen2.5-Omni and Gemma-4). While these are state-of-the-art, the method's performance on other architectures or smaller models might vary. Additionally, the instruction-aware weighting module adds computational overhead during inference, as it requires processing the instruction to generate layer weights, although this is likely negligible compared to the LLM inference. The paper focuses on English speech; performance on multilingual or low-resource languages is not evaluated.
Broader impact
This work bridges the gap between general audio-language understanding and specific retrieval tasks. By demonstrating that LALMs can be effectively adapted for retrieval via layer weighting, it opens avenues for more flexible and context-aware audio search systems. The approach could be extended to other multimodal retrieval tasks (video, image) where different modalities or tasks might benefit from different feature layers. The public release of the dataset and code will likely accelerate research in instruction-aware retrieval. The paper introduces IRIS-Embed, a novel method for instruction-aware speech retrieval that adapts Large Audio-Language Models by learning instruction-conditioned weights to fuse hidden states from different audio encoder layers. This approach significantly outperforms existing baselines on the INSPIRE benchmark and demonstrates strong generalization to broader audio embedding tasks, offering a scalable and effective framework for context-aware audio retrieval.
Music source separation extracts vocals and instruments from a mixed recording, enabling remixing, content production, and music analysis. Existing systems commonly operate on waveforms or full-band spectrograms. Waveform inputs contain many samples even in short segments…
Abstract
Music source separation extracts vocals and instruments from a mixed recording, enabling remixing, content production, and music analysis. Existing systems commonly operate on waveforms or full-band spectrograms. Waveform inputs contain many samples even in short segments, whereas a single spectrogram resolution imposes a shared time-frequency resolution trade-off across frequency ranges. We propose SubRoFormer, a Transformer-based system comprising a subband module and a residual RoFormer module. The subband module decomposes the mixture into frequency-localized, lower-rate signals for separation. It then reconstructs the target waveforms from the estimated subbands. The residual RoFormer module models temporal and cross-band relationships on temporally compressed features. It combines this context with fine-rate features through gated residual fusion to estimate source-specific complex masks. On the MUSDB18-HQ test set, training only on MUSDB18-HQ yields an average signal-to-distortion ratio (SDR) of 10.63 dB across four stems using target-specific models, achieving state-of-the-art performance among the compared systems without extra training data. With 500 additional training songs, a single model jointly separating all four stems achieves an average SDR of 11.59 dB, establishing state-of-the-art performance among the compared single-model multi-stem systems trained with extra data. Code and checkpoints are released at https://github.com/ArchiMickey/SubRoformer.
AI-generated analysis. Check claims against the paper.
Summary
SubRoFormer introduces a subband time-domain representation that effectively resolves the time-frequency resolution trade-off in music source separation, achieving state-of-the-art performance on MUSDB18-HQ with improved parameter efficiency. The paper demonstrates a rigorous integration of signal processing principles with Transformer-based architectures, providing a robust and reproducible framework for high-fidelity audio separation.
Methodology
The paper proposes SubRoFormer, a hybrid architecture that combines a fixed subband time-domain frontend with a residual RoFormer separator. The frontend decomposes the mixture into frequency-localized, lower-rate complex baseband signals using a triangular filter bank, demodulation, and decimation. This addresses the resolution trade-off of full-band STFTs by allowing different frequency resolutions. The separator operates on local STFTs of these subbands, using a temporally patched axial RoFormer to model long-range dependencies while retaining fine-rate features via a skip connection. A GEGLU fusion module combines the coarse-rate context with fine-rate features to predict complex masks. The method is mathematically sound, leveraging standard signal processing principles (Nyquist, polyphase implementation) to ensure efficient and near-perfect reconstruction of the subbands before separation. The use of a fixed, non-learned frontend is a deliberate design choice to isolate the separation task from representation learning, which is a valid and rigorous approach.
Results & evaluation
The experiments are conducted on the MUSDB18-HQ dataset, the standard benchmark for music source separation. The paper reports state-of-the-art results, achieving 10.63 dB average SDR with target-specific models and 11.59 dB with a single multi-stem model using extra data. The ablation studies are thorough, examining the impact of the subband frontend, temporal patching, and fusion mechanisms. The paper also provides a detailed analysis of the reconstruction quality of the subband frontend, showing it introduces minimal distortion compared to the STFT. The comparisons are fair, using the same training data and augmentation policies as baselines where possible. The inclusion of parameter efficiency comparisons is a strong point, showing that the method achieves high performance with fewer parameters than some baselines.
Reproducibility
The paper provides high reproducibility. Code and checkpoints are released on GitHub. The implementation details, including filter bank specifications, training hyperparameters, and augmentation policies, are described in detail. The use of standard libraries (SciPy for FIR filters) and common frameworks (PyTorch implied by the architecture) further enhances reproducibility. The paper clearly distinguishes between the fixed frontend and the learned separator, making it easier to replicate the system.
Limitations
The primary limitation is the computational cost of the subband frontend, although the polyphase implementation mitigates this. The method relies on a fixed filter bank, which may not be optimal for all types of music or sources. The paper does not extensively discuss the impact of the subband count and bandwidth on separation performance, only providing a single ablation with 64 subbands. The use of an internal dataset of 500 songs for the extra-data experiments limits the generalizability of those results, as the dataset is not publicly available. The paper also does not provide subjective listening tests, which are important for evaluating the perceptual quality of separated stems.
Broader impact
The proposed method has significant potential for applications in music production, remixing, and audio restoration. The ability to separate stems with high fidelity and efficiency could enable new tools for musicians and producers. The subband approach could be extended to other audio separation tasks, such as speech enhancement or environmental sound separation. The paper contributes to the broader field of audio signal processing by demonstrating the effectiveness of combining classical signal processing techniques with modern deep learning architectures. SubRoFormer introduces a subband time-domain representation that effectively resolves the time-frequency resolution trade-off in music source separation, achieving state-of-the-art performance on MUSDB18-HQ with improved parameter efficiency. The paper demonstrates a rigorous integration of signal processing principles with Transformer-based architectures, providing a robust and reproducible framework for high-fidelity audio separation.
Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real…
Abstract
Music production typically involves multiple editing operations that each require specialized tools and expertise. Existing text-guided music editing systems support few operations or instruments, operate on short clips, or handle only one operation per instruction, unlike real workflows where several changes are requested at once. Here, we present ProCraft, a lightweight 0.5B-parameter autoregressive codec language model that edits 30s music excerpts end-to-end from free-form instructions, including compound requests. ProCraft supports seven operations across eight instruments, from tasks with a unique target (e.g., track removal) to creative ones (e.g., accompaniment generation). We align intermediate representations with pretrained music embeddings and pitch and instrument targets to preserve content outside the edit, and gated cross-attention in every decoder layer to keep generation tied to the instruction. To train on compound requests, we introduce On-the-fly Editing-pair Synthesis, which generates paired instructions and audio combining track-level operations with effect changes. ProCraft performs competitively against task-specific systems on single-task editing, and outperforms four free-text editors on compound instructions for track removal, extraction and denoising on real (MoisesDB) and synthetic (Slakh) recordings. In a user study (16 listeners), ProCraft was rated highest on removal and denoising and on-par with a 16-times larger model on extraction and instrument transfer. Ablations further show that representation alignment and cross-attention reduce leakage from unwanted tracks and raise editing success rates, especially on out-of-domain data.
AI-generated analysis. Check claims against the paper.
Summary
ProCraft introduces a lightweight autoregressive codec language model capable of performing diverse and compound music editing tasks from free-form text instructions. The paper makes a significant contribution by addressing the challenge of preserving unedited content while applying multiple simultaneous edits, using novel architectural components like a persistent edit head and intermediate layer alignment, along with a synthetic data generation pipeline that enables training on compound instructions.
Methodology
The paper proposes ProCraft, a 0.5B parameter autoregressive codec language model for music editing. The methodology is sound, leveraging the InspireMusic backbone and WavTokenizer. The introduction of a "persistent edit head" with gated cross-attention to maintain instruction relevance over long sequences is a logical architectural choice for autoregressive models. The "On-the-fly Editing-pair Synthesis" is a clever data generation strategy using a tool-calling LLM to create compound instruction-audio pairs, addressing a significant data scarcity issue in this niche. The alignment of intermediate layers with MERT and pitch/instrument targets is a strong technical contribution to preserve unedited content.
Results & evaluation
The evaluation is comprehensive, covering seven operations across eight instruments. The use of both synthetic (Slakh) and real (MoisesDB) data for out-of-domain testing is rigorous. The metrics are well-chosen, combining general audio metrics (FAD, CLAP) with task-specific ones (SI-SDR, edit rate, preservation metrics). The user study, while small (16 listeners), provides necessary subjective validation. The model outperforms baselines on compound instructions, which is the core claim.
Reproducibility
The paper promises to release code and weights. The training procedure, including the two-pass training for the edit head, is described in detail. The data synthesis pipeline is explained, though the specific prompts and tool-calling logic for the LLM are likely in the appendix. The use of standard components (WavTokenizer, MERT) aids reproducibility.
Limitations
The model is limited to 30-second excerpts, which is short for full music production workflows. The user study is small. The model is trained primarily on synthetic data (Slakh) and tested on real data, which may limit generalization to complex, real-world mixes with non-standard instruments or production styles. The 0.5B parameter size, while efficient, may limit the complexity of creative edits compared to larger models.
Broader impact
This work contributes to the field of controllable audio generation by demonstrating that a single lightweight model can handle diverse and compound editing tasks. It bridges the gap between task-specific models and general-purpose text-guided editors. The data synthesis technique is applicable to other audio editing tasks. ProCraft introduces a lightweight autoregressive codec language model capable of performing diverse and compound music editing tasks from free-form text instructions. The paper makes a significant contribution by addressing the challenge of preserving unedited content while applying multiple simultaneous edits, using novel architectural components like a persistent edit head and intermediate layer alignment, along with a synthetic data generation pipeline that enables training on compound instructions.
Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for…
Abstract
Perceptual evaluation of generated music is essential for developing text-to-music (TTM) generation systems. Existing automatic music perceptual evaluation methods rely solely on the contrastive language audio pretraining (CLAP) model to extract both audio and text features for predicting human-annotated mean opinion scores (MOS), particularly for overall musical impression (OMI) and textual alignment (TA). However, CLAP is limited in capturing fine-grained musical attributes such as timbre and expressiveness \emph{etc}. To address this, we propose FUSEMOS, a dual-encoder fusion framework that integrates CLAP with MERT, a self-supervised model pretrained on large scale music audio, to better capture both high level semantic alignment and detailed musical characteristics for more accurate OMI and TA estimation. Furthermore, to overcome the limitations of conventional regression losses in measuring human preferred relative ranking of music segments, we design a novel ranking-aware composite loss function. Experiments conducted on the MusicEval benchmark demonstrate that our approach outperforms the existing SOTA methods in both mean squared error and rank correlation metrics by a large margin.
AI-generated analysis. Check claims against the paper.
Summary
FUSEMOS proposes a dual-encoder fusion framework combining CLAP and MERT with a ranking-aware composite loss to improve the accuracy of automatic perceptual evaluation for text-to-music generation systems. The paper demonstrates significant improvements in rank correlation metrics over single-encoder baselines on the MusicEval benchmark, validating the complementary nature of semantic and structural audio representations for MOS prediction.
Methodology
The paper proposes FUSEMOS, a dual-encoder framework that fuses features from CLAP (for semantic alignment) and MERT (for musical structure) to predict MOS scores for Overall Musical Impression (OMI) and Textual Alignment (TA). The core architectural contribution is a "late fusion" strategy where independent MLP heads predict scores from each encoder, which are then combined via a linear layer. This is a standard ensemble technique rather than a deep feature-level fusion. The loss function combines a truncated regression loss (Huber-like) with a contrastive ranking loss. While the motivation to capture ordinal preferences is sound, the specific formulation of the ranking loss is a minor adaptation of existing contrastive learning objectives. The methodology is competent but relies heavily on existing pretrained models without significant architectural innovation in the fusion mechanism itself.
Results & evaluation
Experiments are conducted on the MusicEval benchmark, which is a relevant and recent dataset for TTM evaluation. The results show substantial improvements over the CLAP-only baseline, particularly in rank correlation metrics (SRCC, KTAU). The ablation studies are thorough, validating the necessity of both encoders, the choice of late fusion over early fusion/attention, and the components of the composite loss. The improvement in S-SRCC from 0.861 to 0.940 for TA is significant. However, the comparison is limited to the baseline provided by the dataset creators; comparisons with other recent MOS prediction methods (e.g., UTMOS variants adapted for music, or other dual-encoder approaches) are absent.
Reproducibility
The paper provides specific hyperparameters (learning rate, batch size, margins) and references specific pretrained checkpoints (HuggingFace links for CLAP and MERT). The use of a single H800 GPU and SGD optimizer is noted. However, the code repository is not provided in the text, and the specific implementation details of the "learnable query-based attention pooling" in the MERT branch are described briefly. The reliance on a specific, potentially proprietary or limited-access dataset (AISHELL_7A) may hinder full reproduction by the broader community if the dataset is not widely available.
Limitations
The primary limitation is the lack of generalization testing. The model is trained and tested on the MusicEval benchmark, which consists of clips from 31 specific models. It is unclear how well FUSEMOS generalizes to music generated by unseen models or different TTM architectures. Additionally, the "late fusion" approach is computationally redundant as it requires running two large encoders and multiple MLP heads, which could be a bottleneck for real-time evaluation. The paper does not discuss the computational cost or latency of the inference process.
Broader impact
This work contributes to the development of objective metrics for text-to-music generation, which is a rapidly growing field. By improving the correlation between automatic scores and human perception, it enables more efficient iteration and benchmarking of TTM systems. The integration of musical structure models (MERT) with semantic alignment models (CLAP) sets a precedent for multimodal evaluation frameworks. However, the impact is currently limited to the specific domain of TTM evaluation and does not extend to broader audio quality assessment tasks. FUSEMOS proposes a dual-encoder fusion framework combining CLAP and MERT with a ranking-aware composite loss to improve the accuracy of automatic perceptual evaluation for text-to-music generation systems. The paper demonstrates significant improvements in rank correlation metrics over single-encoder baselines on the MusicEval benchmark, validating the complementary nature of semantic and structural audio representations for MOS prediction.
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely…
Abstract
Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
AI-generated analysis. Check claims against the paper.
Summary
LiveFoley introduces a streaming causal autoregressive diffusion architecture that achieves imperceptible latency and high-quality audio-visual alignment by decoupling frontend look-ahead from receptive field size via a Trailing Sliding Window and enhancing backend denoising with historical and future-warmup contexts. The technical contribution is significant for the field of real-time multimodal generation, offering a practical solution to the latency-quality trade-off that has hindered the adoption of high-fidelity V2A models in interactive environments.
Methodology
The paper proposes LiveFoley, a streaming video-to-audio generation system that addresses the latency-quality trade-off in real-time applications. The core contribution is the "Trailing Sliding Window" mechanism, which allows non-causal segment-level encoders (like Synchformer) to operate in a streaming context without retraining by anchoring the window tail to the current frame and using first-frame padding. This decouples look-ahead latency from the receptive field size. Additionally, the "Unified Varhop Framework" introduces a Synchronizer module to handle variable hop sizes, allowing the system to adapt to different latency budgets dynamically. For the audio backend, the authors modify the Local Diffusion Transformer (LocDiT) by introducing "History LocDiT" (using past context) and "Future-Warmup LocDiT" (using approximate future conditions) to improve generation quality compared to isolated patch denoising. The methodology is sound and logically addresses the identified bottlenecks in existing streaming V2A systems.
Results & evaluation
The paper includes a user study to quantify human perceptual thresholds for audio-visual delay, which is a valuable contribution to the field. Experiments demonstrate that LiveFoley matches offline models in temporal alignment while maintaining low latency. It claims to lead streaming systems in generation quality. The use of a single A800 GPU for real-time performance is a strong practical result. However, the text provided is truncated, so the full extent of quantitative comparisons (e.g., FAD, CLAP scores) against specific baselines like V-AURA or SoundReactor is not fully visible, though the abstract and intro suggest comprehensive testing.
Reproducibility
The paper provides architectural details and a project page. The use of standard components (MiniCPM, Synchformer, flow-matching) aids reproducibility. The specific implementation details of the Synchronizer and the training protocol for the variable hop sizes are described, but code availability is not explicitly confirmed in the provided text (only a project page URL is given).
Limitations
The system relies on a causal LM backbone which may still struggle with very long-range dependencies compared to global bidirectional models. The "Future-Warmup" mechanism introduces a small amount of additional latency (0.08s) which, while below perceptual thresholds, adds complexity. The evaluation is primarily focused on latency and alignment; subjective audio quality metrics (MOS) are likely included in the full paper but are not detailed in the truncated text.
Broader impact
This work has significant potential for immersive interactive applications such as AR/VR, video games, and live streaming, where real-time audio feedback is crucial. By providing a plug-and-play sound engine that works with arbitrary visual pipelines, it lowers the barrier for integrating high-quality synchronized audio into interactive media. LiveFoley introduces a streaming causal autoregressive diffusion architecture that achieves imperceptible latency and high-quality audio-visual alignment by decoupling frontend look-ahead from receptive field size via a Trailing Sliding Window and enhancing backend denoising with historical and future-warmup contexts. The technical contribution is significant for the field of real-time multimodal generation, offering a practical solution to the latency-quality trade-off that has hindered the adoption of high-fidelity V2A models in interactive environments.
Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from…
Abstract
Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.
AI-generated analysis. Check claims against the paper.
Summary
The paper proposes CRESiST, an end-to-end interleaved read-write framework for simultaneous speech translation that leverages an omni-modal LLM and an Actor-Critic self-enhancement pipeline to optimize streaming policies without rigid offline alignments. This work represents a strong technical contribution to the field of LLM-integrated audio processing, addressing key bottlenecks in latency and cache efficiency for infinite-length inference. While it does not reach the level of a paradigm-shifting landmark like the Transformer or BERT, it offers a significant architectural improvement over existing prefix-incremental methods, establishing new state-of-the-art Pareto frontiers for quality and latency. The integration of reinforcement learning (Actor-Critic) for policy distillation in a speech translation context is a novel and meaningful advance, likely to be cited by researchers working on streaming speech models and real-time translation systems.
A speech deepfake detector estimates whether a recording is synthetic, but a forensic analyst also asks which system produced it, whether recordings share a source, and what evidence supports each conclusion. We present VeriSpoof, an audio language model that unifies five tasks…
Abstract
A speech deepfake detector estimates whether a recording is synthetic, but a forensic analyst also asks which system produced it, whether recordings share a source, and what evidence supports each conclusion. We present VeriSpoof, an audio language model that unifies five tasks with shared weights: deepfake detection, generator attribution (generator, acoustic model and vocoder), recording-condition analysis, synthesized-span localisation, and source verification. Each decision is supported by grounded rationales whose statements can be independently verified. Answers are scored by sentence likelihood, allowing detection and source verification to be thresholded without separate heads. Rationale statements can be checked against waveform measurements, span labels, or generator metadata. Counterfactual examples train decisions to follow the stated evidence. Across the 14 Speech-DF-Arena test sets, VeriSpoof-Plus achieves 2.14% pooled EER, ranking third of 28 systems and first among open-weight systems. VeriSpoof-Base, trained only on official training splits of eight corpora, achieves 4.43% EER, less than half that of the next open-weight detector. With the same weights, VeriSpoof-Plus verifies whether recordings from generators unseen in training share a source at 6.2% EER. Controlled studies show that the audio representation governs detection, the LLM governs recording comparison, and source supervision is necessary for neural-codec fakes. Statement-checkable rationales catch partial fakes missed by one-step answers (99.6% vs. 78.4% on PartialEdit) and determine decisions under edits unseen in training (99.5%). Training updates only 29.7M of 2.85B parameters, fitting a single 24 GB GPU with gradient checkpointing. We release both models, inference code, a demo, and VeriSpoof-Bench, the evaluation suite covering these questions.
AI-generated analysis. Check claims against the paper.
Summary
VeriSpoof unifies multiple speech forensics tasks into a single audio language model with verifiable, grounded rationales. The paper makes a substantial contribution by demonstrating that LLMs can perform complex forensic reasoning (attribution, source verification) with high accuracy when equipped with checkable intermediate steps, outperforming specialized detectors on open-weight benchmarks and providing a new standard for explainable AI in audio security.
Methodology
The paper proposes VeriSpoof, an audio language model (ALM) that unifies five distinct forensic tasks (deepfake detection, generator attribution, recording-condition analysis, synthesized-span localisation, and source verification) into a single architecture. The core methodological innovation is the "grounded rationale" framework, where the model generates explanations composed of fixed-field statements (e.g., SNR, bandwidth, span labels) that can be independently verified against waveform measurements or metadata. This addresses the "faithfulness" problem in LLM explanations by training with counterfactual examples that force the decision to depend on the stated evidence rather than just the audio input. The model uses a Qwen3-ASR encoder and MiniCPM5-2B LLM, employing parameter-efficient fine-tuning (LoRA/DoRA) on only 29.7M of 2.85B parameters. The scoring mechanism reads the likelihood of specific answer sentences, eliminating the need for separate classification heads for different tasks.
Results & evaluation
The experimental evaluation is extensive and rigorous. On the Speech-DF-Arena benchmark (14 test sets), VeriSpoof-Plus achieves a pooled EER of 2.14%, ranking third overall and first among open-weight systems. VeriSpoof-Base, trained on fewer data, achieves 4.43% EER, significantly outperforming other open-weight detectors. The paper includes controlled ablation studies to determine the contribution of the encoder vs. the LLM, showing that the audio representation governs detection while the LLM governs source comparison. It also demonstrates that source supervision is critical for detecting neural-codec fakes. The rationale mechanism is shown to catch partial fakes (99.6% vs 78.4% for one-step answers) and maintain decision faithfulness under unseen edit types (99.5%).
Reproducibility
High. The authors release both model weights (Base and Plus), inference code, a demo, and a comprehensive evaluation suite (VeriSpoof-Bench) with 177,852 items. The training recipe, data composition, and hyperparameters are detailed. The claim that training fits on a single 24GB GPU with gradient checkpointing is supported by memory analysis, making the work accessible to a broader research community.
Limitations
The model relies on specific base models (Qwen3-ASR, MiniCPM5-2B) which may not be the most optimal for all audio tasks. The "grounded rationale" approach requires careful construction of checkers for each statement type, which may not generalize automatically to new forensic questions without manual checker design. The performance on unseen languages (SpoofRadar) shows a calibration cost where pooled metrics hide per-set threshold issues. The paper is a preprint (arXiv) and has not yet undergone peer review at a major conference.
Broader impact
This work has significant implications for digital forensics and media integrity. By providing explainable, verifiable, and multi-functional deepfake detection, it moves the field beyond simple binary classification toward actionable forensic analysis. The ability to attribute generators and verify sources with grounded evidence could aid law enforcement and platform moderation. The release of open-weight models and benchmarks lowers the barrier to entry for developing forensic audio tools. VeriSpoof unifies multiple speech forensics tasks into a single audio language model with verifiable, grounded rationales. The paper makes a substantial contribution by demonstrating that LLMs can perform complex forensic reasoning (attribution, source verification) with high accuracy when equipped with checkable intermediate steps, outperforming specialized detectors on open-weight benchmarks and providing a new standard for explainable AI in audio security.
Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two…
Abstract
Speaker-recognition benchmarks test how accurately a model recovers the speaker label of a recording, treating voice identity as a property of the person who produced it. However, these benchmarks do not evaluate how humans perceive voice identity. Whether a listener hears two voices as the same person has consequences for intellectual property rights, cybersecurity, and everyday life. We introduce the Voice Identity Perception benchmark (VIPBench), which contains 124,876 same-or-different judgments from 1,290 listeners on 9,800 recording pairs from 100 public figures. The pairs include real recordings, voice clones generated by a text-to-speech system, and morphs between two speakers' voices. We also define four tasks that evaluate the alignment between speaker-representation models and human identity perception. The first two tasks measure how well a model's similarity score predicts the proportion of listeners who respond "same" and the majority response. The other two measure whether a model and listeners find the same speaker pairs similar, and whether a model's relationship to listener judgments holds equally for real and synthetic recordings. We evaluate eleven publicly available speech models. Listener judgments diverge from speaker labels most often for voice clones: in 41% of responses, listeners judged a clone and a real recording of the same speaker to be different people. No model reaches the noise ceiling; the best model explains 61% of the explainable variance. Models trained with speaker labels align with listeners better than self-supervised models do, yet a model with a 12% error rate on speaker labels aligns with listeners as well as models with error rates below 2%. VIPBench enables the evaluation of speaker models against human perception on real, cloned, and morphed speech, and our results motivate training speaker models directly on listener judgments.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces VIPBench, a comprehensive benchmark for evaluating the alignment between speech representation models and human perception of voice identity. It makes a significant contribution to the field by providing a large-scale dataset of human judgments on real, cloned, and morphed voices, along with a rigorous evaluation protocol that reveals the limitations of current speaker verification models in capturing human perceptual nuances.
Methodology
The paper introduces VIPBench, a large-scale benchmark (124,876 judgments) designed to evaluate how well speech representation models align with human perception of voice identity, rather than just metadata-based speaker labels. The methodology is rigorous, involving a stratified sampling of 100 public figures across sociophonetic groups, genders, and ages. The stimuli include real recordings, state-of-the-art voice clones (Cartesia Sonic 3), and continuous morphs. The evaluation protocol defines four distinct tasks: continuous prediction of perceived sameness, Platt-calibrated verification, speaker-level representational similarity, and real-to-synthetic calibration transfer. The use of cross-validation over speakers and the inclusion of a noise ceiling based on inter-rater reliability demonstrates a sophisticated statistical approach to benchmarking.
Results & evaluation
The experiments evaluate 11 publicly available models, including supervised (x-vector, ECAPA-TDNN, Resemblyzer) and self-supervised (wav2vec 2.0, HuBERT, WavLM, XLS-R) models, as well as Whisper. The results reveal that while supervised models generally outperform self-supervised ones in aligning with human perception, none reach the noise ceiling. A key finding is the significant divergence between metadata labels and human perception for voice clones, where listeners often perceive clones as different speakers. The paper provides detailed ablations on layer selection for SSL models and robustness checks using unfamiliar trials and different TTS systems (CosyVoice 2). The inclusion of demographic fairness analysis is a strong addition, showing consistent performance across sociophonetic groups for most models.
Reproducibility
The paper is highly reproducible. The dataset, code, and embeddings are released on Hugging Face. The authors provide detailed information on stimulus generation, including the specific TTS settings and random seeds used for morphing. The analysis notebook is provided, allowing users to reproduce the results without needing to re-extract embeddings. The clear documentation of the IRB protocol and data collection process further enhances trust and reproducibility.
Limitations
The benchmark is limited to American English speakers and public figures, which may not generalize to other languages or private individuals. The reliance on a single commercial TTS system (Cartesia) for the main set of clones and morphs could introduce system-specific biases, although a small extension with CosyVoice 2 mitigates this partially. The lack of test-retest reliability for individual listeners is a minor limitation, though the high inter-rater reliability compensates for this.
Broader impact
This work has significant implications for the development of voice biometric systems, cybersecurity, and intellectual property protection. By highlighting the gap between metadata-based speaker verification and human perception, the paper motivates the training of speaker models directly on human perceptual judgments. This could lead to more robust and human-aligned voice authentication systems that are better equipped to handle voice cloning attacks. The benchmark also provides a valuable resource for researchers studying human voice perception and the alignment of machine representations with human cognition. The paper introduces VIPBench, a comprehensive benchmark for evaluating the alignment between speech representation models and human perception of voice identity. It makes a significant contribution to the field by providing a large-scale dataset of human judgments on real, cloned, and morphed voices, along with a rigorous evaluation protocol that reveals the limitations of current speaker verification models in capturing human perceptual nuances.
Dialectal Arabic speech recognition remains constrained by the scarcity of dialect-resolved transcribed speech despite the abundance of web audio. We introduce GigaSpeech 3, a corpus of 39,000 hours and 10 million weakly transcribed speech segments covering Algerian, Egyptian…
Abstract
Dialectal Arabic speech recognition remains constrained by the scarcity of dialect-resolved transcribed speech despite the abundance of web audio. We introduce GigaSpeech 3, a corpus of 39,000 hours and 10 million weakly transcribed speech segments covering Algerian, Egyptian, Iraqi, Saudi, and Syrian Arabic. Starting from 60 hours of manual transcription per dialect, we train labeling models using speech-recognition-biased self-supervised learning and apply them to dialect-matched web audio. Data selection and four rounds of noisy-student refinement for Saudi and Syrian Arabic yield the final corpus. We validate its utility through dialect-specific and multi-dialect speech recognition training and cross-domain evaluation. On GigaSpeechBench, the final dialect-specific models outperform their labeling models across all five dialects and achieve a macro-average word error rate of 23.86%, a 16.6% relative reduction from the strongest third-party system covering all five dialects, while outperforming the best third-party result on three dialects. A multi-dialect model using self-predicted dialect identities achieves a macro-average word error rate of 25.51% while also supporting Modern Standard Arabic. Across eight conditions from three external corpora, a GigaSpeech 3-trained Zipformer achieves the lowest word error rate among six evaluated models in every condition, while adapting the Qwen3 speech recognizer on GigaSpeech 3 improves performance across all eight conditions. We release the corpus and recipes as a foundation for dialect-aware Arabic speech processing.
AI-generated analysis. Check claims against the paper.
Summary
The paper presents GigaSpeech 3, a large-scale weakly labeled corpus for five Arabic dialects, constructed via ASR-biased self-supervised learning and noisy-student refinement, demonstrating that such weak supervision enables compact models to outperform strong third-party systems and large pretrained models in dialect-specific and multi-dialect recognition tasks.
Methodology
The paper proposes a data-centric approach to low-resource dialectal Arabic ASR. The core methodology involves constructing a large-scale weakly labeled corpus (GigaSpeech 3) using a "VietASR" pipeline. This pipeline leverages ASR-biased self-supervised learning (SSL) to train labeling models from small manual seeds (60 hours/dialect), which are then applied to dialect-matched web audio. The process includes data selection and iterative Noisy Student Training (NST) refinement for specific dialects (Saudi, Syrian). The novelty lies not in a new model architecture, but in the rigorous application of existing SSL and weak-labeling techniques to a specific, under-served linguistic domain (Arabic dialects) at a significant scale (39k hours). The validation strategy is sound: training independent downstream models (Zipformer) from scratch on the weak labels to prove the utility of the data beyond the labeling model's parameters.
Results & evaluation
The experimental evaluation is comprehensive and rigorous. The authors validate the corpus by showing that models trained on the weak labels outperform the labeling models themselves, confirming the data's utility. They compare against strong third-party baselines (including commercial APIs and large open-source models like Qwen3 and Whisper) on GigaSpeechBench, achieving state-of-the-art results for compact models. Cross-domain evaluation on MASC, Casablanca, and SADA further demonstrates generalizability. The inclusion of a multi-dialect model with automatic language identification (LID) and the analysis of routing errors adds depth to the evaluation. The results are consistent and well-supported by ablation studies (e.g., Labeled-only vs. SSL+FT vs. Weak-labels).
Reproducibility
High. The authors release the code, dataset, and demo. The pipeline is described in detail, including the specific stages of the VietASR recipe and the NST refinement process. The use of standard architectures (Zipformer) and public benchmarks (GigaSpeechBench) enhances reproducibility. The separation of training and test channels at the source level is a strong methodological choice that is clearly documented.
Limitations
The primary limitation is the reliance on web audio, which may contain noise, code-switching, or non-dialectal content that is difficult to fully filter. The country-level dialect labels are coarse and may not capture intra-country variation. The multi-dialect model's performance is affected by LID errors, particularly for Egyptian and Syrian dialects, as shown in the confusion matrix. The paper focuses on five specific dialects, leaving other Arabic varieties unaddressed.
Broader impact
This work has significant impact on the field of low-resource speech recognition. By providing a large, dialect-resolved corpus and a validated pipeline for weak labeling, it enables the development of more accurate and inclusive ASR systems for Arabic speakers. The methodology can be adapted to other low-resource languages or dialects. The release of the corpus and tools will likely accelerate research in dialectal Arabic processing and serve as a benchmark for future work. The paper presents GigaSpeech 3, a large-scale weakly labeled corpus for five Arabic dialects, constructed via ASR-biased self-supervised learning and noisy-student refinement, demonstrating that such weak supervision enables compact models to outperform strong third-party systems and large pretrained models in dialect-specific and multi-dialect recognition tasks.
Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical…
Abstract
Multimodal language models continue to improve on broad music understanding benchmarks, but which musical features can they perceive and reason about? We investigate this question by introducing PianoBench, a benchmark that tasks language models with reasoning about musical performances across congruent acoustic and symbolic input representations. PianoBench is derived from our new multimodal, training-scale dataset comprising more than 20,000 hours of interleaved transcripts of music and spoken musical commentary from instructional piano videos. Open-weight audio-language models achieve at most 52.8% accuracy on PianoBench, and at best only modestly outperform their accuracy on a no-audio control. In contrast, frontier models using chain of thought achieve up to 93.2% accuracy when provided transcripts of music as text, exposing a gap between perceiving musical features from audio and reasoning about musical events when they are given explicitly. We then test whether mid-training on interleaved musical instruction can improve question answering without chain of thought, raising Qwen 3.5 9B Base's zero-shot accuracy on PianoBench with music transcripts from 42.8% to 56.0% after instruction tuning. These experiments suggest that instructional videos may represent an overlooked source of multimodal supervision for music. Available: https://gclef-cmu.org/pianobench.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces PianoBench, a benchmark and dataset for evaluating musical perception and reasoning in language models, revealing a significant gap between audio perception and symbolic reasoning capabilities in current models. Through a rigorous data pipeline and comprehensive experiments, the authors demonstrate that while frontier models excel at reasoning over explicit note events, open-weight models struggle with audio perception, and that interleaved musical instruction can improve zero-shot reasoning without chain-of-thought.
Methodology
The paper introduces a novel data pipeline to construct a large-scale (20k hours) dataset of interleaved musical commentary and symbolic transcripts from instructional piano videos. The methodology is robust, utilizing source separation, speech recognition, and MIDI transcription to create aligned multimodal data. The benchmark construction is rigorous, employing LLMs for question generation and distractor creation, followed by human verification to ensure quality. The proposed "MIDI-text" tokenization scheme is a clever contribution, allowing text-only LLMs to process symbolic music efficiently. The experimental design effectively isolates perception (audio) from reasoning (symbolic) by comparing performance across different input modalities.
Results & evaluation
The evaluation is comprehensive, testing a wide range of open-weight and proprietary models. The key finding—that open-weight models struggle with audio perception while frontier models excel at symbolic reasoning with CoT—is well-supported by the data. The mid-training experiment on Qwen 3.5 9B Base provides strong evidence that interleaved instruction data improves zero-shot reasoning without CoT, outperforming separated data. The inclusion of a no-music control is crucial for validating the necessity of the musical input.
Reproducibility
High. The authors release the dataset, benchmark, and code. The data pipeline is described in detail, including the specific models used for transcription and separation. The hyperparameters for mid-training are provided. The use of specific commercial APIs (GPT 5.6, Gemini 3.1) limits full reproducibility for those specific results, but the open-weight model results are reproducible.
Limitations
The benchmark is limited to piano music, which may not generalize to other instruments or genres. The reliance on YouTube instructional videos may introduce biases in repertoire and teaching style. The "MIDI-text" representation, while efficient, is a lossy compression of the original audio and may not capture timbral or expressive nuances. The small size of the benchmark (250 questions) limits statistical power for fine-grained analysis.
Broader impact
This work highlights a critical gap in current audio-language models: the ability to reason about music when given symbolic representations versus perceiving it from audio. It provides a valuable resource (PianoTalk dataset) for training models on fine-grained musical understanding. The findings suggest that current open-weight models need better supervision for audio perception, while frontier models' reasoning capabilities can be leveraged to generate better training data. This could drive future research in multimodal music understanding and the development of more perceptually grounded audio models. The paper introduces PianoBench, a benchmark and dataset for evaluating musical perception and reasoning in language models, revealing a significant gap between audio perception and symbolic reasoning capabilities in current models. Through a rigorous data pipeline and comprehensive experiments, the authors demonstrate that while frontier models excel at reasoning over explicit note events, open-weight models struggle with audio perception, and that interleaved musical instruction can improve zero-shot reasoning without chain-of-thought.
Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely…
Abstract
Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely separate. We present Spatial Transfusion, a unified first-order ambisonics (FOA) framework that extends a pretrained spatial audio understanding model to generation and editing through flow matching over VAE latents. We formulate editing as a structured scene transformation over a taxonomy of atomic edits. Keeping the pretrained backbone frozen, we introduce modality-based LoRA routing and a curriculum that progressively adds generation, editing, and natural-language interaction. Experiments show that Spatial Transfusion largely preserves spatial understanding while outperforming full fine-tuning in generation and editing. The resulting model supports controllable FOA generation and editing through both structured and natural-language interfaces. Audio samples are available at https://microsoft.github.io/spatial-transfusion/
AI-generated analysis. Check claims against the paper.
Summary
Spatial Transfusion introduces a unified framework for FOA audio understanding, generation, and editing by extending a pretrained spatial audio language model with flow matching and modality-based LoRA routing. The paper demonstrates that this approach preserves spatial understanding while enabling high-quality, controllable generation and editing, outperforming full fine-tuning baselines and establishing a new paradigm for spatial audio manipulation.
Methodology
The paper proposes Spatial Transfusion, a unified framework for First-Order Ambisonics (FOA) audio understanding, generation, and editing. The core methodological contribution is the adaptation of a pretrained spatial audio language model (Sci-Phi) using a "Transfusion" paradigm, which combines autoregressive text modeling with flow-matching for continuous audio latents. A key technical innovation is the use of modality-based LoRA routing, where separate LoRA adapters are applied to audio and text tokens, allowing the frozen backbone to learn generative capabilities without catastrophic forgetting of understanding tasks. The authors define a structured taxonomy of atomic edits (semantic, acoustic, spatial, temporal) and employ a four-stage curriculum to progressively introduce generation and editing capabilities. The approach effectively leverages the structured nature of spatial scenes to enable precise control over generation and editing, distinguishing it from generic audio generation models.
Results & evaluation
The experimental setup is rigorous, utilizing a held-out set from the FOA-MEIR dataset with real room impulse responses. The evaluation metrics are comprehensive, covering spatial fidelity (FAD-PSELD, FOA covariance), semantic alignment (CLAP), and editing-specific metrics (EditFLAM). The results demonstrate that the proposed LoRA routing strategy outperforms full fine-tuning in generation and editing while preserving understanding capabilities. Ablation studies on routing strategies (modality-based vs. task-based vs. dense) provide strong evidence for the effectiveness of the chosen approach. The inclusion of natural language instruction tuning (Stage 4) further validates the flexibility of the framework.
Reproducibility
The paper provides detailed descriptions of the architecture, training curriculum, and hyperparameters. However, specific details on the FOA-VAE implementation and the exact simulation pipeline parameters are referenced to prior work (Sci-Phi, FOA-VAE), which may limit immediate reproducibility without access to those codebases. The availability of a demo page with audio samples is a positive indicator, but a full code release is not explicitly mentioned in the text provided.
Limitations
The model is trained primarily on simulated data, which may limit its generalization to real-world noisy environments, although the evaluation uses real RIRs. The framework is specific to FOA, and extension to higher-order ambisonics or binaural audio is not explored. The editing operations are limited to the defined atomic taxonomy, and complex, multi-step edits may not be supported. The reliance on a specific pretrained backbone (Phi-4-MM) may constrain the model's scalability.
Broader impact
This work bridges the gap between spatial audio understanding and generation, offering a unified interface for immersive audio creation. The structured editing capabilities have potential applications in virtual reality, gaming, and film production, where precise control over soundscapes is crucial. The framework's ability to handle both structured JSON and natural language inputs makes it accessible to both technical and non-technical users. Spatial Transfusion introduces a unified framework for FOA audio understanding, generation, and editing by extending a pretrained spatial audio language model with flow matching and modality-based LoRA routing. The paper demonstrates that this approach preserves spatial understanding while enabling high-quality, controllable generation and editing, outperforming full fine-tuning baselines and establishing a new paradigm for spatial audio manipulation.
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source…
Abstract
Video dubbing aims to synthesize lip movements that match target speech while preserving the speaker identity, head pose, and background of a source video. Existing editors are predominantly trained via self-reconstruction under original paired audio, even though the source mouth becomes a conflicting target when driving speech is replaced. We introduce replacement-aware cross-audio supervision: paired-audio samples use full-frame reconstruction, whereas unpaired replacement-audio samples mask out source-mouth reconstruction losses and supervise the mouth solely through audio-visual synchronization while preserving unmasked facial regions and background. Because offline dubbing provides the complete source video and target speech beforehand, we present VidDub, a full-context diffusion editor distilled into two denoising steps (2 NFE). Our replacement-aware teacher first learns accurate lip synchronization without regressing toward conflicting source mouth movements. Matched trajectory regression then transfers the teacher's CFG-guided trajectories to a two-step student, and distribution matching distillation recovers visual quality after step compression. We will release the curated training dataset, VidDub-Bench, fixed paired- and cross-audio evaluation manifests, preprocessing outputs, and unified evaluation code. Across three training seeds, masking out conflicting source-mouth reconstruction improves cross-audio Sync-C by 0.833; the final two-step model combines strong lip synchronization with superior visual quality, identity preservation, and visual speech recognition across four benchmarks.
AI-generated analysis. Check claims against the paper.
Summary
The paper introduces a replacement-aware distillation framework for two-step video dubbing, effectively resolving the conflict between source mouth reconstruction and target audio synchronization through masked supervision. The technical contribution is significant, combining a novel training objective with efficient distillation to achieve state-of-the-art performance in lip synchronization and visual quality, supported by a comprehensive evaluation and the release of a new benchmark.
Methodology
The paper proposes a two-stage approach for video dubbing: a replacement-aware teacher training strategy and a two-step distillation pipeline. The core methodological contribution is the "replacement-aware cross-audio supervision," which addresses the conflict between source mouth reconstruction and target audio synchronization by masking out source-mouth reconstruction losses during training on unpaired audio. This is a logical and effective solution to a known problem in video editing models. The distillation process uses matched trajectory regression and distribution matching to compress a 48-step teacher into a 2-step student, which is a standard but well-executed application of recent diffusion distillation techniques. The architecture relies on Wan-1.3B with VACE conditioning, which is a solid baseline.
Results & evaluation
The experimental evaluation is comprehensive, covering multiple benchmarks (HDTF, TalkVid-Bench, VidDub-Bench) and metrics (Sync-C, Sync-D, FVD, SSIM, PSNR, VSR). The paper demonstrates state-of-the-art performance in cross-audio synchronization while maintaining high visual quality. The ablation studies clearly show the benefit of the masking strategy. The comparison with both feed-forward and multi-step diffusion methods is fair and informative. The release of a curated dataset and benchmark (VidDub-Bench) adds significant value to the community.
Reproducibility
The authors commit to releasing the training dataset, benchmark, preprocessing outputs, and unified evaluation code. The GitHub repository is provided. The detailed description of the training pipeline, loss functions, and distillation steps suggests high reproducibility, assuming the released code is complete and well-documented.
Limitations
The model is designed for offline dubbing, requiring the full source video and target speech beforehand, which limits its applicability to real-time or streaming scenarios. The reliance on a specific base model (Wan-1.3B) may limit generalizability to other architectures. The evaluation, while extensive, is primarily quantitative; subjective human evaluation (MOS) would further strengthen the claims about visual quality and naturalness.
Broader impact
This work has significant potential for applications in video localization, accessibility, and content creation. By enabling high-quality, fast (2-step) video dubbing, it can reduce the cost and time associated with traditional dubbing processes. The open-source release of the dataset and code will likely accelerate research in audio-visual synchronization and video editing. The paper introduces a replacement-aware distillation framework for two-step video dubbing, effectively resolving the conflict between source mouth reconstruction and target audio synchronization through masked supervision. The technical contribution is significant, combining a novel training objective with efficient distillation to achieve state-of-the-art performance in lip synchronization and visual quality, supported by a comprehensive evaluation and the release of a new benchmark.