Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
Primary: Microsoft Corporation
All Institutions: Wuhan University, Microsoft Corporation, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
The methodology proposes ImageTTS, a novel framework that repurposes a pretrained image diffusion transformer (FLUX.2) for speech synthesis. A key strength is the systematic disentanglement of two forms of vision-to-speech transfer: representation transfer via a frozen image VAE and generative-prior transfer via DiT initialization. The approach involves converting log-mel spectrograms into pseudo-RGB images, encoding them with the frozen FLUX.2 image VAE, and then fine-tuning only the pretrained DiT backbone using conditional latent flow matching. This pseudo-RGB conversion and subsequent VAE encoding is a creative way to bridge the modalities, though it relies on the image VAE's ability to meaningfully compress and reconstruct spectrograms. The formulation of zero-shot TTS as reference-conditioned latent infilling and instruction-guided TTS as full latent generation conditioned on text and description is well-defined. The use of classifier-free guidance for both text and speech conditions (and style for instruction-guided) is standard practice in generative models and appropriately applied here. The architecture leverages FLUX.2's double-stream and single-stream Transformer blocks for multimodal interaction, which is a sensible adaptation for handling both text and speech latents. The overall methodology is sound, innovative, and systematically designed to answer the research questions posed.
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio
Primary: University College London
All Institutions: University College London, King's College London
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
MFlowAudio introduces a novel latent audio generation framework that combines Flow Matching with a custom-designed TCMamba backbone. The core methodological innovation lies in the TCMamba block, which addresses the limitations of Mamba for 2D spectrograms by proposing a dual-scan mechanism: TimeMamba for long-range causal dependencies along the temporal axis (linear complexity) and ChannelMamba for cross-channel correlations like harmonic structures. This is a clever adaptation of SSMs to the audio domain, specifically designed for 2D latent representations. The formulation of Stateful Flow Matching (SFM) is another significant contribution, enabling chunk-wise training and streaming generation with O(1) caching complexity. This directly tackles the memory and computational bottlenecks of traditional diffusion/flow models for long-form audio. The theoretical analysis provided in the appendix, demonstrating the equivalence of chunked and one-time integration under certain causal assumptions, provides a strong foundation for SFM. Furthermore, the State-Decoupled Conditional Guidance (SDCG) mechanism is a novel approach to handle dynamic prompt switching, aiming to neutralize vector field collisions during on-the-fly semantic transitions. This mechanism, including the Orthogonal Score Projection, is well-motivated to balance prompt adherence and acoustic continuity. The overall architecture is clearly described, leveraging standard components like VAE for audio encoding and T5 for text encoding, integrated with the TCMamba blocks. The use of FiLM and cross-attention for conditioning is standard but well-integrated.
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.
Primary: Institute of Neuroinformatics, University of Zurich and ETH Zurich
All Institutions: Institute of Neuroinformatics, University of Zurich, ETH Zurich, Swiss Data Science Center, CVLab, EPFL
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
The paper proposes a rigorous methodology for constructing a physically verified benchmark for vocal attribution. The core innovation is the use of an on-body accelerometer-derived vibration signal to ground-truth caller identity, which is then withheld from the benchmark models to prevent shortcut learning. This is a sophisticated design choice that distinguishes the work from standard active speaker detection datasets that rely on perceptual annotation or close-talk references. The pipeline is detailed and auditable, involving synchronized multimodal data (5-channel audio, 3-camera video, 3D pose, radio telemetry) and a multi-stage verification process including manual review and geometric filtering. The evaluation protocol is particularly strong, defining three distinct regimes (session-disjoint, held-out-experiment, leave-one-individual-out) to explicitly separate in-domain recognition from transferable spatial reasoning.
The experimental evaluation is comprehensive, testing a wide array of baselines ranging from classical physics-based methods (SRP) to deep learning architectures (SELDnet++, EINV2, CST-former). The results clearly demonstrate the paper's central thesis: models that rely on familiar voice identity perform well in-domain but fail on unseen individuals, while spatially-aware models transfer better. The inclusion of modality intervention experiments (zeroing audio, shuffling geometry) provides strong causal evidence for what drives performance. The "on-body reference" ceiling confirms the consistency of the labels. The analysis of "close birds" and localization diagnostics adds depth to the evaluation.
Reproducibility is high. The authors release the full dataset, code for all evaluated methods, and fixed evaluation protocols. The paper provides extensive details on the data acquisition pipeline, camera calibration, and label construction in the appendices. The use of standard open-source tools (YOLOX, OC-SORT, Whisper) for preprocessing further aids reproducibility. The clear separation of the verification channel from the benchmark inputs ensures that future researchers can fairly evaluate new methods without access to the ground-truth sensor.
The primary limitation is the scale and diversity of the data. The dataset is limited to 10 zebra finches across 3 experiments in a single installation. This limits the generalizability of the findings to other species or larger, more dynamic groups. The radio telemetry features are limited due to carrier drift, which the authors acknowledge. The benchmark currently only covers single-vocalizer events, leaving multi-caller scenarios for the unevaluated V33DA++ release.
This work has significant impact on the fields of bioacoustics, multimodal learning, and behavioral analysis. By providing a physically verified benchmark, it enables the development of more robust and generalizable models for animal communication. The insights into shortcut learning and the importance of spatial reasoning are applicable to other multimodal attribution tasks, such as human active speaker detection. The release of the dataset and code will likely spur further research in this niche but important area. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
Recent advancements in audio generation have been largely driven by Transformer-based diffusion models. However, these models suffer from quadratic complexity of self-attention, severely bottlenecking the long-form audio synthesis. To overcome this limitation, we propose MFlowAudio, a novel latent audio generation framework that synergizes the continuous-time dynamics of Flow Matching with a custom-designed TCMamba backbone. TCMamba uses an innovative dual-scan mechanism: a TimeMamba module to capture long-range causal dependencies with linear complexity, and a ChannelMamba module to model cross-channel correlations such as harmonic structures. Exploiting this structural foundation, we formulate the Stateful Flow Matching (SFM) paradigm. This framework inherently enables chunk-wise training and streaming generation, maintaining an $\mathcal{O}(1)$ caching complexity without incurring extra computational overhead. To enable fine-grained controllable synthesis, we devise a novel guidance mechanism that neutralizes vector field collisions precipitated by on-the-fly semantic transitions of prompts. Comprehensive empirical evaluations confirm that MFlowAudio yields generative fidelity on par with state-of-the-art baselines while establishing significant computational efficiency, achieving above $14.5$% acceleration in generation and facilitating perceptually seamless streaming synthesis with a constant latency of $1.8$ seconds. Demo: https://huggingface.co/spaces/mflowaudio/MFlowAudio
Primary: University College London
All Institutions: University College London, King's College London
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
MFlowAudio introduces a novel latent audio generation framework that combines Flow Matching with a custom-designed TCMamba backbone. The core methodological innovation lies in the TCMamba block, which addresses the limitations of Mamba for 2D spectrograms by proposing a dual-scan mechanism: TimeMamba for long-range causal dependencies along the temporal axis (linear complexity) and ChannelMamba for cross-channel correlations like harmonic structures. This is a clever adaptation of SSMs to the audio domain, specifically designed for 2D latent representations. The formulation of Stateful Flow Matching (SFM) is another significant contribution, enabling chunk-wise training and streaming generation with O(1) caching complexity. This directly tackles the memory and computational bottlenecks of traditional diffusion/flow models for long-form audio. The theoretical analysis provided in the appendix, demonstrating the equivalence of chunked and one-time integration under certain causal assumptions, provides a strong foundation for SFM. Furthermore, the State-Decoupled Conditional Guidance (SDCG) mechanism is a novel approach to handle dynamic prompt switching, aiming to neutralize vector field collisions during on-the-fly semantic transitions. This mechanism, including the Orthogonal Score Projection, is well-motivated to balance prompt adherence and acoustic continuity. The overall architecture is clearly described, leveraging standard components like VAE for audio encoding and T5 for text encoding, integrated with the TCMamba blocks. The use of FiLM and cross-attention for conditioning is standard but well-integrated.
The experimental evaluation is comprehensive and rigorous. The authors compare MFlowAudio against state-of-the-art full-band T2A models (AudioLDM2-48K, Stable Audio Open, TangoFlux) and an autoregressive 16kHz model (AudioGen) using a wide array of objective metrics: CLAP Score, PANNs-based Inception Score, VGGish/OpenL3-based FAD, and KL divergence. The results demonstrate that MFlowAudio achieves generative fidelity highly competitive with SOTA, even outperforming TangoFlux on several key acoustic quality metrics (FD, FADv, FADopenl3, KL divergence) with a near equivalent parameter budget. A notable strength is the detailed evaluation of computational efficiency, showing significant acceleration (81.8% speedup over TangoFlux) and the practical realization of perceptually seamless streaming generation with a constant latency of 1.8 seconds (Time-To-First-Chunk
Recent progress in visual scene generation has extended beyond static 3D scenes toward the more challenging problem of generating dynamic 4D scenes from minimal input. However, sound, despite being intrinsically coupled with dynamic objects and events, remains largely unmodeled in prior work. We introduce Video-to-4DAVScene, a new task that turns a video into an explorable, time-varying 4D audio-visual scene, and present the SonoWorld4D framework for this task. Given a monocular video, our method generates a 360 static background, reconstructs explicit motion trajectories of dynamic, sound-emitting foreground objects, and synthesizes a physically-grounded sound field consistent across multiple sources through dual-space audio diffusion synchronization. The resulting 4D scene supports real-time novel-view visual and spatial audio rendering. Evaluation on real-world dynamic scenes and a user study demonstrates SonoWorld4D generates 4D audio-visual scenes with perceptually realistic, physically grounded spatial audio consistent with scene geometry and visual content.
Primary: University of Maryland, College Park
All Institutions: University of Maryland, College Park
The paper introduces a novel task of Video-to-4DAVScene and a framework that effectively synchronizes visual dynamics with spatial audio using dual-space diffusion. The technical approach is rigorous, combining explicit 4D geometry with physically grounded acoustic rendering, and the introduction of a new dataset and evaluation protocol significantly advances the field of dynamic audio-visual scene generation.
The paper proposes SonoWorld4D, a framework for generating explorable 4D audio-visual scenes from a single monocular video. The methodology is technically sophisticated, relying on a decomposed 4D visual representation (static background Gaussians + dynamic foreground objects with explicit trajectories) and a novel "dual-space diffusion synchronization" mechanism. This synchronization couples camera-space audio generation with object-space audio generation via forward and inverse acoustic rendering operators during the diffusion sampling process. The use of flow-matching and the specific bidirectional consistency constraints (rendering object sounds to camera space and separating/inverting camera sounds to object space) represent a strong technical contribution to the field of spatial audio generation. The integration of VLM-derived prompts for acoustic context is also a relevant modern addition.
The authors introduce a new dataset (SonoScene4D) with 54 clips of dynamic scenarios featuring moving sound sources, recorded with synchronized FOA audio at novel listener poses. This is a valuable contribution to the field, as existing datasets often lack dynamic spatial audio ground truth. The evaluation includes objective metrics (DoA errors, CC, AUC, CLAP) and a user study with 50 participants. The results show significant improvements over baselines like Sonic4D and OmniAudio in both spatial accuracy and semantic consistency. The qualitative analysis of Doppler effects and novel listener adaptation is convincing.
The paper provides detailed descriptions of the pipeline, including the specific models used for depth estimation, segmentation, and Gaussian generation. However, the reliance on several proprietary or very recent models (e.g., HYWorld 2.0, Marble, SAM3, Kimi VLM) may limit immediate reproducibility for the community. The code is not explicitly linked in the provided text (only a demo page), which is a minor drawback for open science, though the demo page likely hosts assets.
The method assumes outdoor scenes with limited reverberation, omitting reflected paths and complex room acoustics, which limits its applicability to indoor environments. The pipeline is complex and relies on a chain of foundation models, which may introduce error propagation. The dataset size (54 clips) is relatively small for robust statistical validation, though it is a new task. The computational cost of the dual-space diffusion synchronization and 4D Gaussian rendering is likely high, though real-time rendering is claimed for the final scene.
This work bridges the gap between 4D visual generation and spatial audio synthesis, enabling the creation of immersive, interactive worlds from minimal input. It has potential applications in virtual reality, game asset generation, and film pre-visualization. The introduction of a benchmark for dynamic spatial audio generation is particularly impactful for the research community. The paper introduces a novel task of Video-to-4DAVScene and a framework that effectively synchronizes visual dynamics with spatial audio using dual-space diffusion. The technical approach is rigorous, combining explicit 4D geometry with physically grounded acoustic rendering, and the introduction of a new dataset and evaluation protocol significantly advances the field of dynamic audio-visual scene generation.
A capable voice assistant must do more than recognize words; it must understand requests, reason over context, sustain conversations, and act through tools. We present Apriel Voice 27B Preview, an open speech-language model that unifies these capabilities within a shared reasoning backbone while retaining text and image input modalities. Built on the post-trained Gemma 4 26B-A4B checkpoint, Apriel Voice directly processes speech and generates both responses and tool calls without requiring a separate dialogue model. We introduce a two-stage training procedure to adapt the model to speech. The first stage establishes the speech interface, while the second jointly adapts the projector and backbone through supervised fine-tuning, text replay, and source-dependent cross-entropy and forward Kullback--Leibler divergence supervision. Automatically constructed speech-conditioned dialogues provide conversational training data, while local teacher serving, overlapping teacher and student computation, and fused full-vocabulary losses enable efficient online distillation. In thinking mode, Apriel Voice achieves a score of 98.90 on Big Bench Audio, 81.13 on Speech-BFCL, and a task success rate of 65.51% on $ฯ$-Voice, with a mean latency of 1.91 seconds. It also achieves 51.27% English EVA accuracy and 74.13% experience pass@1. Finally, we deploy Apriel Voice 27B Preview on a custom serving platform to support interactive voice interactions.
Primary: ServiceNow
All Institutions: ServiceNow
Apriel Voice 27B Preview is an open speech-language model that unifies speech understanding, reasoning, and tool use within a shared backbone. The paper presents a rigorous two-stage adaptation recipe combining projector training with joint fine-tuning using source-dependent CE/FKL supervision, demonstrating strong performance in agentic speech tasks while retaining text and image capabilities.
The paper proposes a two-stage training procedure to adapt a text-and-image LLM (Gemma 4 26B-A4B) into a speech-language model. Stage 1 freezes the backbone and trains a speech projector (Whisper encoder + MLP) on transcription, translation, and continuation tasks. Stage 2 jointly fine-tunes the projector and backbone using a mixture of speech tasks, synthetic speech-conditioned dialogues, tool interactions, and text replay. A key technical contribution is the source-dependent loss function combining Cross-Entropy (CE) for task targets and Forward Kullback-Leibler (FKL) divergence from a frozen teacher model to preserve original capabilities. The system also details an efficient training infrastructure using local teacher serving, overlapping computation, and fused full-vocabulary loss kernels. While the individual components (projector, distillation, SFT) are established, the specific combination for agentic speech tasks and the engineering optimizations for online distillation are well-executed.
The evaluation is comprehensive, covering foundational speech metrics (WER, BLEU), reasoning (Big Bench Audio, GSM8K), instruction following (IFEval), and agentic task completion (EVA, ฯ-Voice, EnterpriseOps-Gym). The model demonstrates strong performance, particularly in reasoning (98.90 on BBA) and tool use (81.13 on BFCL), outperforming other open-source 2-stage models. However, the reliance on synthetic data for conversation training and the lack of ablation studies isolating the impact of specific training components (e.g., FKL vs. CE, specific data mixes) limit the depth of the empirical analysis. The comparison with proprietary systems is competitive but relies on different evaluation harnesses and simulators, making direct comparison difficult.
The paper provides detailed architectural specifications (projector dimensions, parameter counts) and training hyperparameters (learning rates, batch sizes, GPU configuration). The release of an interactive demo and the use of open-source base models (Gemma, Whisper) enhance reproducibility. However, the synthetic data generation pipeline and specific internal datasets are not fully released, and the "thinking mode" implementation details are not fully specified, which may hinder exact reproduction of the reported reasoning scores.
The model is primarily optimized for English, with limited multilingual speech understanding. The evaluation lacks paired significance tests against the base model, making it hard to quantify the exact cost of adaptation on text/image capabilities. The synthetic data pipeline may propagate errors from the TTS or LLM generators. The paper acknowledges that frozen visual components do not guarantee unchanged image reasoning, yet the drop in some vision benchmarks suggests potential degradation.
This work contributes to the development of open-source speech-language models capable of agentic tasks, which is a critical area for enterprise and consumer applications. By demonstrating that a single backbone can handle speech, text, images, and tool calls, it supports the trend toward unified multimodal agents. The efficient training techniques for online distillation are valuable for the broader community adapting large models to new modalities. Apriel Voice 27B Preview is an open speech-language model that unifies speech understanding, reasoning, and tool use within a shared backbone. The paper presents a rigorous two-stage adaptation recipe combining projector training with joint fine-tuning using source-dependent CE/FKL supervision, demonstrating strong performance in agentic speech tasks while retaining text and image capabilities.
Deciding which member of an animal group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3D candidate geometry. We close this gap with V33DA, a dataset-centered multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an on-body reference ceiling. Given a detected vocalization and the visible candidate birds, the task is to select the caller. We evaluate it under three regimes: session-disjoint testing on familiar birds from held-out recording dates, held-out-experiment transfer to a new cohort, and leave-one-individual-out evaluation on a bird whose vocalizations are held out from training. Reference methods expose a consistent gap between in-domain accuracy and transfer to unseen individuals: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The on-body reference stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels. We release V33DA with fixed protocols and code for the evaluated methods, and V33DA++, an unevaluated companion set of verified overlapping-call events and +/-2 s context windows.
Primary: Institute of Neuroinformatics, University of Zurich and ETH Zurich
All Institutions: Institute of Neuroinformatics, University of Zurich, ETH Zurich, Swiss Data Science Center, CVLab, EPFL
[One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
The paper proposes a rigorous methodology for constructing a physically verified benchmark for vocal attribution. The core innovation is the use of an on-body accelerometer-derived vibration signal to ground-truth caller identity, which is then withheld from the benchmark models to prevent shortcut learning. This is a sophisticated design choice that distinguishes the work from standard active speaker detection datasets that rely on perceptual annotation or close-talk references. The pipeline is detailed and auditable, involving synchronized multimodal data (5-channel audio, 3-camera video, 3D pose, radio telemetry) and a multi-stage verification process including manual review and geometric filtering. The evaluation protocol is particularly strong, defining three distinct regimes (session-disjoint, held-out-experiment, leave-one-individual-out) to explicitly separate in-domain recognition from transferable spatial reasoning.
The experimental evaluation is comprehensive, testing a wide array of baselines ranging from classical physics-based methods (SRP) to deep learning architectures (SELDnet++, EINV2, CST-former). The results clearly demonstrate the paper's central thesis: models that rely on familiar voice identity perform well in-domain but fail on unseen individuals, while spatially-aware models transfer better. The inclusion of modality intervention experiments (zeroing audio, shuffling geometry) provides strong causal evidence for what drives performance. The "on-body reference" ceiling confirms the consistency of the labels. The analysis of "close birds" and localization diagnostics adds depth to the evaluation.
Reproducibility is high. The authors release the full dataset, code for all evaluated methods, and fixed evaluation protocols. The paper provides extensive details on the data acquisition pipeline, camera calibration, and label construction in the appendices. The use of standard open-source tools (YOLOX, OC-SORT, Whisper) for preprocessing further aids reproducibility. The clear separation of the verification channel from the benchmark inputs ensures that future researchers can fairly evaluate new methods without access to the ground-truth sensor.
The primary limitation is the scale and diversity of the data. The dataset is limited to 10 zebra finches across 3 experiments in a single installation. This limits the generalizability of the findings to other species or larger, more dynamic groups. The radio telemetry features are limited due to carrier drift, which the authors acknowledge. The benchmark currently only covers single-vocalizer events, leaving multi-caller scenarios for the unevaluated V33DA++ release.
This work has significant impact on the fields of bioacoustics, multimodal learning, and behavioral analysis. By providing a physically verified benchmark, it enables the development of more robust and generalizable models for animal communication. The insights into shortcut learning and the importance of spatial reasoning are applicable to other multimodal attribution tasks, such as human active speaker detection. The release of the dataset and code will likely spur further research in this niche but important area. [One sentence main contribution]. [Comprehensive analysis of the technical contribution, methodology, and significance to the field]. The paper introduces V33DA, a physically verified multimodal benchmark for vocal attribution in zebra finches, demonstrating that transferable attribution requires spatial reasoning rather than familiar-voice recognition, and providing a rigorous framework for evaluating multimodal models in social animal communication.
We present EmbeddingGemma 2, an open multimodal embedding model which natively represents text, images, audio, video, and combinations thereof in a unified space. Our approach introduces improvements such as simplified per-layer embeddings and a multi-stage curriculum with modality and objective progression which ensure strong quality across unimodal, cross-modal, and mixed-modal evaluations. Notably, in comprehensive embedding benchmarks across all modalities, EmbeddingGemma 2 establishes a new Pareto frontier, outperforming both specialist models and models two to twelve times larger. Furthermore, our design offers flexibility for resource-constrained environments, supporting selective modality encoder loading, adjustable vision token budgets, truncatable embedding dimensions, and quantized weights. We present ablations analyzing key decisions and release EmbeddingGemma 2 to promote further research.
Primary: Google
All Institutions: Google
EmbeddingGemma 2 introduces a unified, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks while significantly reducing computational costs through architectural simplifications and modular design. The paper demonstrates that a 744M-parameter model can outperform much larger specialists by leveraging a multi-stage curriculum learning strategy that transitions from distillation to contrastive learning, effectively transferring knowledge from a larger teacher model while optimizing for the student's specific architecture and deployment constraints.
The paper proposes EmbeddingGemma 2, a 744M-parameter multimodal embedding model. The core architectural contribution is the simplification of Per-Layer Embeddings (PLE) by removing the dedicated lookup table and instead projecting the main input embeddings, which reduces memory overhead and improves multimodal performance. The training methodology is rigorous, employing a three-stage curriculum: text pre-finetuning (primarily distillation), multimodal pre-finetuning (distillation from a larger teacher model, Gemini Embedding 2), and multimodal finetuning (contrastive learning with spread-out regularization and Matryoshka Representation Learning). This progression effectively transfers knowledge from a large teacher to a smaller student while allowing the student to optimize its own representation space. The inclusion of quantization-aware training (QAT) and modular encoder loading demonstrates a strong focus on practical deployment constraints.
The evaluation is comprehensive, covering seven major benchmark suites (MTEB, MMEB-v2, MIEB-lite, MSEB, MAEB) across text, code, image, video, document, speech, and sound. The model establishes a new Pareto frontier, outperforming specialist models and general-purpose models 2-12x larger. Ablation studies are particularly strong, validating the removal of the PLE lookup table, the necessity of co-training the vision encoder, and the impact of vision token budgets. The comparison against ASR-based pipelines for audio tasks convincingly demonstrates the value of native audio understanding.
The paper provides high reproducibility for the model usage, as the checkpoints are released. It details the architecture, loss functions, and training recipe in sufficient depth. However, as is common with large-scale industrial models, the specific data mixtures and hyperparameters for the 2.9T token training run are not fully disclosed, limiting the ability to replicate the training process from scratch.
The model is still significantly larger than pure text embedding models, posing challenges for extremely low-resource edge devices despite the modular design. The paper notes that video performance degrades at higher token budgets due to attention dilution, suggesting room for improvement in long-context handling. Additionally, while it outperforms many specialists, it does not always beat the absolute best specialists in every single niche audio task (e.g., some paralinguistic clustering).
This work significantly lowers the barrier to entry for high-quality multimodal retrieval. By providing a unified, efficient, and open model, it enables developers to build complex multimodal applications (RAG, search, recommendation) without assembling complex pipelines of ASR, captioners, and separate embedding models. The focus on on-device deployment (191MB footprint for text-only) is particularly impactful for privacy-preserving applications. EmbeddingGemma 2 introduces a unified, efficient multimodal embedding model that achieves state-of-the-art performance across text, vision, and audio benchmarks while significantly reducing computational costs through architectural simplifications and modular design. The paper demonstrates that a 744M-parameter model can outperform much larger specialists by leveraging a multi-stage curriculum learning strategy that transitions from distillation to contrastive learning, effectively transferring knowledge from a larger teacher model while optimizing for the student's specific architecture and deployment constraints.
Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
Primary: Adobe Research
All Institutions: University of Surrey, Adobe Research
TAC2 introduces a scalable framework for timestamped audio captioning by fusing weak teacher predictions and eliciting latent paralinguistic capabilities, resulting in a composable, temporally grounded representation that enhances both audio understanding and controllable generation.
The paper proposes a two-stage framework, TAC2, for timestamped audio captioning. Stage 1 addresses the data scarcity problem for fine-grained temporal grounding by programmatically fusing predictions from weak teachers (sound event detectors, diarizers, captioners) using a time-registered IoU matching algorithm. This creates a structured, speaker-attributed timeline without expensive manual annotation. The model is trained using an Instruction-Input-Output (IIO) completion format, which allows for composable functions where outputs of one task can condition subsequent tasks. Stage 2 extends this by eliciting latent paralinguistic capabilities (stress, emotion, etc.) from a frozen backbone using lightweight LoRA specialists trained on small datasets, then distilling these into a unified student model. The methodology is sound, leveraging the strengths of weak supervision and parameter-efficient fine-tuning to build a complex, structured representation.
The authors validate the model in two settings: audio understanding (cascading with frozen LLMs) and audio generation (paired supervision). In understanding, TAC2 achieves strong results on MMSU and MSU-Bench, outperforming baselines like TAC and Qwen3Omni-Captioner, particularly in perception tasks. The generation experiments demonstrate zero-shot control over event timing and speaker attributes. The evaluation is comprehensive, covering both the utility of the structured representation for reasoning and its application in generative tasks.
The paper provides detailed descriptions of the fusion algorithm, the IIO format, and the specialist training procedure. However, specific hyperparameters, dataset sizes for the weak supervision corpus, and the exact implementation details of the IoU matching thresholds are not fully specified in the provided text, which may hinder exact reproduction. The use of standard tools like LoRA and common benchmarks aids reproducibility.
The reliance on weak teachers means the quality of the fused timeline is bounded by the accuracy of the individual detectors and diarizers. Errors in the weak supervision stage can propagate to the final model. Additionally, the "latent capability elicitation" approach, while efficient, may not capture all nuances of paralinguistic features if they are not strongly encoded in the backbone. The paper is truncated, so the full extent of the generation results and ablations is not visible.
This work has significant potential for applications requiring precise audio understanding, such as automated video captioning, audio search, and accessible media. By decoupling perception from reasoning, it enables more robust and controllable audio pipelines. The composable nature of the IIO format makes it adaptable to various downstream tasks. TAC2 introduces a scalable framework for timestamped audio captioning by fusing weak teacher predictions and eliciting latent paralinguistic capabilities, resulting in a composable, temporally grounded representation that enhances both audio understanding and controllable generation.
Reinforcement learning has made Audio Large Language Models (Audio LLMs) better reasoners, but the reward that drives it still comes from outside the model: a final-answer check, a hand-crafted process score, or an external LLM judge that stays fixed while the policy improves. We show that an Audio LLM can supply this reward itself, and can do so recursively: each improved model becomes the critic that trains its successor, so the training signal improves along with the model. We introduce AudioRSI (Recursively Self-Improving Audio Reasoners), a reinforcement learning framework in which a single model plays two roles: as an actor it generates reasoning paths, and as a critic it compares them pairwise, turning its own judgments into a process-level reward. The critic is periodically replaced by the current policy, which closes the recursive loop. Because the critic hears the same recording as the actor, it can reject reasoning that cites acoustic evidence the audio does not contain, rather than reward reasoning that merely sounds plausible. The recursion is what sustains progress: under a frozen external judge, out-of-distribution accuracy plateaus during training, while a critic refreshed from the improving policy keeps it rising. Across four out-of-distribution benchmarks, all disjoint from the AVQA training distribution, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, ranks first on MMSU, achieves a clear lead on the challenging Multi-Audio split of MMAU-Pro, and leads the 7B open-source tier on MMAR among methods without external reward models. With no reward specifying how to reason, AudioRSI also develops structured reasoning behaviors (logical reasoning, perception-aware analysis, and reduced hallucination), corroborated by an external LLM-as-Judge evaluation and a keyword-based behavior analysis.
Primary: Amazon
All Institutions: University of Illinois Urbana-Champaign (UIUC), Amazon
AudioRSI introduces a recursively self-improving reinforcement learning framework for audio LLMs where the model serves as its own evolving critic. By periodically synchronizing the critic with the actor, the method ensures that the reward signal for reasoning quality improves alongside the policy, leading to superior out-of-distribution performance and emergent structured reasoning behaviors compared to static external judges or outcome-only rewards.
The paper proposes AudioRSI, a framework where an Audio LLM acts as both actor and critic within a GRPO (Group Relative Policy Optimization) loop. The core innovation is the "recursive" aspect: the critic is periodically synchronized with the actor's weights, allowing the reward signal to improve as the model improves. The methodology is sound, leveraging the unique advantage of audio modality where the critic can verify acoustic claims against the raw waveform, unlike text-only self-rewarding models. The use of pairwise comparison for process-level rewards is a solid adaptation of DPO-style thinking to RLVR (Reinforcement Learning with Verifiable Rewards).
The experiments are conducted on Qwen2.5-Omni-7B using AVQA for training and four OOD benchmarks (MMAU, MMSU, MMAU-Pro, MMAR) for evaluation. The results show consistent improvements over baseline RL methods (Ke-Omni-R) and strong performance against proprietary models like Gemini 2.5 Pro on specific splits. The ablation on synchronization period $T$ is crucial and well-executed, demonstrating the trade-off between stability and reward freshness. The claim that the model develops structured reasoning behaviors is supported by LLM-as-Judge and keyword analysis, though these are somewhat proxy metrics.
The paper provides detailed algorithmic descriptions and hyperparameters in the appendix. The reliance on specific base models (Qwen2.5-Omni) and datasets (AVQA) makes reproduction feasible for groups with similar compute resources. The code is not explicitly linked in the provided text, which is a minor gap for immediate reproducibility.
The approach is computationally expensive due to the dual role of the model (generating and critiquing). The "guarded handoff" mechanism, while practical, adds complexity. The evaluation relies heavily on LLM-as-Judge for reasoning quality, which may introduce bias if the judge model shares biases with the actor. The gains are modest in some domains (e.g., Music), suggesting the method may be more effective for domains with clearer acoustic evidence.
This work pushes the boundary of self-improving AI systems in multimodal domains. It demonstrates that models can provide meaningful process-level supervision for their own reasoning, reducing reliance on external, static reward models. This has implications for scaling audio reasoning capabilities without massive human annotation of reasoning steps. AudioRSI introduces a recursively self-improving reinforcement learning framework for audio LLMs where the model serves as its own evolving critic. By periodically synchronizing the critic with the actor, the method ensures that the reward signal for reasoning quality improves alongside the policy, leading to superior out-of-distribution performance and emergent structured reasoning behaviors compared to static external judges or outcome-only rewards.
Pretrained image-generation models have been successfully repurposed for various downstream visual tasks. Whether their learned representations and generative priors can transfer beyond vision to other modalities remains largely unresolved, particularly for speech synthesis, which requires linguistic fidelity and preservation of speaker identity from a short reference. We present ImageTTS, which repurposes FLUX.2 for speech synthesis by encoding pseudo-RGB log-mel spectrograms with its frozen image VAE and adapting only its pretrained Diffusion Transformer (DiT) using conditional latent flow matching. This formulation disentangles two forms of vision-to-speech transfer: representation transfer through the image VAE and generative-prior transfer through DiT initialization. The frozen image VAE largely preserves intelligibility and speaker identity with 0.99 STOI and 0.95 speaker similarity, while image-pretrained DiT initialization accelerates convergence and improves linguistic accuracy, reducing WER from 4.2\% to 2.7\% at 100k matched updates relative to random initialization. The same framework supports both zero-shot and instruction-guided TTS and, when scaled to 100K hours of speech data, achieves 1.12\% WER on Seed-TTS test-en and a Style-ACC of 0.66 on CapSpeech, outperforming speech-native baselines trained on comparable data across both tasks. These results suggest that pretrained image models can transfer beyond architectural reuse, supplying useful latent representations and generative priors for intelligible and controllable speech synthesis.
Primary: Microsoft Corporation
All Institutions: Wuhan University, Microsoft Corporation, The Chinese University of Hong Kong
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
The methodology proposes ImageTTS, a novel framework that repurposes a pretrained image diffusion transformer (FLUX.2) for speech synthesis. A key strength is the systematic disentanglement of two forms of vision-to-speech transfer: representation transfer via a frozen image VAE and generative-prior transfer via DiT initialization. The approach involves converting log-mel spectrograms into pseudo-RGB images, encoding them with the frozen FLUX.2 image VAE, and then fine-tuning only the pretrained DiT backbone using conditional latent flow matching. This pseudo-RGB conversion and subsequent VAE encoding is a creative way to bridge the modalities, though it relies on the image VAE's ability to meaningfully compress and reconstruct spectrograms. The formulation of zero-shot TTS as reference-conditioned latent infilling and instruction-guided TTS as full latent generation conditioned on text and description is well-defined. The use of classifier-free guidance for both text and speech conditions (and style for instruction-guided) is standard practice in generative models and appropriately applied here. The architecture leverages FLUX.2's double-stream and single-stream Transformer blocks for multimodal interaction, which is a sensible adaptation for handling both text and speech latents. The overall methodology is sound, innovative, and systematically designed to answer the research questions posed.
The experimental evaluation is comprehensive and addresses the three core research questions effectively. 1. **Speech reconstruction with frozen image VAE:** This experiment rigorously evaluates the representation transfer. By varying the frequency resolution ($F$) and comparing with a spectrogram-oriented VAE (Riffusion), the authors demonstrate that FLUX.2's VAE can largely preserve speech intelligibility (STOI 0.99) and speaker identity (SIM 0.95) at $F=512$, approaching vocoder-only baselines. This provides strong evidence for the utility of image VAEs for speech representation. 2. **Impact of image-pretrained DiT initialization:** This experiment directly assesses generative-prior transfer. Comparing FLUX.2 initialization against random initialization under matched training conditions shows significant improvements in convergence speed and linguistic accuracy (WER reduced from 4.2% to 2.7%). This is a clear demonstration that image-learned priors are beneficial for speech generation. 3. **Scaling to zero-shot and instruction-guided TTS:** This section evaluates the full system's competitiveness at scale (100K hours of speech data). ImageTTS-base achieves competitive zero-shot TTS performance (1.1
Spatial audio authoring coordinates source decisions with soundfield synthesis. For editing, plans specify what to change and retain, while reference audio supplies further acoustic detail. We present AMBIT, a unified framework for native First-Order Ambisonics (FOA) generation and editing with three connected contributions. First, a shared Transfusion backbone couples autoregressive planning with rectified-flow rendering through executable ScenePlans that specify source content, activity, and 3D trajectories. For editing, the plan describes the full target scene, including retained sources; contrastive features ground the requested change, while full reference latents supply residual acoustic detail for preservation. Second, an FOA-aware continuous VAE combines shared-decoder omnidirectional recovery, asymmetric grouped KL, and spatial covariance matching to preserve content and inter-channel structure. Third, joint Transfusion on-policy self-distillation (OPSD) refines planning on student-generated prefixes and rendering at sampling states reached under student plans, with target-conditioned teachers supervising both modes of the shared student. Evaluations across speech, music, and sound show improved generation content and spatial quality, acoustic fidelity during instruction-guided editing, and unedited-source preservation over the compared baselines. Project page: https://wjm-george.github.io/AMBIT/
Primary: HKUST
All Institutions: HKUST, Noiz AI, University of Washington
The paper explicitly states a primary limitation:
The paper introduces AMBIT, a unified framework for First-Order Ambisonics (FOA) generation and editing. The core methodology is built upon three interconnected contributions. First, a shared Transfusion backbone effectively couples autoregressive planning with rectified-flow rendering. This is a technically sound approach to integrate discrete planning (ScenePlans) with continuous generation, allowing for explicit control over source content, activity, and 3D trajectories. The ScenePlans are executable and deterministically compiled into semantic and frame-aligned spatial controls, which is a strong design choice for grounding language instructions in 3D audio. For editing, the use of contrastive features for grounding changes and full reference latents for preserving acoustic detail is well-conceived, addressing a key challenge in instruction-guided editing. Second, the proposed FOA-aware continuous Variational Autoencoder (VAE) is a significant improvement over standard audio VAEs for spatial audio. The combination of shared-decoder omnidirectional (W-channel) recovery, asymmetric grouped KL regularization, and especially Spatial Covariance Matching (SCM) directly addresses the unique requirements of FOA data. SCM, by supervising relative energy, coherence, and phase between channels, is a novel and effective way to preserve inter-channel structure crucial for accurate spatial perception. The detailed formulation of SCM, including stabilization and weighting, demonstrates a deep understanding of FOA signal processing. Third, the joint Transfusion On-Policy Self-Distillation (OPSD) mechanism is a sophisticated method for refining both the discrete planner and the continuous renderer. Adapting OPSD to jointly refine both modes, where student plans drive DiT trajectories and target-conditioned teachers supervise on student-generated prefixes and visited sampling states, is a non-trivial technical contribution. This approach addresses the common challenge of distribution shift between training with ground-truth plans and inference with student-generated plans, ensuring that the student learns to perform well in its own operational context. The detailed loss formulations for planning and rendering distillation, including the use of detached weights and target plan memory/audio reference tokens, are well-justified. The overall architecture is complex but appears coherently integrated, leveraging existing strong components (DiT, CLAP) while introducing novel adaptations for spatial audio.
The experimental evaluation is comprehensive and rigorous, covering generation, editing, and VAE performance across speech, music, and sound domains. The use of a large dataset (1.6M synthetic and recorded FOA examples, 1M editing pairs, 1M VAE clips) is commendable, although the reliance on synthetic data for a significant portion is a common limitation in spatial audio research. The evaluation metrics are extensive and appropriate for spatial audio, including content-based metrics (paired CLAP, FAD, WER/CER, UTMOS), spatial metrics (directional and binaural errors, GCC, CRW, wCS), and preservation metrics for editing (unchanged-source direction/waveform/level errors). The inclusion of listener ratings in the appendix further strengthens the subjective evaluation. The VAE evaluation clearly demonstrates the benefits of the FOA-aware design, with SCM and grouped KL significantly improving reconstruction quality and spatial accuracy over baselines like DAC, EnCodec, and WavTokenizer. The generation and editing quality results show AMBIT outperforming strong baselines like AudioX-Turbo and SwanWeave across multiple metrics, indicating superior content fidelity, acoustic fidelity, and spatial accuracy. The fixed-renderer comparisons effectively isolate the planner's contribution, while the unedited-source preservation metrics highlight a critical aspect of editing. The post-training OPSD experiments are particularly strong, showing that joint OPSD consistently improves all nine reported metrics over continued supervised training, ReFL, GRPO, and individual branch supervision variants. This provides compelling evidence for the effectiveness of the proposed joint refinement strategy. The detailed ablation studies for the VAE and OPSD components add significant credibility to the individual contributions. The use of Qwen3.5-0.8B as a frozen text backbone and CLAP features for grounding is a practical choice that leverages powerful pre-trained models. The experimental setup, including the use of 8 NVIDIA L40S GPUs, indicates a substantial computational effort.
The paper provides a good level of detail for reproducibility. The architecture details (15 shared Transfusion blocks, width, attention heads, VAE base width, channel multipliers, strides, latent channels, hop size) are specified. Training details, including learning rates, optimizers (AdamW), batch sizes, loss weights for OPSD, and specific coefficients for various loss terms (STFT, SI-SDR, SCM, phase supervision, etc.), are provided in the main text and appendices. The flow time sampling distributions for training and inference settings (Euler steps, guidance scales) are also given. The datasets are described in detail, including sources (LibriTTS, AudioCaps, AudioSet, FSD50K, MusicCaps, Sphere360, STARSS23, Spatial LibriSpeech) and how synthetic scenes are constructed (Pyroomacoustics, room distributions, source placement, trajectories). The process for generating pseudo-annotations using Qwen3-Omni is also outlined. The project page URL is provided, which typically hosts code, models, and/or audio samples, further aiding reproducibility. While the full code is not directly embedded, the level of detail suggests a strong commitment to enabling replication.
The paper explicitly states a primary limitation:
Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive objectives encourage agreement between pooled audio representations, but do not explicitly supervise local time-frequency representations. We introduce AudioFM, a self-supervised framework that combines recording-level contrast with patch-wise masked latent discrimination. Using visible acoustic context, a student predicts representations of masked patches and identifies their corresponding full-view teacher targets among candidates from other recordings. We further incorporate weaker supervision from eligible temporal neighbors to encourage local temporal consistency. Extensive evaluations show that AudioFM achieves state-of-the-art performance across diverse audio domains in linear probing, fine-tuning, and zero-shot transfer, demonstrating superior event detection, robust scene-level understanding, and strong domain generalizability.
Primary: Google Research
All Institutions: Google Research, Google DeepMind, Vanderbilt University, University of Washington
The paper introduces AudioFM, a self-supervised audio foundation model that effectively bridges global and local representation learning through a novel masked latent discrimination objective. By combining recording-level contrast with patch-wise discrimination against cross-recording negatives and incorporating temporal neighborhood consistency, AudioFM achieves state-of-the-art performance across diverse domains, including general audio, bioacoustics, and medical acoustics, demonstrating superior transferability and robustness in both fine-tuning and zero-shot settings.
The paper proposes AudioFM, a self-supervised framework that integrates recording-level contrastive learning with a novel local objective called Masked Latent Discrimination (MLD). The core innovation lies in the local branch, where a student model predicts representations for masked time-frequency patches and discriminates the correct full-view teacher target from negative candidates drawn from *other* recordings at the same spatial coordinate. This design avoids the common pitfall of forcing the model to distinguish acoustically similar regions within the same recording solely based on position. Additionally, the method incorporates a temporal neighborhood extension, providing weaker supervision from adjacent masked patches to enforce local temporal consistency. The use of a 2D Transformer predictor with axis-separated positional encodings is a solid architectural choice for handling the grid structure of spectrograms. The joint optimization of global and local losses is well-motivated, addressing the gap between holistic scene understanding and fine-grained event detection.
The experimental evaluation is extensive and rigorous. The authors compare AudioFM against strong baselines (AudioMAE, BEATs, EAT, SSLAM, BAT, AudioMosaic) across multiple settings: supervised fine-tuning, linear probing (frozen encoder), and zero-shot transfer. A standout feature is the systematic cross-domain evaluation spanning general audio, bioacoustics, medical acoustics, and speech emotion recognition. The inclusion of medical acoustics (PSG-Audio, CirCor) and bioacoustics (BEANS, BirdSet) is particularly valuable, as these domains are often underrepresented in standard audio SSL benchmarks. The results show consistent state-of-the-art performance, especially in linear probing where the quality of the frozen representation is isolated. The scaling analysis with model size and data volume further strengthens the contribution by demonstrating empirical scaling laws for audio pretraining.
The paper provides detailed hyperparameters, architecture specifications, and training protocols in the appendix. The use of standard datasets (AudioSet, ESC-50, FSD50K, etc.) and clear evaluation metrics (mAP, Macro-F1, PSDS) enhances reproducibility. However, the specific implementation details of the "mask-aware target selection" and the exact handling of the EMA teacher updates could benefit from code release to ensure full reproducibility. The paper mentions that code is not yet available or provides no link, which is a minor drawback for immediate replication.
The pretraining corpus is limited to AudioSet (approx. 1.68M clips), which is smaller than some recent large-scale efforts. The model is trained on single-channel audio, ignoring spatial cues that could be beneficial for certain tasks. The evaluation, while broad, does not include generative tasks or audio-text alignment, limiting the scope of "foundation model" claims. The medical acoustics evaluation, while impressive, uses relatively small datasets, and the generalizability to larger clinical datasets remains to be seen.
This work has significant potential for applications in environmental monitoring, healthcare (respiratory and cardiac sound analysis), and assistive technologies. By demonstrating strong transferability to specialized domains like bioacoustics and medical acoustics, the paper highlights the utility of general-purpose audio representations in niche but critical fields. The systematic evaluation framework provided could serve as a benchmark for future audio SSL research. The paper introduces AudioFM, a self-supervised audio foundation model that effectively bridges global and local representation learning through a novel masked latent discrimination objective. By combining recording-level contrast with patch-wise discrimination against cross-recording negatives and incorporating temporal neighborhood consistency, AudioFM achieves state-of-the-art performance across diverse domains, including general audio, bioacoustics, and medical acoustics, demonstrating superior transferability and robustness in both fine-tuning and zero-shot settings.
Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.
Primary: Mila-Quebec AI Institute
All Institutions: Mila-Quebec AI Institute, Universite Laval, Concordia University, Adobe Research
Audio-R1 introduces a scalable RL post-training framework using self-supervised audio tasks to improve reasoning in LALMs. The paper demonstrates consistent gains across multiple models and benchmarks, with strong ablation studies showing the complementary benefits of multi-task training and the superiority of RL over SFT for downstream generalization and reasoning quality.
The paper proposes Audio-R1, a reinforcement learning post-training framework that leverages self-supervised audio tasks to generate verifiable rewards. The core innovation is the design of three specific tasksโIdentity Detection, Region Inpainting, and Temporal Orderingโthat require the model to rely on acoustic evidence rather than linguistic priors. By using Group Relative Policy Optimization (GRPO) with rule-based verifiers, the method avoids the need for human-annotated reasoning traces or learned reward models. The approach effectively bridges the gap between self-supervised representation learning and reasoning-oriented post-training, offering a scalable solution to the data scarcity problem in audio reasoning.
The experiments are comprehensive, evaluating four different LALM backbones across three distinct benchmarks (MMAR, MMAU-mini, MMAU-Pro). The ablation studies are particularly strong, demonstrating that multi-task training yields complementary benefits over single-task training. The comparison between SFT and RL is a key strength, showing that while SFT improves task accuracy, it degrades reasoning trace quality, whereas Audio-R1 improves both. The use of an LLM-as-judge to evaluate reasoning quality provides valuable qualitative insights, although it remains a proxy metric.
The paper provides detailed descriptions of the task construction, reward functions, and training hyperparameters. The inclusion of prompt templates and judge rubrics in the appendix enhances reproducibility. However, the specific implementation details of the audio processing pipeline and the exact composition of the 24K source recordings could be more transparent. The use of standard libraries and frameworks (implied by the use of GRPO and LoRA) suggests reasonable reproducibility for researchers with similar resources.
The primary limitation is the reliance on self-supervised tasks that may not fully capture the complexity of open-ended audio understanding. The paper acknowledges that the tasks are designed for specific capabilities (correspondence, completion, ordering) and may not generalize to all forms of audio reasoning. Additionally, the use of an LLM-as-judge for reasoning quality evaluation is a proxy and does not establish causal links between reasoning traces and accuracy gains. The method is also conditioned on the pretrained backbone's capabilities, meaning it cannot develop reasoning skills absent from pretraining.
This work has significant implications for the development of large audio-language models, providing a scalable and efficient method for improving audio reasoning without the need for extensive human annotation. The approach could be extended to other modalities and tasks, potentially leading to more general-purpose multimodal models. The emphasis on verifiable rewards and self-supervised tasks aligns with broader trends in AI research towards more autonomous and scalable learning methods. Audio-R1 introduces a scalable RL post-training framework using self-supervised audio tasks to improve reasoning in LALMs. The paper demonstrates consistent gains across multiple models and benchmarks, with strong ablation studies showing the complementary benefits of multi-task training and the superiority of RL over SFT for downstream generalization and reasoning quality.
Localized AI manipulations in long-form videos often affect brief events within otherwise authentic content, making video-level classification insufficient for forensic analysis. Existing audio-visual deepfake benchmarks largely focus on short talking-head clips, while long-form temporal forensics remains predominantly visual. We introduce AV-FABLE, a dual-timeline benchmark for sparse manipulations in open-scene long-form audio-visual videos. AV-FABLE contains 16,650 pristine--manipulated twin pairs (998 hours) created with 27 editing configurations, with separate audio and visual timelines that enable modality-specific temporal localization and expose errors hidden by joint-timeline evaluation. On AV-FABLE, zero-shot prompting of audio-visual large language models (AV-LLMs) yields near-chance detection and poor temporal localization. In contrast, supervised readouts of their frozen perception features recover strong forensic signals, suggesting manipulation evidence is encoded in these representations but not effectively exposed through language outputs. Motivated by this observation, we propose NFSR(Native Forensics via Separated Readouts), which reads frozen AV-LLM features with separate audio and visual temporal decoders. NFSR achieves the best audio-visual localization performance in-domain and across all three out-of-distribution settings, including transfer to held-out videos from an external long-video forensic benchmark. AV-FABLE, code, and model weights will be publicly released.
Primary: Tsinghua University
All Institutions: Tsinghua University, Peking University
The paper presents a high-quality, large-scale benchmark (AV-FABLE) and a novel method (NFSR) for audio-visual manipulation forensics in long-form videos, demonstrating that frozen AV-LLM features are more effective for forensic localization than zero-shot language outputs.
The paper introduces AV-FABLE, a substantial benchmark for sparse audio-visual manipulations in long-form videos, featuring a dual-timeline evaluation protocol that distinguishes audio and visual tracks. This is a significant methodological advancement over joint-timeline evaluations which can mask modality-specific errors. The proposed model, NFSR, utilizes frozen features from AV-LLMs (Qwen3-Omni) with separate temporal decoders for each modality. The key methodological insight is the observation that while zero-shot prompting of AV-LLMs fails at forensic detection, their frozen perceptual features contain strong forensic signals accessible via supervised readouts. The use of host-conditioned feature modulation (FiLM) is a clever architectural choice to contextualize local anomalies within the broader video content.
The experimental setup is rigorous, including in-domain and three out-of-distribution (OOD) settings (source transfer, held-out generators, held-out routes). The dataset scale (16,650 pairs, 998 hours) is impressive for a forensic benchmark. Results show NFSR outperforming strong baselines like UMMAFormer and ActionFormer in AV-mAP across all settings. The ablation studies effectively demonstrate the contribution of intermediate-layer features, host conditioning, and modality separation. The comparison with zero-shot LLMs highlights a critical gap in current multimodal LLM capabilities for fine-grained forensic tasks.
The authors commit to releasing the dataset, code, and model weights. The paper provides detailed descriptions of the data construction pipeline (SEE-Forge), including the use of Gemini 3.7 Flash for planning and human review protocols for quality control. The evaluation metrics (AV-mAP, Event-IoU) and protocols are clearly defined. The reliance on specific, potentially proprietary, LLMs for data generation (Gemini) might pose minor reproducibility challenges for the dataset creation itself, but the benchmark and model are designed to be open.
The benchmark relies on synthetic manipulations generated by specific AI models, which may not fully capture the diversity of real-world human-edited forgeries or future, unseen generation techniques. The "explanation" task is noted as challenging and not fully solved by current baselines. The paper acknowledges that visual coverage under distribution shift remains an open problem, with NFSR showing lower V-IoU compared to some visual-only baselines in certain OOD settings.
This work addresses a critical gap in AI safety and media integrity by providing tools to localize and attribute manipulations in long-form content, which is increasingly relevant for misinformation detection. The finding that AV-LLMs encode forensic evidence in their features but fail to expose it via language outputs is a valuable insight for the broader multimodal AI community, suggesting that current alignment or decoding strategies in LLMs may be suboptimal for fine-grained perception tasks. The paper presents a high-quality, large-scale benchmark (AV-FABLE) and a novel method (NFSR) for audio-visual manipulation forensics in long-form videos, demonstrating that frozen AV-LLM features are more effective for forensic localization than zero-shot language outputs.