UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
Abstract
Audio generation and audio-to-text understanding remain largely separate, with diffusion models dominating high-fidelity synthesis and autoregressive (AR) language models driving captioning and semantic prediction. Existing unified approaches typically rely on either heterogeneous modules or AR-centric modeling, which can hinder joint optimization and limit acoustic fidelity. We present UAT, to our knowledge, the first diffusion-centric framework that supports unified audio generation, editing, and captioning. UAT couples continuous latent diffusion for audio with masked discrete diffusion for text, enabling bidirectional audio-text modeling within a shared dual-stream backbone. Experiments show that UAT preserves strong audio generation and editing capabilities while achieving competitive captioning performance, demonstrating a favorable balance between acoustic synthesis and semantic prediction. Demo samples are available at https://UAT-demo.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
UAT presents a novel diffusion-centric framework for unified audio generation, editing, and captioning, achieving a favorable balance between acoustic fidelity and semantic understanding. The paper's contributions are substantial, addressing key challenges in the field of audio processing and setting a foundation for future research in multimodal audio-text integration.
Methodology
The paper introduces UAT, a unified audio-text diffusion framework that innovatively combines continuous latent diffusion for audio with masked discrete diffusion for text. This dual-stream architecture allows for bidirectional modeling, addressing the limitations of existing audio generation and understanding methods that typically operate in isolation. The methodology is well-structured, leveraging pretrained models and incorporating a joint optimization strategy that enhances both audio generation and captioning capabilities. The design choices, such as the depth of the text branch and the use of modality-specific encoders, are carefully justified and contribute to the overall effectiveness of the model.
Results & evaluation
The experiments are comprehensive, utilizing a large-scale dataset of 2.4M audio samples to evaluate the model's performance across audio generation, editing, and captioning tasks. The results demonstrate UAT's competitive performance against both specialized and unified models, with detailed metrics provided for each task. The use of human evaluations alongside objective metrics adds robustness to the findings, indicating that UAT not only performs well in automated assessments but is also favored by human listeners.
Reproducibility
The paper provides sufficient implementation details, including model architecture, training protocols, and evaluation metrics, which facilitate reproducibility. However, the lack of a publicly available code repository may hinder full reproducibility for some researchers.
Limitations
The paper acknowledges that UAT's performance is still constrained by the underlying pretrained audio generation model, and its understanding capabilities lag behind larger autoregressive models. Additionally, the model's potential for broader audio-language tasks remains unexplored, suggesting areas for future research.
Broader impact
UAT has significant potential applications in audio content creation, automatic audio description, and enhancing multimodal accessibility. However, ethical considerations regarding the misuse of generated audio and the biases in training data must be addressed to ensure responsible deployment. UAT presents a novel diffusion-centric framework for unified audio generation, editing, and captioning, achieving a favorable balance between acoustic fidelity and semantic understanding. The paper's contributions are substantial, addressing key challenges in the field of audio processing and setting a foundation for future research in multimodal audio-text integration.
Author of this paper? Send a correction.