VoxAudio: Vocalized Audio Synthesis via Multi-Reward Autoregressive Flow Matching
Abstract
Vocalized audio synthesis, the task of generating audio in which intelligible speech is embedded within an environmental soundscape, underpins applications such as podcast production and video dubbing. Existing Text-to-Audio (T2A) systems either reduce quoted speech to unintelligible vocal murmur or delegate it to a separate TTS model with post-hoc mixing, which forfeits control over when speech occurs and how it interacts with the scene. We present VoxAudio, a causal autoregressive flow matching model that addresses this problem from three complementary aspects. At the architecture level, chunk-wise causal factorization with independent per-chunk noise levels lets audio be emitted through sliding-window streaming inference with KV caching at variable target durations; to enable inference at arbitrary chunk granularities, we further pretrain the model with randomized chunk boundaries. At the preference level, multi-reward Negative-aware FineTuning (NFT) jointly optimizes semantic fidelity, linguistic accuracy, aesthetic quality, and temporal grounding At the data level, to supply the missing supervision for vocal content, we build VoxCorpus, a large-scale corpus whose captions quote the verbatim transcript of embedded speech with time intervals, and VoxBench, an interval-annotated benchmark with a temporal-grounding metric. Experiments on four benchmarks spanning general audio, speech, and unified vocalized audio validate the effectiveness and efficiency of VoxAudio. Our code and demos are available at https://voxaudio.github.io.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a technically sound approach to streaming audio generation with speech, introducing valuable data and benchmarks. The adaptation of autoregressive flow matching with chunk-wise causal factorization is a meaningful contribution to efficient audio synthesis, and the multi-reward alignment strategy offers a robust path for preference optimization in continuous domains.
Methodology
The paper proposes VoxAudio, a causal autoregressive flow matching model for vocalized audio synthesis. The core technical contribution lies in adapting flow matching to a streaming, autoregressive setting. Specifically, it employs chunk-wise causal factorization with independent per-chunk noise levels, allowing for sliding-window inference with KV caching. This addresses the train-test gap often seen in causal diffusion/flow models by randomizing chunk boundaries during pretraining. The methodology also includes a multi-reward Negative-aware FineTuning (NFT) stage to align the model with human preferences across semantic, linguistic, aesthetic, and temporal grounding dimensions. The architectural adaptations (causal convolutions, chunk-wise causal attention) are sound and necessary for the stated streaming objective. The integration of NFT for audio generation is a notable extension of recent RLHF techniques to continuous flow-matching domains.
Results & evaluation
The authors introduce VoxCorpus, a new dataset with verbatim speech annotations and time intervals, and VoxBench, a benchmark with a temporal-grounding metric. Experiments are conducted on four benchmarks. The results claim superiority in speech fidelity and timing precision over baselines. However, the provided text is truncated, preventing a full assessment of the quantitative results (e.g., FAD, UTMOS, WER scores) and ablation studies. The claim of "outperforming current vocalized audio generation baselines" is significant but requires rigorous verification against strong baselines like separate TTS+T2A pipelines or unified models like Dasheng-AudioGen. The introduction of a new benchmark is a strong positive for the community, provided the annotation quality and metric validity are high.
Reproducibility
The paper provides a code and demo link. The methodology describes the use of pre-trained components (Universe Audio VAE, T5, Whisper, CLAP), which aids reproducibility. The specific details of the NFT reward weights and the exact implementation of the randomized chunk boundary training are crucial for reproduction and are likely detailed in the full text (which is partially truncated here). The use of standard open-source models for evaluation (Whisper, CLAP) ensures that the evaluation protocol is replicable.
Limitations
The paper does not explicitly discuss the computational cost of the autoregressive flow matching compared to non-causal diffusion models, which is a significant factor for practical deployment. The reliance on multiple external models (Whisper for WER, CLAP for semantic reward) for NFT introduces potential biases and error propagation. The quality of the VoxCorpus dataset, particularly the accuracy of the "verbatim transcript" and time intervals, is critical and any errors there would limit the model's potential. The truncation of the text prevents assessment of failure cases or specific limitations in handling complex acoustic scenes.
Broader impact
VoxAudio addresses a significant gap in audio generation by enabling intelligible speech within environmental soundscapes, which has applications in podcasting, video dubbing, and immersive media. The release of VoxCorpus and VoxBench provides valuable resources for the community. However, the ability to generate realistic speech in arbitrary contexts raises concerns about misuse for deepfakes or misinformation, although the environmental context may mitigate some immediate risks compared to pure TTS. The paper presents a technically sound approach to streaming audio generation with speech, introducing valuable data and benchmarks. The adaptation of autoregressive flow matching with chunk-wise causal factorization is a meaningful contribution to efficient audio synthesis, and the multi-reward alignment strategy offers a robust path for preference optimization in continuous domains.
Author of this paper? Send a correction.