SoundscapeAgent: Agentic Soundscape Construction for Controllable Synthesis and Scalable Audio-Language Supervision
Abstract
We present an agentic soundscape construction framework for controllable compositional audio generation that makes explicit the scene planning, source selection, temporal layout, and rendering steps typically handled implicitly by single-shot text-to-audio models. An LLM-based agent converts user intent into an executable scene plan, acquires assets through retrieval and on-demand generation, renders controllable multi-event mixtures, and exports aligned scene metadata. The framework also supports human-in-the-loop interaction through user-guided tool selection and editable scene plans. Together, these components provide an inspectable and reusable approach to controllable soundscape synthesis and scalable audio-language data construction. Listener studies and objective metrics demonstrate competitive generation performance against text-to-audio baselines, while models trained with agent-generated data consistently outperform real-only baselines in downstream audio reasoning. Code, demos, and listening-test results are available at https://haozhang6720.github.io/SoundscapeAgentDemoPage/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
SoundscapeAgent presents a structured, agentic approach to soundscape synthesis that enables controllable generation and provides valuable structured supervision for audio-language reasoning tasks. The paper demonstrates that explicit scene planning and hybrid asset acquisition can produce high-quality, multi-event soundscapes that outperform single-shot baselines in complex scenarios and effectively augment training data for downstream audio reasoning models.
Methodology
The paper proposes SoundscapeAgent, an agentic framework that decomposes soundscape synthesis into explicit planning, asset acquisition, and rendering stages. The core innovation lies in using an LLM to generate a structured scene plan (events, roles, timing, gain) which is then executed by deterministic rendering primitives. This contrasts with end-to-end text-to-audio models by offering inspectability and controllability. The methodology combines retrieval from a curated asset library with on-demand generation for missing assets, followed by a timeline-based mixer. While the concept of "agentic" audio is emerging, the specific implementation here is a composite of existing techniques (LLM planning + T2A generation + audio mixing) rather than a novel algorithmic breakthrough. The novelty is moderate, primarily in the system-level integration and the specific focus on structured supervision for downstream reasoning.
Results & evaluation
The evaluation is split into two tracks. Track A compares generation quality against baselines (TangoFlux, EzAudio, AudioLDM2) using human listening tests and CLAP scores. The results show competitive performance, particularly in complex, multi-event, or affective scenes, validating the controllability aspect. Track B evaluates the utility of the generated data for training audio-language models on the MMAU benchmark. The results demonstrate that adding agent-generated data improves reasoning accuracy, especially in sound and music domains, over real-only baselines. This is a strong empirical result supporting the value of structured synthetic data. However, the downstream model uses a frozen encoder and only trains a small projector, which limits the assessment of the data's full potential compared to fine-tuning larger models. The evaluation is solid but relies on standard benchmarks and metrics.
Reproducibility
The paper provides a demo page with code links (implied by the URL, though the specific GitHub repo isn't explicitly named in the text, the URL points to a project page). The methodology is described in detail, including the asset filtering criteria (CLAP score, production complexity) and the rendering pipeline. The use of open-source models (Qwen, Whisper, CLAP) aids reproducibility. However, the "private in-house data" mentioned in the asset library is not accessible, which may limit the exact replication of the asset acquisition phase. The prompt engineering for the LLM agent is not fully detailed, which could affect reproducibility of the scene plans.
Limitations
The authors acknowledge limitations including dependence on asset inventory quality, the domain gap between synthetic and real audio, and the bias towards non-speech events in the asset library. The evaluation of speech capabilities is limited. The reliance on an LLM for planning introduces potential latency and error propagation if the plan is flawed. The "offline prior mode" improves throughput but may reduce the adaptability of the generation to specific user intents compared to online planning.
Broader impact
This work contributes to the field of audio-language modeling by providing a scalable method for generating structured, high-quality synthetic data for training. It addresses the need for fine-grained temporal and compositional supervision in audio understanding. The agentic approach could inspire similar frameworks in other multimodal domains. However, the generation of synthetic audio data raises concerns about data contamination and the potential for propagating biases present in the underlying models or asset libraries. SoundscapeAgent presents a structured, agentic approach to soundscape synthesis that enables controllable generation and provides valuable structured supervision for audio-language reasoning tasks. The paper demonstrates that explicit scene planning and hybrid asset acquisition can produce high-quality, multi-event soundscapes that outperform single-shot baselines in complex scenarios and effectively augment training data for downstream audio reasoning models.
Author of this paper? Send a correction.