Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis
Abstract
Recent advances in diffusion-based generative models have enabled high-quality text-to-audio synthesis, but fine-grained acoustic control remains a significant challenge in open-source research. We present Audio Palette, a diffusion transformer (DiT) based model that extends the Stable Audio Open architecture to address this "control gap" in controllable audio generation. Unlike prior approaches that rely solely on semantic conditioning, Audio Palette introduces four time-varying control signals: loudness, pitch, spectral centroid, and timbre, for precise and interpretable manipulation of acoustic features. The model is efficiently adapted for the nuanced domain of Foley synthesis using Low-Rank Adaptation (LoRA) on a curated subset of AudioSet, requiring only 0.85 percent of the original parameters to be trained. Experiments demonstrate that Audio Palette achieves fine-grained, interpretable control of sound attributes. Crucially, it accomplishes this novel controllability while maintaining high audio quality and strong semantic alignment to text prompts, with performance on standard metrics such as Frechet Audio Distance (FAD) and LAION-CLAP scores remaining comparable to the original baseline model. We provide a scalable, modular pipeline for audio research, emphasizing sequence-based conditioning, memory efficiency, and a three-scale classifier-free guidance mechanism for nuanced inference-time control. This work establishes a robust foundation for controllable sound design and performative audio synthesis in open-source settings, enabling a more artist-centric workflow.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Audio Palette introduces a diffusion transformer-based model for controllable audio generation, significantly advancing the state of the art in Foley synthesis. The combination of multi-signal conditioning and efficient fine-tuning strategies positions this work as a valuable resource for both academic research and practical applications in sound design.
Methodology
The methodology presented in Audio Palette is robust, leveraging a diffusion transformer architecture with a novel multi-signal conditioning framework that integrates four time-varying acoustic control signals. The use of Low-Rank Adaptation (LoRA) for fine-tuning enhances efficiency, allowing the model to adapt to the specific domain of Foley synthesis with minimal computational overhead. The integration of a three-scale classifier-free guidance mechanism for controlling different aspects of audio generation is particularly innovative, providing users with a nuanced interface for sound design.
Results & evaluation
The experiments conducted are thorough, utilizing a well-curated dataset from AudioSet specifically focused on Foley sounds. The evaluation metrics, including Fréchet Audio Distance (FAD) and LAION-CLAP scores, are appropriate for assessing audio quality and semantic alignment. The quantitative results indicate that while there is a slight trade-off in audio quality and semantic adherence, the model achieves significant gains in controllability, which is the primary goal of the research.
Reproducibility
The paper provides sufficient implementation details, including the architecture, dataset, and training procedures, which would allow other researchers to reproduce the results. However, the absence of a publicly available demo or project URL limits immediate accessibility for practical application and experimentation by the community.
Limitations
The model's reliance on reference audio for control signal extraction is a notable limitation, as it restricts the model's ability to generate audio purely from text descriptions. Additionally, the potential for artifacts with extreme guidance values suggests that user tuning is necessary, which could complicate the user experience.
Broader impact
The work has significant implications for the fields of sound design and audio synthesis, particularly in professional settings such as film and game production. By bridging the gap between traditional Foley artistry and modern machine learning techniques, Audio Palette enables a more artist-centric workflow that could enhance creative processes in audio production. Audio Palette introduces a diffusion transformer-based model for controllable audio generation, significantly advancing the state of the art in Foley synthesis. The combination of multi-signal conditioning and efficient fine-tuning strategies positions this work as a valuable resource for both academic research and practical applications in sound design.
Author of this paper? Send a correction.