PEACE: Joint Embeddings of DSP Effects Code and Audio
Abstract
This paper introduces PEACE, the first joint embedding of audio effect code and output audio. Building on SLAP's multimodal objective, we pair an AFx-Rep audio encoder with two code encoders for Faust, a functional language for audio signal processing. First, we evaluate a fine-tuned T5 transformer over Faust source code. Second, we evaluate a message-passing graph neural network over an intermediate representation of the Faust compiler, capturing both topology and UI parameters. We evaluate on audio-to-code retrieval, where masking UI parameters at inference yields embeddings that encode effect chain topology alone. When parameters are visible, the two code encoders tie on retrieval of mixed-length chains but have tradeoffs on single-effect galleries. With parameters fully masked, PEACE recovers ordered chain topology far above chance without the limitations of supervised methods. PEACE outperforms pretrained models on an out-of-distribution reverb retrieval benchmark and can improve frozen audio-only representations. Its dual understanding of topology and parameters lays the groundwork for music information retrieval systems that search, generate, and condition on DSP code.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the first joint embedding of audio effect code and output audio, demonstrating that graph-based encoders of DSP code structure can effectively align with audio representations. The rigorous evaluation, including out-of-distribution benchmarks and detailed ablations of code encoders, establishes a strong foundation for future research in multimodal audio-DSP alignment.
Methodology
The paper proposes PEACE, a joint embedding model for audio effects code (Faust) and audio. The core methodological contribution is the design of two distinct code encoders: a fine-tuned T5 transformer operating on source code and a custom Graph Neural Network (BoxGraph) operating on the Faust compiler's Block Diagram Algebra (BDA) intermediate representation. The use of the BDA graph structure to capture topology and parameters is a strong technical choice, allowing for structural reasoning that token-based models might miss. The training objective utilizes SLAP (Siamese Language-Audio Pretraining), a non-contrastive BYOL-style objective, which is a solid choice for avoiding negative sample issues and reducing modality gap. The masking strategy for parameters during inference to isolate topology is a clever application of the learned representations.
Results & evaluation
The experimental setup is rigorous. The authors construct a large-scale dataset of 200K pairs, which is substantial for this niche. They evaluate on multiple fronts: cross-modal retrieval, per-effect retrieval, chain-length generalization, and an out-of-distribution reverb retrieval benchmark. The inclusion of the RIR benchmark using professional plugins unseen in training is a significant strength, demonstrating generalizability. The comparison between T5 and BoxGraph provides valuable insights into the trade-offs between sequence-based and graph-based code encoders. The results show that BoxGraph excels at topology recovery when parameters are masked, while T5 generalizes better to longer chains.
Reproducibility
The paper is highly reproducible. The authors release their codebase, model weights, and an interactive tool. Detailed hyperparameters, dataset construction steps, and training configurations are provided. The use of standard architectures (T5, CNN14) and well-documented libraries (Faust, DawDreamer) further aids reproducibility.
Limitations
The primary limitation is the reliance on the Faust language, which, while powerful, is not the most common DSP language (e.g., C++ VSTs are more prevalent). The "sim-to-real" gap is acknowledged, as training data uses randomly sampled parameters and potentially non-dry sources. The out-of-domain evaluation is limited to reverb, and a broader range of effect types would strengthen the generalizability claims. The lack of a user study is a minor weakness for a representation learning paper, but the objective metrics are robust.
Broader impact
This work lays the groundwork for advanced Music Information Retrieval (MIR) systems that can search, generate, and condition on DSP code. The ability to retrieve effect chain topologies from audio without supervised labels is a significant step towards automated audio engineering tools. The dual understanding of topology and parameters could enable new applications in style transfer, dry stem recovery, and efficient parameter search. The paper introduces the first joint embedding of audio effect code and output audio, demonstrating that graph-based encoders of DSP code structure can effectively align with audio representations. The rigorous evaluation, including out-of-distribution benchmarks and detailed ablations of code encoders, establishes a strong foundation for future research in multimodal audio-DSP alignment.
Author of this paper? Send a correction.