StemFX: Learning Mixing Style Representations via Autoregressive FX Chain Prediction on Source-Separated Stems
Abstract
Audio mixing style encompasses the artistic and technical decisions a mix engineer makes, including level balancing, spatialization, and the choice, ordering, and parameterization of audio effects (FX) on each stem. FX chains are a key determinant of this style, yet existing approaches to modeling them remain limited. Some operate on stereo mixtures without explicit per-stem FX chain modeling, others fix the number or type of effects per track, and many require differentiable effect implementations or scarce multitrack datasets. We present StemFX, a framework that learns mixing style representations by autoregressively predicting variable-length FX chains on source-separated stems. A Transformer decoder predicts tokenized FX chains autoregressively, while a band-split multi-band CNN encoder with FiLM conditioning captures per-stem spectral structure. To enable large-scale paired training, we extract pseudo-stems from about 105K songs via source separation and augment them using MultiAFx, a toolkit unifying 85 audio effects from 7 Python libraries. Evaluated on mixing style retrieval, StemFX outperforms all baseline models across all tested chain lengths. On paired mixing style transfer, StemFX achieves the best spectral fidelity and the highest listener preference, over 4000 times faster than iterative optimization.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
Methodology
The paper proposes StemFX, a novel framework for learning mixing style representations by treating FX chain prediction as an autoregressive sequence generation problem. The architecture combines a band-split multi-band CNN encoder with FiLM conditioning (incorporating hand-crafted mixing features) and a Transformer decoder. A key methodological innovation is the "StemFX Pipeline," which leverages source separation on large-scale unpaired datasets (FMA) to create pseudo-stems, which are then augmented with random FX chains using a unified toolkit (MultiAFx) to generate paired training data at scale. This approach circumvents the scarcity of real multitrack datasets with ground-truth FX metadata. The tokenization scheme for variable-length FX chains (effect names, parameter names, and quantized values) is well-designed for autoregressive modeling. The use of FiLM conditioning to inject explicit mixing statistics into the CNN layers is a strong technical choice that bridges the gap between low-level spectral features and high-level mixing decisions.
Results & evaluation
The experimental evaluation is comprehensive and robust. The authors evaluate on two tasks: mixing style retrieval and paired mixing style transfer. For retrieval, StemFX significantly outperforms baselines (AFx-Rep, Fx-Encoder++, CLAP, and a controlled contrastive variant BSFiLM-CL) across various chain lengths, demonstrating that the learned embeddings capture mixing style distinct from musical content. The ablation studies effectively isolate the contributions of the encoder architecture, FiLM conditioning, and the data pipeline. The style transfer experiments are particularly compelling, showing superior spectral fidelity (MRSTFT) and human preference (MUSHRA scores) compared to fixed-chain methods and iterative optimization techniques (ST-ITO). The claim of being 4000x faster than iterative optimization is a significant practical advantage. The use of out-of-domain effects (pedalboard) in evaluation tests generalization well.
Reproducibility
The paper provides detailed implementation details, including hyperparameters, dataset sizes, and architecture specifications. The release of both the StemFX codebase and the MultiAFx toolkit on GitHub greatly enhances reproducibility. The description of the data generation pipeline, including source separation models and augmentation strategies, is clear enough for other researchers to replicate the training data generation process. The use of standard metrics (MRSTFT, MUSHRA) also aids in reproducibility of evaluation.
Limitations
The authors acknowledge several limitations. The model is restricted to the 85 effects included in the MultiAFx toolkit; adding new effects requires retraining. The reliance on source separation introduces potential artifacts from the separation model (e.g., SCNet), which may propagate into the learned representations. The use of pseudo-stems from different songs (cross-song mixing) may introduce musical incoherence, although the authors argue this decorrelates content from style. The model currently operates on four stems (vocals, bass, drums, other), which is a coarser decomposition than professional multitracks. Finally, the training data uses random effect chains, which may not reflect the structured, genre-dependent processing choices of human engineers.
Broader impact
StemFX has significant potential impact on music production technology. By enabling efficient and accurate mixing style transfer and retrieval, it can assist producers in learning mixing techniques, automate parts of the mixing workflow, and enhance music information retrieval systems. The open-source release of the toolkit and models promotes further research in audio effect modeling and automatic mixing. The ability to generate interpretable FX chains also provides insights into the decision-making process of mix engineers, potentially serving as an educational tool. This paper presents a significant advancement in automatic mixing and audio representation learning by introducing a scalable, autoregressive framework for FX chain prediction that overcomes data scarcity through a novel pseudo-stem generation pipeline, achieving state-of-the-art performance in both style retrieval and transfer tasks.
Author of this paper? Send a correction.