Separate-and-Detect: Unified Drum Transcription and Stem Generation via Latent Diffusion
Abstract
Automatic Drum Transcription (ADT) is commonly formulated as a direct mapping from a music mixture to symbolic drum events. While effective for transcription, this formulation discards the acoustic stems that are useful for editing, remixing, and production. We revisit an alternative separate-and-detect formulation, where a drum source separation front end first produces five editable drum stems, and a fixed onset detector then converts each stem into symbolic events. The separator is built on a five-stem latent diffusion model that jointly generates kick, snare, toms, hi-hats, and cymbals in a compact VAE latent space. We further study two training-only auxiliary branches--an onset branch (OB) and a timbre branch (TB)--which shape the separator during learning but are discarded at inference. Trained on synthetic drum multitracks and evaluated on MDB Drums and ENST-Drums, the proposed pipeline consistently improves over a strong U-Net-based drum separation baseline in overall transcription F1. It also outperforms a representative end-to-end ADT system on kick and snare F1 under our evaluation protocol, while additionally providing separated audio stems. The ablation results show that OB gives the most stable transcription gains, whereas TB changes the trade-off between reconstruction, acoustic stem quality, and onset detection. These results suggest that generative drum demixing can serve not only as a source separation model, but also as a practical front end for interpretable drum transcription.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel "separate-and-detect" framework for automatic drum transcription that leverages a latent diffusion model with training-only auxiliary branches to optimize for both audio separation quality and transcription accuracy, demonstrating significant improvements over existing baselines on standard datasets.
Methodology
The paper proposes a "separate-and-detect" pipeline for Automatic Drum Transcription (ADT). Instead of direct event prediction, it uses a five-stem latent diffusion model (based on MSG-LD) to separate drum stems (kick, snare, toms, hi-hats, cymbals) from a drum-dominant mixture, followed by a fixed onset detector. The core methodological contribution is the introduction of two training-only auxiliary branches: an Onset Branch (OB) that encourages sparse percussive structure, and a Timbre Branch (TB) that regularizes acoustic features using DrumGAN descriptors. This design aims to optimize the generative model not just for audio reconstruction, but for downstream transcription utility. The approach is technically sound, leveraging the strengths of diffusion models for high-fidelity separation while addressing the specific needs of rhythmic transcription through auxiliary supervision.
Results & evaluation
The evaluation is conducted on MDB Drums and ENST-Drums datasets. The authors compare their method against a strong U-Net baseline (LarsNet) and an end-to-end ADT system (ADTOF). Results show that the proposed method, particularly with the +OB auxiliary branch, significantly improves overall transcription F1 over LarsNet and outperforms ADTOF on kick and snare classes. The ablation study effectively isolates the impact of the auxiliary branches, showing that OB provides stable transcription gains, while TB influences the trade-off between reconstruction quality and timbral accuracy. The use of Fréchet Audio Distance (FAD) and mel-MSE provides a comprehensive view of separation quality, complementing the transcription metrics.
Reproducibility
The paper provides clear descriptions of the dataset preprocessing, model architecture modifications, and training details. The code and demo are publicly available on GitHub and a dedicated website, which significantly enhances reproducibility. The use of standard datasets (MDB, ENST) and established baselines (ADTOF, LarsNet) allows for fair comparison. The specific hyperparameters for the onset detector tuning are mentioned, though the exact random seeds are not explicitly detailed in the text, the public code likely contains them.
Limitations
The authors acknowledge several limitations. The system operates at 16 kHz, which limits high-frequency content recovery, affecting hi-hats and cymbals. The iterative nature of diffusion sampling makes inference slower (approx. 25 seconds for 10 seconds of audio) compared to single-pass models. The method struggles with dense, broadband textures like cymbals due to inter-class confusion and VAE latent compression. Additionally, the reliance on a fixed upstream source separation model (Demucs) means errors in the initial drum extraction cannot be corrected by the downstream components.
Broader impact
This work contributes to the field of audio signal processing and music information retrieval by demonstrating that generative models can be optimized for symbolic transcription tasks. The ability to provide both editable stems and symbolic events is valuable for music production, education, and analysis. The "separate-and-detect" paradigm offers a new perspective on ADT, potentially inspiring similar hybrid approaches in other domains where both audio fidelity and symbolic interpretation are required. The paper presents a novel "separate-and-detect" framework for automatic drum transcription that leverages a latent diffusion model with training-only auxiliary branches to optimize for both audio separation quality and transcription accuracy, demonstrating significant improvements over existing baselines on standard datasets.
Author of this paper? Send a correction.