On a Separate Note: Robust Score-Informed Note Separation with a Two-Stream TFC-TDF U-Net and Adaptive Set Ownership
Abstract
Score-informed note separation seeks to extract the performed waveform of all individual notes, often from a polyphonic recording. Existing deep learning systems generally only target instrument-level stems. We present, to our knowledge, the first deep learning approach to score-informed note separation, NoteSep. NoteSep extracts the queried notes by applying an extraction stage model, NoteGrab, once per note. Conditioned on pitch, onset, and offset, NoteGrab separates harmonic and percussive components in two U-Nets linked by bidirectional cross-attention; selective harmonic gating suppresses lower-octave interference while preserving percussive attacks. Finally, a joint separation stage applies Adaptive Set Ownership (ASO) to compare concurrent NoteGrab estimates and reallocate mixture energy. We curate SCNS-Train (25,729 mixtures and 743,920 targets) for training and SCNS-Eval (16 instruments, disjoint scores and libraries) for evaluation. On SCNS-Eval, NoteSep reaches a median SI-SDR of 7.39~dB, compared with 2.49~dB for our strongest baseline. See the demo page at https://benschou.com/notesep.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the first deep learning framework for score-informed note separation, achieving state-of-the-art performance by combining a dual-stream extraction network with a novel joint energy reallocation mechanism. By addressing the specific challenges of harmonic overlap and mixture consistency, NoteSep provides a robust tool for note-level audio manipulation, significantly advancing the field of music source separation beyond instrument-level stems.
Methodology
The paper proposes NoteSep, a two-stage framework for score-informed note separation. The first stage, NoteGrab, utilizes a dual-stream TFC-TDF U-Net architecture to process harmonic and percussive components (via HPSS) separately, linked by bidirectional cross-attention. A key methodological contribution is the "selective harmonic gating," which conditionally suppresses lower-octave interference based on score alignment, addressing a specific physical challenge in polyphonic music. The second stage, Adaptive Set Ownership (ASO), is a novel joint separation module that reallocates mixture energy among concurrent note estimates to enforce consistency, using a learned gate and log-gain prediction. This approach effectively tackles the "double counting" problem inherent in independent note extraction.
Results & evaluation
The authors curate a substantial training set (SCNS-Train) and a disjoint evaluation set (SCNS-Eval) covering 16 instruments. The evaluation is rigorous, comparing against both traditional methods (Score-Informed NMF) and commercial software (Melodyne). The results show a significant improvement in SI-SDR (7.39 dB vs 2.49 dB for the strongest baseline). Furthermore, the paper evaluates the system on real-world ensemble recordings (PHENICX-Anechoic, Bach10) by aggregating note estimates into instrument stems, demonstrating competitive performance against specialized instrument separation models. The inclusion of an editing evaluation (SCNS-Edit) further validates the utility of the separated notes for downstream tasks.
Reproducibility
The paper commits to releasing code, model weights, and datasets. The architectural details are described with sufficient specificity (e.g., TFC-TDF v3, FiLM conditioning, specific loss functions), and the demo page provides audio examples. The use of standard libraries (librosa) for preprocessing aids reproducibility.
Limitations
The method relies heavily on accurate score alignment (pitch, onset, offset); errors in the input score will propagate to the separation quality. The computational cost is non-trivial, requiring K passes for K notes plus an ASO pass, which may limit real-time application on consumer hardware. The evaluation on real recordings is indirect (via instrument stem aggregation), so the fidelity of individual notes in complex, noisy real-world scenarios is not directly quantified with the same rigor as the synthetic benchmark.
Broader impact
This work enables precise, note-level audio editing, which has significant applications in music production, education (isolating specific notes for learning), and restoration. It bridges the gap between symbolic music information and audio signal processing, potentially facilitating more granular music information retrieval tasks. The paper introduces the first deep learning framework for score-informed note separation, achieving state-of-the-art performance by combining a dual-stream extraction network with a novel joint energy reallocation mechanism. By addressing the specific challenges of harmonic overlap and mixture consistency, NoteSep provides a robust tool for note-level audio manipulation, significantly advancing the field of music source separation beyond instrument-level stems.
Author of this paper? Send a correction.