CODA: Cascaded Online Discontinuity-Aware Alignment for Real-Time Image-Based Score Following
Abstract
Real-time score following from sheet images remains chal- lenging because the model must process streaming au- dio while resolving highly repetitive visual patterns un- der strict latency constraints. Recent image-based meth- ods have attempted to use multi-resolution prediction by simultaneously predicting the positions of the active sys- tem, bar, and note. However, their predictions across these different levels of notation are independent, which makes the predictions unstable and introduces unnecessary ex- tra search space for bar- and note-level predictions. Most existing methods also lack mechanisms to recover from score discontinuities, such as repeats, da capo (D.C.), or coda jumps. This paper proposes CODA, to the best of our knowledge, the first real-time score following system that addresses both gaps. CODA explicitly exploits the cascaded structure of music scores: it first selects the ac- tive system, then the active bar within it, and finally the active note within the selected bar. This enforces pre- diction consistency across resolutions. A silence-driven break mode enables recovery from arbitrary score discon- tinuities without requiring knowledge of the repeat struc- ture. Evaluated on the Multimodal Sheet Music Dataset (MSMD) piano benchmarks, CODA achieves state-of-the- art tracking accuracy and discontinuity-recovery perfor- mance under real-time throughput. Code is available at https://github.com/ValleyC/CODA.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
Methodology
The paper proposes CODA, a cascaded online alignment framework for real-time image-based score following. The core methodological innovation is the reformulation of the tracking problem from independent multi-resolution object detection (as seen in prior work like CYOLO-SB) to a cascaded selection-and-regression task. By explicitly predicting the system, then the bar within that system, and finally the note within that bar, the model enforces geometric consistency and significantly reduces the search space at each stage. The architecture utilizes a causal Mamba encoder for audio processing, which efficiently handles streaming data with linear complexity, paired with a CNN visual backbone. A key technical contribution is the "silence-driven break mode," which leverages the observation that score discontinuities (repeats, da capo, coda jumps) are often preceded by silence. This allows the system to reset its temporal priors and re-localize without requiring explicit symbolic knowledge of the score's repeat structure. The use of beam search with learned temporal priors further stabilizes the predictions.
Results & evaluation
The evaluation is conducted on the Multimodal Sheet Music Dataset (MSMD), a standard benchmark for this task. The authors report state-of-the-art results in tracking accuracy on both synthetic and real-audio settings compared to strong baselines like CYOLO-SB. Crucially, they introduce a new "repeat-aware jump test benchmark" with manually annotated repeat structures for the 94 test pieces, addressing a significant gap in the literature where discontinuity handling was previously unstandardized. The results show substantial improvements in system and bar accuracy, and notably, CODA demonstrates robust recovery from jumps, whereas baselines fail. The ablation studies effectively isolate the contributions of the cascade structure, beam search, and cross-attention mechanisms.
Reproducibility
The paper provides detailed implementation details, including audio preprocessing (log-filterbank, hop size), model architecture dimensions, training schedules (two-phase curriculum with scheduled sampling), and hyperparameters. The code is made publicly available on GitHub. The dataset (MSMD) is publicly accessible. The inclusion of a specific benchmark for jump recovery enhances reproducibility for future work in this niche area.
Limitations
The method is evaluated exclusively on solo piano music from the MSMD dataset. The performance on more complex musical textures (chamber music, orchestral) or different instruments is not reported. The reliance on synthesized audio for the primary evaluation setting (Setting I) may not fully capture the challenges of real-world acoustic recordings, although Setting II provides some real-audio validation. The "silence" assumption for jump recovery, while effective for many classical pieces, might fail in continuous musical passages where performers execute jumps without significant pauses.
Broader impact
This work advances the field of Human-Computer Interaction (HCI) in music, specifically interactive accompaniment and automated page-turning systems. By enabling robust real-time score following from images without requiring symbolic scores, it lowers the barrier to entry for using digital sheet music in performance settings. The standardized benchmark for discontinuity handling will facilitate fairer comparisons in future research. This paper presents a significant methodological advancement in real-time image-based score following by introducing a cascaded selection architecture and a novel silence-driven recovery mechanism, effectively addressing the critical challenges of geometric consistency and score discontinuity handling.
Author of this paper? Send a correction.