Beyond Piano: Cross-Instrument MIDI Velocity Estimation via Differentiable SoundFont Proxies
Abstract
Many music datasets contain MIDI notes but lack reliable velocities, defaulting to a constant value. This absence is especially problematic outside the piano domain, as velocity is a core component for expressive rendering, music generation, and performance analysis. This paper studies cross-instrument MIDI velocity estimation in this label-scarce setting. Starting from a piano-trained velocity estimator, we recast target-instrument adaptation as predicting renderer-conditioned velocities whose rendering matches the dynamics of the performance audio. This adaptation can be driven by either differentiable synthesizers (Diff-Synth) or our proposed differentiable SoundFont proxies (Diff-SFProxy). We highlight the Diff-SFProxy: it supervises velocity through note-wise, loudness-related acoustic parameters rather than waveform reconstruction, focusing gradients on velocity-dependent behavior. Experiments on piano and guitar show that Diff-SFProxy is effective for cross-instrument MIDI velocity estimation, while waveform-domain Diff-Synth degrades performance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a robust and innovative solution to cross-instrument MIDI velocity estimation by introducing a differentiable SoundFont proxy that leverages perceptual loudness parameters for gradient-based adaptation, effectively overcoming the limitations of waveform-based methods in label-scarce settings.
Methodology
The paper proposes a novel framework for cross-instrument MIDI velocity estimation in label-scarce settings. The core innovation is the "Diff-SFProxy," a differentiable proxy that maps MIDI note events to loudness-related acoustic parameters (Pitch-conditioned Harmonic Energy and Onset-window Spectral Flux) derived from SoundFont renders. This allows gradient-based adaptation of a piano-trained velocity estimator (VeloEst) to target instruments (guitar) without requiring ground-truth velocity labels. The methodology effectively addresses the "gradient mismatch" problem inherent in waveform-based differentiable synthesis (Diff-Synth), where timbral and room-acoustic mismatches dominate the loss landscape. The use of perceptual loudness metrics (Bark-scale specific/total loudness) for evaluation is a strong methodological choice, aligning the objective with the perceptual goal of velocity estimation.
Results & evaluation
The experimental design is rigorous and well-structured. The authors include a critical "velocity-recovery diagnostic" on synthetic audio to verify that the Diff-SFProxy gradients point in the correct direction, which is essential for trusting the adaptation process. The evaluation on real-world datasets (MAESTRO, SMD, GAPS, FL) demonstrates that Diff-SFProxy significantly outperforms both flat-velocity baselines and zero-shot transfer. Crucially, it shows that Diff-Synth often degrades performance on guitar, validating the paper's central hypothesis that parameter-space supervision is superior to waveform-space supervision for this specific task. The ablation studies on segment length and loss components further strengthen the findings.
Reproducibility
The paper provides substantial implementation details, including FFT settings, optimizer hyperparameters, and architecture specifications for the Transformer proxy. The code is explicitly linked via GitHub. The use of standard datasets (MAESTRO, GAPS) and public SoundFonts enhances reproducibility. The description of the training protocol, including the mixture sampler and caching strategy, is clear enough for replication.
Limitations
The approach relies on the availability of a suitable SoundFont for the target instrument, which requires manual selection or future automation. The recovered velocities are "renderer-conditioned," meaning they are optimal for the specific SoundFont used, not necessarily the canonical physical velocity of the performance. The method is currently limited to instruments with available SoundFonts and may struggle with complex polyphony or instruments with significant inharmonicity not captured by the proxy. The evaluation is limited to piano and guitar; generalization to other instruments (violin, wind) is speculative.
Broader impact
This work addresses a significant bottleneck in symbolic music processing: the lack of expressive velocity data for non-piano instruments. By enabling label-scarce adaptation, it facilitates the creation of more expressive MIDI representations for music generation, analysis, and education. The Diff-SFProxy framework could be extended to other expressive parameters (e.g., pedal, bow pressure), potentially enriching the symbolic music domain. The paper presents a robust and innovative solution to cross-instrument MIDI velocity estimation by introducing a differentiable SoundFont proxy that leverages perceptual loudness parameters for gradient-based adaptation, effectively overcoming the limitations of waveform-based methods in label-scarce settings.
Author of this paper? Send a correction.