Harmonic Representations in Symbolic Music Transformers are Localized and Controllable
Abstract
Transformer models trained on symbolic music have become remarkably capable. However, it is yet unclear how they represent music structure, and whether understanding their internal representations can be used to unlock methods for model control. We examine this through a case study of harmonic analysis: when a music transformer navigates a cadence, what does it internally "know" about its harmonic context, and can that knowledge be used to affect the generation? Using Aria, a LLaMA-based symbolic music transformer evaluated on Bach chorales, we find that harmonic decisions are causally localized progressively from middle layers onward. Linear probes show that mode becomes decodable in the same layer range where patching identifies growing causal effect, while key identity emerges later. We then confirm these representations are actionable for music generation control: steering via latent directions corresponding to harmonic changes reliably produces a deceptive or minor resolution, while preserving structural coherence and generation quality. This demonstrates that, at inference time, the model's internal harmonic knowledge can be directly manipulated to control the harmonic output.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper demonstrates that harmonic decisions in a symbolic music transformer are causally localized in middle layers and can be manipulated via latent steering to control cadence resolution. By combining causal interventions, linear probing, and representation steering on Bach chorales, the authors reveal that the model's internal geometry mirrors music theory (with relative keys closer than parallel keys) and that these representations are actionable for inference-time control, offering a new pathway for interpretable and controllable symbolic music generation.
Methodology
The paper employs a rigorous mechanistic interpretability framework applied to a symbolic music transformer (Aria). The methodology is sound, utilizing three complementary techniques: (1) Causal interventions (activation patching/interchange interventions) to localize harmonic computation, (2) Linear probing to assess decodability of harmonic attributes (mode, key), and (3) Representation steering to test actionability. The construction of counterfactual pairs via minimal single-note edits to Bach chorales is a clever and effective way to isolate specific harmonic decisions (cadence resolution) from other musical features. The derivation of steering vectors from probe weights is a standard but effective technique in this domain. The analysis of the geometric relationship between relative and parallel keys in the latent space provides a strong theoretical link between model internals and music theory.
Results & evaluation
The experiments are well-designed and comprehensive. The use of Bach chorales is appropriate for this specific query (harmonic structure), though it limits generalizability. The evaluation metrics are robust, combining structural integrity checks (error rates, repetition), perceptual quality (FMD), and harmonic specificity (Roman numeral entropy, top-1 dominance). The results clearly demonstrate that harmonic representations are localized in middle layers and that steering these representations effectively alters the harmonic outcome (e.g., shifting from authentic to deceptive cadences) without degrading overall musical coherence. The finding that parallel key steering is more effective than relative key steering due to latent space geometry is a significant and insightful result.
Reproducibility
The paper provides sufficient detail for reproducibility. It specifies the model (Aria), the dataset (Bach chorales), the tool for harmonic analysis (music21), and the specific intervention methods (layers, alpha values, probe training details). The code and data are available on the project webpage. The specific layer ranges for intervention (5-10 for mode, 11-15 for key) are clearly stated.
Limitations
The primary limitation is the narrow scope of the dataset (Bach chorales). The authors acknowledge that whether these findings hold for more chromatic or less conventional music is an open question. The model used (Aria) is a specific LLaMA-based transformer; results may not generalize to other architectures (e.g., diffusion models for audio, or different tokenization schemes). The steering is limited to harmonic resolution at cadences; it is unclear if this method can control other musical aspects like melody or rhythm with similar precision.
Broader impact
This work contributes to the growing field of mechanistic interpretability in generative models, specifically for music. It demonstrates that high-level musical concepts are not just distributed noise but are structured and manipulable. This has implications for controllable music generation, allowing users to steer harmonic progressions without retraining. It also provides insights into how neural networks internalize symbolic rules, bridging the gap between music theory and machine learning. The paper demonstrates that harmonic decisions in a symbolic music transformer are causally localized in middle layers and can be manipulated via latent steering to control cadence resolution. By combining causal interventions, linear probing, and representation steering on Bach chorales, the authors reveal that the model's internal geometry mirrors music theory (with relative keys closer than parallel keys) and that these representations are actionable for inference-time control, offering a new pathway for interpretable and controllable symbolic music generation.
Author of this paper? Send a correction.