What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models
Abstract
Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper makes a significant contribution to the understanding of representation quality in music foundation models by introducing a systematic layer-wise analysis and a novel pitch-transposition equivariance metric, demonstrating that intrinsic properties can effectively guide layer selection, particularly for tonal tasks where standard metrics fail.
Methodology
The paper presents a rigorous and systematic layer-wise analysis of 12 music foundation models across three distinct pre-training paradigms (masked, autoregressive, contrastive). The methodology is sound, employing a comprehensive suite of label-free intrinsic metrics (Intrinsic Dimension, Curvature, Anisotropy, Effective Rank, LiDAR, InfoNCE) to characterize representation geometry. The key methodological contribution is the introduction of Pitch-Transposition Equivariance (PTE), a novel metric designed to capture tonal structure that standard geometric metrics miss. The approach of correlating these intrinsic properties with downstream probe performance across 15 diverse MIR tasks provides a robust framework for understanding representation quality without relying solely on task-specific supervision.
Results & evaluation
The experimental evaluation is extensive, covering a wide range of downstream tasks including tonal, rhythmic, timbral, semantic, and similarity tasks. The results are well-supported, demonstrating that while standard metrics correlate well with non-tonal tasks, they fail for tonal tasks, necessitating the new PTE metric. The paper effectively demonstrates that intrinsic metrics can serve as effective proxies for layer selection, often outperforming trainable multi-layer fusion methods, particularly in low-data regimes. The analysis of depth-wise trends across different model families provides valuable insights into how representation properties evolve.
Reproducibility
The paper provides significant detail on the models, datasets (MTG-Jamendo, GiantSteps, NSynth, etc.), and evaluation protocols. The code and extended results are available on the project page, enhancing reproducibility. The use of standard datasets and publicly available models facilitates independent verification.
Limitations
The study is correlational; it identifies properties associated with good performance but does not establish causality. The metrics are evaluated on frozen representations, so their utility for guiding pre-training or fine-tuning is not directly addressed. The analysis is limited to 12 models, which, while diverse, may not cover all architectural variations. The PTE metric, while promising, is specific to tonal tasks and may not generalize to other musical attributes.
Broader impact
This work provides practical guidelines for selecting layers in music foundation models, potentially reducing the computational cost of model evaluation and deployment. It advances the theoretical understanding of representation quality in audio models, bridging the gap between geometric analysis and practical MIR performance. The findings are relevant to the broader community working with self-supervised audio representations. This paper makes a significant contribution to the understanding of representation quality in music foundation models by introducing a systematic layer-wise analysis and a novel pitch-transposition equivariance metric, demonstrating that intrinsic properties can effectively guide layer selection, particularly for tonal tasks where standard metrics fail.
Author of this paper? Send a correction.