Do Text-to-Music Models Really Follow Instructions? A Counterfactual Evaluation of Key and Beat Grouping
Abstract
Prompted attribute agreement is widely used as evidence of text-to-music controllability, yet a requested attribute may occur simply because it is already common in the model's output distribution. We introduce a matched counterfactual evaluation that separates target occurrence from instruction-attributable control. Each family contains a neutral input that omits the scored attribute and two otherwise matched inputs that swap the requested target. All three are rendered through frozen native-interface adapters with a shared seed. Applied to global key and beat grouping in three open systems, this design changes the empirical conclusion. ACE-Step 1.5 and Stable Audio 3 Medium exhibit substantial key control, whereas LeVo2 does not. For beat grouping, the same models redirect toward the rare three-beat target, but high four-beat agreement is largely inherited from neutral outputs: Stable Audio 3 produces four-beat grouping in 0.97 of neutral cases but only 0.56 under its explicit four-beat treatment. Off-attribute placebos, external recognizer validation, blind expert annotation, and multi-seed sentinels support the attribution. When targets have unequal output priors, agreement describes what a model produced, while matched neutral and target-swap contrasts test whether the instruction changed it.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel counterfactual evaluation framework that rigorously distinguishes between attribute occurrence and instruction-attributable control in text-to-music models, revealing that high agreement rates often reflect data priors rather than genuine controllability, thereby providing a more accurate and actionable benchmark for the field.
Methodology
The paper introduces a rigorous counterfactual evaluation framework for text-to-music models, specifically targeting the distinction between "prompted attribute agreement" (mere occurrence) and "instruction-attributable control" (causal influence of the prompt). The core methodological contribution is the "matched neutral--A--B contrast family" design. By generating audio with a neutral prompt (omitting the target attribute) and two treatment prompts (specifying different target values) using a shared seed and frozen adapters, the authors isolate the effect of the instruction from the model's inherent output priors. This approach is statistically sound and addresses a critical flaw in current evaluation practices where high agreement rates may simply reflect the model's bias toward common attributes (e.g., 4/4 time). The metrics proposed—Treatment Agreement, Enhancement ($\Delta$), and Margin—are well-defined and provide a granular view of controllability.
Results & evaluation
The evaluation is applied to three prominent open systems: ACE-Step 1.5, Stable Audio 3 Medium, and LeVo2. The experiments cover global key and beat grouping. The results are compelling and counter-intuitive to standard benchmarks: while ACE-Step and Stable Audio 3 show strong key control, LeVo2 shows little attributable response. More importantly, for beat grouping, the paper demonstrates that high four-beat agreement in Stable Audio 3 is largely inherited from neutral outputs (0.97 neutral rate), and the explicit instruction actually *decreases* the probability of four-beat grouping (negative $\Delta$). This finding fundamentally changes the empirical conclusion about these models' capabilities. The use of off-attribute placebos, external recognizer validation, blind expert annotation, and multi-seed sentinels adds significant robustness to the claims.
Reproducibility
The paper provides extensive details on the experimental setup, including frozen native-interface adapters, specific decoding settings, and seed handling. The authors release a code-and-data package with instantiated cases, prompts, configurations, and analysis code. The use of deterministic seeds and frozen model revisions ensures that the generation process is reproducible. The detailed appendix provides literal renderings of inputs and full provenance, making it highly reproducible.
Limitations
The evaluation is limited to global key and beat grouping. It does not address local structural edits, modulation, or continuous controls. The neutral contrast necessarily changes the rendered carrier (e.g., omitting a field vs. filling it), which introduces a confound between the absence of the attribute and the change in prompt structure. The authors acknowledge this and use A/B swaps and off-attribute placebos to mitigate it, but a factorial design crossing carrier templates would be stronger. The evaluation is also limited to the specific public interfaces of the models; internal architectural changes that might improve control are not explored.
Broader impact
This work has significant implications for the development and evaluation of generative audio models. By establishing that "agreement" is not sufficient evidence of "control," it provides a new standard for benchmarking. This can guide model developers to focus on improving genuine instruction-following capabilities rather than just aligning with data priors. It also helps practitioners choose models based on their actual controllability profiles (e.g., good at rare targets vs. common targets). The framework is generalizable to other multimodal generation tasks where output priors are strong. The paper presents a novel counterfactual evaluation framework that rigorously distinguishes between attribute occurrence and instruction-attributable control in text-to-music models, revealing that high agreement rates often reflect data priors rather than genuine controllability, thereby providing a more accurate and actionable benchmark for the field.
Author of this paper? Send a correction.