What does the model actually see? Evaluation protocols and input availability in data-driven prediction of room acoustic parameters
Abstract
Machine-learnt models are increasingly used to predict ISO 3382-1 room acoustic parameters from sparse measurements, with reported coefficients of determination frequently above 0.85. This paper shows that such figures are often determined by the evaluation protocol rather than by the model. Using a multi-condition measurement campaign in a 264-seat conference hall and a 180-seat concert hall, three model families were evaluated under a factorial protocol ablation: validation splits either row-based or grouped by receiver position, and input features either including measured-at-test quantities or restricted to source-receiver geometry and environmental state. Row-based splits with measured-at-test inputs reproduce the high reported accuracies (mean $R^2$ 0.81 for the core parameters); grouping the splits by position and restricting inputs to information available at an unmeasured position reduces these to 0.09-0.57, reordering the apparent difficulty of parameter classes. A hybrid CNN evaluated with the target's own impulse response as input is shown to exploit it as a position fingerprint rather than as transferable acoustic information; training-only signal access yields no gain for any parameter tested, including reverberation time. Under the deployment-consistent protocol, the spread between Random Forest, the hybrid CNN, and inverse-distance weighting is an order of magnitude smaller than the spread between protocols for a fixed model; the learnt models retain a genuine advantage for sound strength and reverberation time, and the high accuracy of the original pipelines re-emerges as condition interpolation at measured positions (band means 0.80-0.88), a distinct and operationally useful task.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
Methodology
The paper employs a rigorous factorial ablation study to dissect the evaluation protocols of machine learning models in room acoustics. It introduces a taxonomy distinguishing between "condition interpolation" (predicting parameters at measured positions under varying environmental states) and "spatial prediction" (predicting parameters at unmeasured positions). The methodology involves re-evaluating existing pipelines (Random Forest, Hybrid CNN) under strict "deployment-consistent" protocols where validation folds are grouped by receiver position and inputs are restricted to geometry and environmental state. A key methodological insight is the demonstration that neural networks can exploit the target position's own impulse response as a "position fingerprint" rather than learning transferable acoustic features, particularly in venues with few measurement positions. This approach effectively isolates the source of high reported accuracy, revealing it to be largely an artifact of data leakage via correlated rows and privileged inputs.
Results & evaluation
The experimental design is robust, utilizing a multi-condition measurement campaign in two distinct halls (a concert hall and a conference hall). The authors systematically vary split designs (row-based vs. grouped) and input features (full vs. geometry-only). The results are compelling: while row-based splits with full inputs yield high $R^2$ scores (0.81), the deployment-consistent protocol reduces these significantly (0.09-0.57), with simple baselines like inverse-distance weighting often matching or exceeding complex neural networks for certain parameters (e.g., clarity). The paper provides a clear reordering of parameter difficulty, showing that sound strength and reverberation time are more predictable spatially than clarity parameters, which depend on fine-grained early reflection structures not captured by the available geometric features. The inclusion of a "privileged information" experiment further strengthens the argument by showing that training-time signal access does not confer a significant advantage for spatial prediction.
Reproducibility
The paper demonstrates high commitment to reproducibility. The author explicitly states that evaluation scripts, split definitions, random seeds, and per-fold results will be deposited in a public repository. The underlying measurement data is available upon request. The detailed description of the model architectures, feature sets, and experimental protocols allows for independent verification. The clear distinction between the two evaluation modes and the provision of specific $R^2$ values for each configuration enhance the transparency of the findings.
Limitations
The primary limitation is the scope of the data: the study is based on a single multi-condition campaign in only two halls. While the findings are likely generalizable to similar indoor environments, they may not hold for outdoor acoustics or highly complex, non-diffuse fields. The author acknowledges this and suggests cross-venue replication is needed. Additionally, the study focuses on specific ISO 3382-1 parameters; the generalizability to other acoustic metrics or perceptual qualities is not addressed. The network architecture, while effective for the task, is not novel in itself, and the paper's contribution lies in the evaluation rather than the model design.
Broader impact
This paper has significant implications for the field of data-driven acoustics and spatial audio. By exposing the fragility of high accuracy claims in sparse measurement scenarios, it urges the community to adopt more rigorous evaluation standards. The proposed reporting checklist provides a practical framework for future research, ensuring that claims about spatial prediction are substantiated by appropriate protocols. This work helps prevent the dissemination of misleading performance metrics and guides the development of more robust and truly generalizable models for room acoustic prediction. It also highlights the enduring value of simple baselines and the importance of understanding what models are actually learning. This paper provides a critical methodological audit of machine learning in room acoustics, demonstrating that reported high accuracies are often artifacts of evaluation protocols rather than model capability, and establishes a rigorous framework for distinguishing between condition interpolation and spatial prediction.
Author of this paper? Send a correction.