Taste-aware music retrieval from audio embeddings
Abstract
Crossmodal correspondences between sound and taste are well established in psychology and neuroscience, but largely absent from content-based multimedia retrieval. We formalise taste-from-audio prediction as a content-based music information retrieval benchmark over a perceptually validated multi-source corpus, comparing ten frozen audio encoders from the four HEAR families under a shared multi-task regression head, with gated late-fusion as a configurable variant. In order to assess the effectiveness of the models, we compute absolute error and rank correlation. The strongest systems predict the five tastes within a macro RMSE of 0.134; on held-out real music their error is less than half a single rater's deviation from the consensus (RMSE 0.13 vs. 0.28), so the model tracks the group consensus more closely than an average human rater, and well below the previous state of the art baseline (0.219). On absolute error the encoders are statistically flat, with a single VGGish matching the best fusion, but gated late-fusion's advantage is confined to rank correlation (macro Pearson r 0.724 vs. 0.666). Operationalised as a content-based retrieval index, the predicted taste space ranks a 309-item pool far more faithfully than a CLAP-text baseline, which sits at chance; ridge probes and an audio-bandstop knockout read the strongest representations against documented sound-taste correspondences.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a well-executed benchmark for taste-from-audio prediction, demonstrating that frozen audio encoders can predict taste intensities with accuracy exceeding average human raters, while providing valuable insights into the spectral features underlying crossmodal correspondences.
Methodology
The paper proposes a systematic benchmark for predicting five basic taste intensities from audio embeddings. The methodology is straightforward: it freezes ten pre-trained audio encoders (spanning supervised, self-supervised, multimodal, and music-specific families) and trains a lightweight multi-task regression head (MLP) on top. A key methodological contribution is the "gated late-fusion" mechanism to combine complementary encoders. The approach is technically sound but not architecturally novel; it applies standard transfer learning and ensemble techniques to a new domain (crossmodal taste prediction). The use of masked MSE loss to handle partial annotations is a practical and necessary detail. The interpretability layer using ridge probes and bandstop knockouts is a strong methodological addition, linking model behavior to psychophysical literature.
Results & evaluation
The experimental setup is rigorous for the domain. The authors evaluate on a perceptually validated corpus, comparing against a previous SOTA. They provide a detailed breakdown of performance by taste and by data source (real vs. generated music). The comparison against human inter-rater agreement is a compelling metric, showing the model outperforms average human raters on absolute error for real music. The retrieval experiments, while limited by the small pool size (309 items), effectively demonstrate the utility of the predicted taste space for content-based retrieval, showing it outperforms text-based baselines. The statistical analysis includes bootstrap tests, adding credibility to the encoder rankings.
Reproducibility
The paper provides significant detail for reproducibility. It specifies the encoders, the architecture of the head (hidden size, dropout, activation), training hyperparameters (AdamW, LR, batch size), and the fusion strategy. Crucially, it provides code and dataset links. The use of frozen encoders and cached embeddings makes the computational cost low and the setup easy to replicate. The five-seed averaging for reported metrics is a good practice for small datasets.
Limitations
The primary limitation is the small dataset size (269 train, 40 test clips), which restricts the generalizability of the findings and the complexity of the models that can be effectively trained. The authors acknowledge this and frame the work as a benchmark rather than a scalable solution. The crossmodal mappings are largely based on Western psychophysics, potentially limiting cross-cultural applicability. The "spicy" label is noted as less constrained by prior evidence. The retrieval evaluation is limited to a small pool, which may not reflect performance on large-scale collections.
Broader impact
This work opens a new avenue for content-based music retrieval and recommendation by introducing "taste" as a semantic axis. It bridges the gap between MIR and psychophysics/neuroscience, providing a computational framework for studying crossmodal correspondences. The potential applications in sonic seasoning, multisensory design, and personalized audio experiences are significant. It also highlights the limitations of current text-based multimodal models (like CLAP-text) in capturing non-linguistic perceptual qualities. The paper presents a well-executed benchmark for taste-from-audio prediction, demonstrating that frozen audio encoders can predict taste intensities with accuracy exceeding average human raters, while providing valuable insights into the spectral features underlying crossmodal correspondences.
Author of this paper? Send a correction.