Audio ML PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-18

Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy

Anurag Singh, Prajwal Singh, Shanmuganathan Raman

IIT Bombay (Inferred from PMRF/Vilas Mujumdar Chair) +1 · BMVC 2026

Affiliations: IIT Bombay (Inferred)

Abstract

Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them in distinct spatial regions. We formalize this as the geometric displacement phenomenon and expose it through a two-level clustering analysis comparing BEATs and CLAP embeddings at the raw encoder and projected CLIP token levels. Three geometric properties emerge as important: intra-class compactness, angular alignment with the CLIP text manifold, and inter-token spatial separation. We show that CLAP, whose contrastive language-audio pretraining aligns embeddings with natural language semantics, substantially outperforms BEATs on all three properties, improving the silhouette score by more than 3x and the Audio-Image Content score as well. Building on this analysis, we introduce a multi-token extension that requires no diffusion-backbone retraining and assigns each audio source to a distinct placeholder token in the CLIP tokenizer, complemented by a mixed inference-time objective that combines Attend-and-Excite with an A* spatial separation loss to resolve residual cross-token interference. For evaluation, we construct multi-audio compositional benchmarks from VGGSound (VGG2, VGG3, and VGGN) and extend six standard audio-image metrics to the multi-source setting. Across five benchmarks and four strong baselines, our method achieves strong results on the metrics most directly tied to compositional faithfulness, with AIC improvements of up to 2.25x over the strongest baseline and FID reduced by nearly 2x on two-source generation. We propose geometric displacement as a diagnostic framework for cross-modal generation systems that project learned embeddings into a pretrained contrastive space.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper identifies a critical geometric failure mode in multi-source audio-to-image generation and provides a principled, effective remedy through encoder substitution and inference-time optimization. By formalizing the "geometric displacement" phenomenon and constructing new compositional benchmarks, the work offers both a diagnostic tool and a state-of-the-art solution, significantly advancing the field of audio-conditioned generation.

Methodology

The paper proposes a geometric diagnosis for the failure of multi-source audio-to-image generation. The core insight is that audio embeddings (specifically from BEATs) are geometrically displaced in the CLIP text-token space, leading to poor cross-attention behavior. The authors formalize this using clustering metrics (silhouette, Davies-Bouldin) and angular alignment. The remedy involves two main components: (1) switching to CLAP embeddings, which are naturally aligned with the text manifold due to contrastive pretraining, and (2) a Multi-Token Extension (MTE) that assigns distinct placeholder tokens to each audio source, combined with an inference-time optimization objective that blends Attend-and-Excite (to ensure activation) and A* spatial separation losses (to prevent overlap). The methodology is sound, leveraging existing components (textual inversion, frozen diffusion backbone) but applying them in a novel geometric context. The redesign of the embedder for CLAP's global representation is a necessary and well-explained technical detail.

Results & evaluation

The experimental setup is robust. The authors construct three new benchmarks (VGG2, VGG3, VGGn) from VGGSound to evaluate compositional generation, which is a significant contribution as prior work lacked such multi-source benchmarks. They compare against four strong baselines (AudioToken, SonicDiffusion, Sound2Scene, MACS). The results show substantial improvements in Audio-Image Content (AIC) and FID, particularly for the CLAP-based variant. The ablation studies clearly isolate the contributions of the encoder choice (CLAP vs BEATs) and the inference-time losses. The observation that BEATs performs better out-of-distribution (IS3) while CLAP dominates in-distribution is a nuanced and valuable finding.

Reproducibility

The paper provides extensive implementation details in the supplementary material, including hyperparameters for the embedder, training schedules, and the specific configuration of the inference-time optimization. The code for the benchmarks and metrics is implied to be available or easily reconstructible given the detailed descriptions. The use of standard open-source components (Stable Diffusion, CLAP, BEATs) further aids reproducibility.

Limitations

The method assumes separated audio streams, which may not always be available in real-world scenarios (though they suggest using Demucs as a preprocessing step). The embedder is trained on a curated 50-class subset, limiting the vocabulary size compared to the full VGGSound dataset. The inference-time optimization increases computational cost by roughly 2.8-3.1x. The out-of-distribution performance of the CLAP variant is weaker than BEATs, indicating a potential vulnerability to unseen classes.

Broader impact

The geometric diagnosis framework is broadly applicable to any cross-modal generation system that projects learned embeddings into a pretrained contrastive space (e.g., video, 3D). The new benchmarks will be useful for the community to evaluate future multi-source audio generation models. The insights into the importance of angular alignment over Euclidean distance in cross-attention are valuable for designing future conditioning mechanisms. The paper identifies a critical geometric failure mode in multi-source audio-to-image generation and provides a principled, effective remedy through encoder substitution and inference-time optimization. By formalizing the "geometric displacement" phenomenon and constructing new compositional benchmarks, the work offers both a diagnostic tool and a state-of-the-art solution, significantly advancing the field of audio-conditioned generation.

Author of this paper? Send a correction.