Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment
Abstract
Music generation has advanced markedly through multimodal deep learning, enabling models to synthesize audio from text and, more recently, from images. However, existing image-conditioned systems suffer from two fundamental limitations: (i) they are typically trained on natural photographs, limiting their ability to capture the richer semantic, stylistic, and cultural content of artworks; and (ii) most rely on an image-to-text conversion stage, using language as a semantic shortcut that simplifies conditioning but prevents direct visual-to-audio learning. Motivated by these gaps, we introduce ArtSound, a large-scale multimodal dataset of 105,884 artwork-music pairs enriched with dual-modality captions, obtained by extending ArtGraph and the Free Music Archive. We further propose ArtToMus, the first framework explicitly designed for direct artwork-to-music generation, which maps digitized artworks to music without image-to-text translation or language-based semantic supervision. The framework projects visual embeddings into the conditioning space of a latent diffusion model, enabling music synthesis guided solely by visual information. Experimental results show that ArtToMus generates musically coherent and stylistically consistent outputs that reflect salient visual cues of the source artworks. While absolute alignment scores remain lower than those of text-conditioned systems-as expected given the substantially increased difficulty of removing linguistic supervision-ArtToMus achieves competitive perceptual quality and meaningful cross-modal correspondence. This work establishes direct visual-to-music generation as a distinct and challenging research direction, and provides resources that support applications in multimedia art, cultural heritage, and AI-assisted creative practice. Code and dataset will be publicly released upon acceptance.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of ArtToMus, a framework for direct artwork-to-music generation that eliminates the need for textual intermediaries, thereby enhancing the potential for richer cross-modal learning. This work significantly advances the field of multimodal deep learning by addressing existing limitations and providing a comprehensive dataset and methodology for future exploration.
Methodology
The paper introduces a novel framework, ArtToMus, for direct artwork-to-music generation that bypasses the traditional image-to-text conversion, which is a significant advancement in multimodal learning. The methodology is well-structured, detailing the construction of a large-scale multimodal dataset and the architecture of the proposed model. The use of latent diffusion models for music synthesis based solely on visual embeddings is innovative, and the introduction of a dedicated image-to-audio interface enhances the model's ability to capture the nuances of artistic expression in music. The approach is theoretically sound, leveraging existing technologies like CLIP and GPT-2 while innovatively combining them for a new application.
Results & evaluation
The experimental setup is robust, utilizing a large dataset of 105,884 artwork-music pairs, which is a significant contribution to the field. The evaluation metrics are comprehensive, including KL-Divergence, Fréchet Audio Distance, and ImageBind Score, which effectively assess the quality of generated audio in relation to the visual inputs. The results demonstrate that the proposed model achieves competitive perceptual quality and meaningful cross-modal correspondence, although absolute alignment scores are lower than text-conditioned systems due to the inherent complexity of the task.
Reproducibility
The paper mentions that code and dataset will be publicly released upon acceptance, which is a positive aspect for reproducibility. However, specific implementation details, such as hyperparameters and training configurations, are provided, which aids in replicating the experiments. The clarity in the methodology section enhances the likelihood of successful reproduction by other researchers.
Limitations
One limitation noted is the lower absolute alignment scores compared to text-conditioned systems, which is expected given the increased difficulty of direct visual-to-audio learning. Additionally, the reliance on the quality of the dataset and the potential biases in artwork-music pairings could affect the model's generalizability. The paper could also benefit from a more detailed discussion on the limitations of the proposed architecture and potential challenges in real-world applications.
Broader impact
The implications of this research extend to various domains, including multimedia art, cultural heritage, and AI-assisted creative practices. By enabling direct visual-to-music generation, the work opens new avenues for artistic expression and could foster innovative collaborations between artists and AI technologies. The establishment of a large-scale dataset also provides a valuable resource for future research in multimodal learning. The main contribution of this paper is the introduction of ArtToMus, a framework for direct artwork-to-music generation that eliminates the need for textual intermediaries, thereby enhancing the potential for richer cross-modal learning. This work significantly advances the field of multimodal deep learning by addressing existing limitations and providing a comprehensive dataset and methodology for future exploration.
Author of this paper? Send a correction.