Direct Conditioning of Audio Diffusion Transformers on fMRI Reveals Cortical Contributions to Sound Reconstruction
Abstract
Reconstructing audio from brain activity remains challenging. Existing brain-to-audio pipelines typically map neural responses to predefined stimulus representations before synthesis. While recent work in the visual domain has explored direct conditioning on brain activity, this approach has not yet been extended to audio diffusion models. We introduce MAAS (Multi-Area Auditory Synthesis), an end-to-end framework that conditions a pretrained audio diffusion transformer on ROI-wise fMRI responses without first supervising a decoder to predict an external stimulus representation. MAAS uses region-specific projections and learned brain tokens to condition Stable Audio through a dedicated cross-attention pathway. On a 7T fMRI dataset, we demonstrate consistent improvements over baselines, including two-stage fMRI-to-latent conditioning, across both acoustic and semantic evaluation metrics. In addition, we introduce a fidelity metric grounded in a computational model of auditory neural processing, which allows us to quantify the contribution of individual auditory cortical regions to reconstructed audio representations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a direct conditioning framework for audio diffusion transformers on fMRI data, demonstrating improved reconstruction fidelity and providing interpretable insights into cortical contributions. The technical contribution is significant in adapting diffusion transformer architectures for non-text, continuous neural signals, and the experimental design, including the novel biologically grounded fidelity metric, is rigorous and well-motivated.
Methodology
The paper proposes MAAS, a framework that directly conditions a pretrained audio diffusion transformer (Stable Audio) on fMRI data, bypassing the traditional two-stage pipeline of decoding neural signals into intermediate embeddings (like CLAP or YamNet) before synthesis. The core technical contribution is an adapter mechanism that injects region-specific fMRI projections into the cross-attention layers of the DiT backbone. This requires handling architectural mismatches, specifically preserving Rotary Positional Embeddings (RoPE) asymmetry and Grouped-Query Attention (GQA) structures while introducing a parallel "brain" branch for key-value projections. The method uses learned brain tokens to aggregate voxel-level activity from six bilateral auditory ROIs, preserving anatomical organization. The approach is sound in its theoretical motivation to avoid information bottlenecks inherent in intermediate representation prediction.
Results & evaluation
The experiments are conducted on a 7T fMRI dataset with 5 participants and 288 natural sounds. The evaluation is rigorous, comparing MAAS against two-stage baselines (Brain-to-text, Brain-only) and oracle caption-based conditions. The paper introduces a novel fidelity metric grounded in the NSL Auditory-Cortical Toolbox to quantify the contribution of individual cortical regions, which is a significant methodological addition. Results show MAAS outperforms baselines in CLAP similarity and pairwise identification. Ablation studies reveal a hierarchical contribution of auditory regions, with early areas (HG, PT) providing more self-sufficient spectrotemporal information. External validation on a 3T dataset further supports generalizability. However, the small sample size (5 participants) is a notable constraint.
Reproducibility
The authors provide code via an anonymous repository and a demo website with audio samples. Hyperparameters, training details (AdamW, cosine annealing, gradient clipping), and dataset preprocessing steps are described in detail. The use of standard libraries (fMRIprep) and open-source backbones (Stable Audio) enhances reproducibility. The specific voxel selection criterion (Pearson r with YamNet) is clearly defined, though it introduces a dependency on YamNet for preprocessing, which is explicitly stated not to be a conditioning signal.
Limitations
The primary limitation is the small dataset size (5 participants, 288 stimuli), which limits statistical power and generalizability claims. The temporal resolution of fMRI (seconds) is inherently low for audio, and the paper acknowledges this, noting that the model relies on spatial patterns rather than fine-grained temporal dynamics. The reliance on a specific 7T dataset for primary results, while external validation is provided, means the method's performance on standard 3T clinical data might vary. The "empty prompt" training strategy, while effective for isolating brain signals, may limit the model's ability to leverage semantic text cues if available in future applications.
Broader impact
This work bridges the gap between neuroscience and generative AI, offering a new paradigm for brain-to-sound reconstruction. The direct conditioning approach could be extended to other sensory modalities or different generative architectures. The interpretability provided by the ROI-level ablation analysis offers insights into the functional organization of the auditory cortex, potentially benefiting both computational neuroscience and the development of auditory brain-computer interfaces for clinical applications. The paper introduces a direct conditioning framework for audio diffusion transformers on fMRI data, demonstrating improved reconstruction fidelity and providing interpretable insights into cortical contributions. The technical contribution is significant in adapting diffusion transformer architectures for non-text, continuous neural signals, and the experimental design, including the novel biologically grounded fidelity metric, is rigorous and well-motivated.
Author of this paper? Send a correction.