Dual-BEATs: Unlocking Zero-Shot Stereo Audio Perception in Audio Large Language Models via Dithering
Abstract
Multimodal Large Language Models (LLMs) have remarkable semantic audio understanding, yet they remain "spatially agnostic" due to their reliance on mono-channel audio representations. Currently, spatial audio perception methods mainly focus on complex room simulations and custom-trained, geometry-aware stereo encoders, which limits their accessibility and generalizability. In this paper, we introduce the Dual-BEATs architecture, in which the left and right audio channels are routed independently through two identical semantic encoders as an alternative to specialized spatial modules. To circumvent the architectural bottleneck where internal normalization otherwise erases the inter-channel variance of stereo audio, we inject a static, uncorrelated dithering noise floor prior to encoding. This dithering intervention establishes a macro-variance floor that "smuggles" spatial geometry across the normalization layers. Evaluated on a ternary directional classification task (Left, Center, Right), we demonstrate that dithered models achieve exceptional spatial resolution--reaching up to 97.2% localization accuracy even on subtle 0.5 panning amplitudes--and demonstrates robust, zero-shot generalization to entirely unseen spatial configurations. Our results suggest that with the appropriate acoustic regularization, standard multimodal models are natively capable of generalized stereo audio understanding.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Dual-BEATs, a novel architecture that unlocks zero-shot stereo perception in Audio LLMs by using uncorrelated dithering noise to bypass normalization-induced spatial information loss, achieving high-resolution spatial localization without specialized encoders.
Methodology
The paper proposes a novel architectural intervention called "Dual-BEATs" to address the "spatial agnosticism" of standard Multimodal LLMs. The core insight is that standard normalization layers (LayerNorm/RMSNorm) act as compressors that erase subtle inter-channel amplitude differences (ICLD) essential for stereo perception. To bypass this, the authors inject static, uncorrelated Gaussian dithering noise into the left and right audio channels before encoding. This creates a "macro-variance floor" that allows the normalization layers to preserve the relative amplitude delta between channels, effectively "smuggling" spatial geometry into the LLM's latent space. The approach is theoretically grounded in Stochastic Resonance and avoids the need for complex, custom-trained spatial encoders or 3D room simulations. The methodology is elegant, leveraging existing frozen semantic encoders (BEATs) and standard LLM backbones (Gemma, OLMo) with minimal modification (QLoRA adapters).
Results & evaluation
The evaluation is rigorous and well-designed. The authors construct a controlled dataset using AudioSet samples, applying deterministic amplitude panning to create stereo pairs with varying panning amplitudes (PA). They evaluate on a ternary classification task (Left, Center, Right) and, crucially, test zero-shot generalization across unseen panning amplitudes. The results are compelling: undithered models suffer a "Center-Pan Crash" (performance drops to random chance for center-panned audio), while dithered models maintain high accuracy (>97%) across the entire spectrum, including subtle 0.5 PA shifts. The ablation studies on correlated vs. uncorrelated noise effectively demonstrate that the model is learning true spatial geometry rather than exploiting noise artifacts. The trade-off between spatial resolution and semantic retention ("Semantic Tax") is also quantified, showing a manageable ~5% drop in semantic F1.
Reproducibility
The paper provides extensive implementation details, including specific hyperparameters for dithering amplitude (DA=0.05), hardware specifications (V100/H100), and optimization settings (QLoRA, AdamW). The dataset curation process is described in detail, referencing specific snapshots of AudioSet. The strict isolation of random seeds for training and evaluation is a strong point for reproducibility and validity, preventing shortcut learning. The code is not publicly linked, but the method is simple enough to be reimplemented.
Limitations
The primary limitation is the reliance on simple amplitude panning (ICLD) rather than full binaural rendering (which includes Interaural Time Differences - ITD and Head-Related Transfer Functions - HRTFs). This limits the ecological validity of the spatial cues compared to real-world binaural recordings. Additionally, the evaluation is restricted to single-source scenarios; multi-source separation and localization are not addressed. The "Semantic Tax" indicates that while spatial perception is unlocked, it comes at the cost of some semantic fidelity, which may be a concern for downstream tasks requiring high-precision semantic understanding.
Broader impact
This work has significant implications for the development of spatially-aware AI assistants, accessibility tools for the visually impaired, and immersive media applications. By demonstrating that standard LLMs can be endowed with spatial perception through simple signal-level interventions, it lowers the barrier to entry for spatial audio understanding. However, the dual-use nature of such technology (e.g., enhanced surveillance or tracking) is acknowledged. The approach also highlights a fundamental architectural weakness in current multimodal models, potentially guiding future model designs to better preserve physical signal properties. The paper introduces Dual-BEATs, a novel architecture that unlocks zero-shot stereo perception in Audio LLMs by using uncorrelated dithering noise to bypass normalization-induced spatial information loss, achieving high-resolution spatial localization without specialized encoders.
Author of this paper? Send a correction.