Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models
Abstract
Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
Methodology
The paper proposes IAAN (Identifying and Amplifying Acoustic Neurons), a training-free, label-free inference-time intervention method. The core methodology involves contrasting the activation of feed-forward neurons in the audio encoder of Large Audio-Language Models (LALMs) between the real audio waveform and a noise reference (Gaussian noise with matched length/variance). Neurons that activate more strongly on real audio are identified as "acoustic neurons" and their activations are amplified by a gain factor $g$ during inference. The approach is conceptually simple and elegant, leveraging the intuition that acoustic-specific neurons should be sensitive to the presence of structured audio versus unstructured noise. The method operates at the individual neuron level within the audio encoder, which is a finer granularity than previous interventions that typically target the language model backbone or output logits. The technical execution involves standard transformer operations (FFN activations) and does not require backpropagation or external datasets for calibration, relying instead on an internal contrastive signal.
Results & evaluation
The authors evaluate IAAN on three open-source LALMs (Audio-Flamingo-3, Qwen2.5-Omni, Kimi-Audio) and one specialized model (AF3-PD) using the VoxParadox benchmark, which tests ten non-semantic speech attributes (e.g., emotion, gender, pitch) where semantic content may contradict acoustic cues. The results are significant: IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. Crucially, the paper includes rigorous ablation studies. It compares IAAN against other inference-time interventions (Audio-aware decoding, LLM-amplify, Hidden-state steering) and demonstrates that intervening *inside* the encoder is necessary, as post-encoder interventions yield little to no gain. It also shows that the specific selection of neurons (via the acoustic score) is critical, as random selection or whole-layer amplification fails to reproduce the gains. The method also improves a model (AF3-PD) that was already fine-tuned for acoustic tasks, suggesting broad applicability. The use of a held-out development set (LISTEN) for hyperparameter tuning adds credibility, although the transfer of optimal hyperparameters across models is noted to be variable.
Reproducibility
The paper provides sufficient detail for reproduction. It defines the neuron definition (FFN hidden activations), the reference signal generation (Gaussian noise with fixed seed), the scoring equation, and the amplification hook. The models and benchmark (VoxParadox) are open-source and widely available. The hyperparameter selection process is clearly described, though the reliance on an external development set (LISTEN) for tuning $(K, g)$ is a minor constraint for fully zero-shot application. The code is not explicitly linked in the text provided, but the method is algorithmic and straightforward to implement given the open-source nature of the base models.
Limitations
The primary limitation is the computational overhead of running two forward passes through the audio encoder (real vs. noise reference) for every inference step. While the encoder is smaller than the LLM, this still doubles the encoder compute time. Second, the method's effectiveness is somewhat model-dependent; Kimi-Audio showed smaller gains and higher sensitivity to hyperparameters compared to Audio-Flamingo-3. The noise reference, while effective, is a heuristic; it assumes that neurons responding to "non-noise" are the acoustic ones, which might not capture all nuanced acoustic features equally well. Finally, the method is evaluated only on multiple-choice and free-text description tasks; it does not address potential negative impacts on semantic understanding or other modalities, though the authors claim no deterioration in semantic tasks.
Broader impact
This work opens a new direction for improving multimodal models by focusing on the encoder side, which has been largely ignored in favor of LLM-centric interventions. It demonstrates that fine-grained, neuron-level steering within encoders can be highly effective for specific perceptual tasks. This could lead to more efficient and targeted methods for enhancing multimodal capabilities without the cost of retraining. However, the ability to easily steer model behavior via activation amplification also raises questions about robustness and potential adversarial vulnerabilities, although these are not explored in depth here. The paper introduces IAAN, a training-free inference-time intervention that identifies and amplifies specific feed-forward neurons in the audio encoder of LALMs by contrasting their activations on real audio versus noise, significantly improving non-semantic acoustic perception. This paper makes a valuable contribution to the field of multimodal machine learning by shifting the focus of inference-time interventions from the language model backbone to the audio encoder. By proposing a simple yet effective contrastive scoring mechanism at the individual neuron level, IAAN addresses a critical gap in how LALMs process fine-grained acoustic information. The rigorous experimental evaluation, including comprehensive ablations against other intervention loci and granularities, strongly validates the necessity of encoder-side, neuron-level targeting. The significant performance gains across multiple state-of-the-art LALMs on a challenging benchmark (VoxParadox) demonstrate the practical utility of this approach. While the computational cost of dual encoder passes is a minor drawback, the method's label-free and training-free nature makes it a highly attractive tool for enhancing the acoustic grounding of existing models.
Author of this paper? Send a correction.