Beyond Textual Chain-of-Thought: JEPA-Conditioned Latent Reasoning for Large Audio Language Models
Abstract
Explicit textual Chain-of-Thought (CoT) has improved the reasoning ability of large audio language models (LALMs). However, textual CoTs are often constructed from text captions of audio and provide limited access to the acoustic evidence, which can introduce problems like hallucination. To address this modality-gap issue, we introduce JELAR, a Joint-Embedding Predictive Architecture (JEPA)-based latent reasoning framework that conditions latent reasoning supervision on acoustic representations learned from raw waveforms. During training, a frozen WavJEPA model provides representations learned directly from raw waveforms. A non-causal expert first constructs answer-aware queries, which cross-attend to WavJEPA embeddings to produce latent reasoning targets. The LALM is trained to predict these targets before generating its response. Experimental results show that JELAR improves the Audio-Reasoner baseline by 2.70 and 9.10 absolute percentage points on MMAU-mini and MMAR, respectively, demonstrating the effectiveness of JEPA-conditioned latent reasoning as an alternative to explicit textual CoT supervision.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces JELAR, a JEPA-conditioned latent reasoning framework that improves audio reasoning accuracy by grounding latent supervision in raw waveform representations, effectively addressing the modality gap inherent in textual Chain-of-Thought approaches.
Methodology
The paper proposes JELAR, a framework that replaces explicit textual Chain-of-Thought (CoT) with latent reasoning conditioned on acoustic representations. The core methodological contribution is the use of a frozen WavJEPA encoder to generate acoustic embeddings, which are then processed by a non-causal expert (BERT-based) to create answer-aware latent targets. These targets serve as supervision for the LALM during training. The approach is logically sound, addressing the "modality gap" where text-based CoT may hallucinate or miss acoustic details. However, the reliance on a separate, non-causal expert for target generation introduces complexity and a potential train-test mismatch (teacher forcing vs. autoregressive inference), which is a known challenge in latent reasoning. The integration of JEPA is a creative application of predictive coding principles to audio reasoning.
Results & evaluation
The experiments are conducted on MMAU-mini and MMAR benchmarks, comparing JELAR against the Audio-Reasoner baseline. The reported improvements are significant, particularly on MMAR (+9.10 points), suggesting the method is effective for multi-step reasoning. Ablation studies confirm the contribution of acoustic conditioning and latent reasoning length. However, the evaluation is limited to these two benchmarks, and comparisons with other state-of-the-art audio LLMs (like GPT-4o or Gemini) are not directly made in the main results table for the proposed method, only for context. The lack of qualitative analysis of the latent reasoning process is a minor weakness.
Reproducibility
The paper provides sufficient detail on the architecture and training procedure, including the use of frozen WavJEPA and BERT-base. Hyperparameters like K=20 are specified. However, specific details on the training data composition, optimization hyperparameters (learning rates, batch sizes), and the exact implementation of the cross-attention module are not fully detailed in the provided text. The code repository is not linked, which hinders immediate reproducibility.
Limitations
The method requires a separate non-causal expert during training, which increases computational overhead and complexity. The latent reasoning space is not interpretable, making it difficult to debug errors. The performance gains, while positive, are based on a specific baseline (Audio-Reasoner) and may not generalize to other LALM architectures without further adaptation. The paper does not discuss the computational cost of inference compared to standard CoT or direct answering.
Broader impact
This work contributes to the growing field of latent reasoning in multimodal models, offering an alternative to text-centric CoT. It highlights the utility of JEPA-style representations for audio tasks, potentially inspiring similar approaches in other modalities. The focus on reducing hallucination by grounding reasoning in acoustic evidence is a valuable direction for improving the reliability of audio LLMs. The paper introduces JELAR, a JEPA-conditioned latent reasoning framework that improves audio reasoning accuracy by grounding latent supervision in raw waveform representations, effectively addressing the modality gap inherent in textual Chain-of-Thought approaches.
Author of this paper? Send a correction.