VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models
Abstract
Spoken conversational systems must recover information from prior interactions (i.e., memory), yet relevant information in speech extends beyond what was said to who said it, how it was spoken, and what was audible, information that exists only in the audio signal and cannot be recovered from a transcript. Beyond what to remember, memory also demands diverse operations: retrieving a single fact, integrating evidence across turns, tracking an evolving state. Real interactions further unfold across sessions, meaning information accumulates across distinct episodes rather than a single continuous recording. Existing benchmarks fall short on all three dimensions: they focus primarily on lexical content, adopt limited and ad hoc memory operations, and treat memory as a single-session problem. We argue that principled memory evaluation requires jointly characterizing the acoustic evidence to be retained and the operations applied to it, and introduce a taxonomy along these two axes. Building on this taxonomy, we present VoxMem: 3,196 evaluation instances over 34,743 spoken sessions (177 hours) crossing four acoustic evidence types (speech semantics, speaker identity, paralinguistic cues, environmental sound) with four memory operations (information extraction, multi-session reasoning, temporal tracking, and answer refusal), grounded in multi-session histories and stratified across context budgets from 8K to 64K tokens. Evaluating 15 LALMs, no model exceeds 40% at 32K. Models retain what was said far better than who said it, how, or what was audible, a gap that widens for complex operations, grows with history length, and manifests as qualitatively distinct failure modes across evidence types. VoxMem aims to provide a foundation to measure and drive progress on the full scope of spoken conversational memory.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces VoxMem, a comprehensive benchmark for evaluating multi-session spoken conversational memory across acoustic evidence types and memory operations. It provides a principled taxonomy and rigorous construction pipeline, revealing significant gaps in current LALMs' ability to retain non-lexical acoustic information over long contexts.
Methodology
The paper proposes a principled two-dimensional taxonomy for spoken conversational memory, crossing four acoustic evidence types (speech semantics, speaker identity, paralinguistic cues, environmental sound) with four memory operations (information extraction, multi-session reasoning, temporal evolution tracking, and answer refusal). This is a significant methodological advance over prior benchmarks that treated memory as a single-session, lexical-only problem. The construction pipeline is rigorous, utilizing a three-stage process (planning, dialogue writing, speech synthesis) with specific controls to ensure "acoustic necessity" (i.e., the answer cannot be derived from the transcript alone). The use of Higgs-TTS-3 and VCTK voices, along with mixed environmental sounds from ESC-50, provides a controlled yet realistic audio environment. The inclusion of "haystack" and "filler" sessions to create multi-session histories of varying lengths (8K-64K tokens) is a strong design choice that isolates the effect of context length from question difficulty.
Results & evaluation
The evaluation covers 15 Large Audio Language Models (LALMs), including both open-weight and proprietary models. The results reveal a clear hierarchy of difficulty: models perform significantly better on speech semantics than on audio-native cues (speaker, paralinguistic, environmental). The finding that no model exceeds 40% accuracy at the 32K token budget highlights a critical gap in current LALM capabilities. The error analysis is particularly valuable, distinguishing between binding failures (wrong speaker) and retention failures (lost cue), which provides actionable insights for future model development. The controlled scaling analysis shows that performance degrades as history length increases, with non-lexical information degrading faster than lexical information.
Reproducibility
The paper provides detailed descriptions of the construction pipeline, quality control checks, and evaluation protocol. The use of specific TTS systems, voice datasets, and sound event datasets enhances reproducibility. The release of the benchmark instances, question templates, and judge prompts (mentioned in the text) further supports reproducibility. However, the reliance on proprietary LLMs (Gemini-3.7-Flash, GPT-5.6-Luna) for generation and evaluation introduces a dependency on external services, which may limit full reproducibility for all researchers.
Limitations
The benchmark relies on synthetic speech generated by TTS systems, which may not fully capture the variability and noise of real-world human speech. The "acoustic necessity" check, while rigorous, uses a specific LLM (Gemini-3.7-Flash) for validation, which could introduce bias if that model has specific strengths or weaknesses in audio understanding. The paper does not extensively discuss the computational cost of evaluating models on 64K token audio histories, which may be prohibitive for some researchers. Additionally, the focus on English speech (implied by the datasets used) limits the generalizability of the findings to other languages.
Broader impact
VoxMem addresses a fundamental challenge in the development of long-term conversational AI systems. By providing a standardized benchmark for multi-session, audio-native memory, it enables fair comparison of LALMs and guides future research toward improving non-lexical acoustic memory. The taxonomy and benchmark are likely to become a standard reference in the field, driving progress in speaker identification, paralinguistic understanding, and environmental sound recognition within conversational contexts. The findings on the distinct failure modes for different evidence types will inform the design of specialized memory modules in LALMs. The paper introduces VoxMem, a comprehensive benchmark for evaluating multi-session spoken conversational memory across acoustic evidence types and memory operations. It provides a principled taxonomy and rigorous construction pipeline, revealing significant gaps in current LALMs' ability to retain non-lexical acoustic information over long contexts.
Author of this paper? Send a correction.