Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation
Abstract
While recent Vision-Language-Action (VLA) models have begun to incorporate audio, they typically treat sound as static pre-execution prompts or focus exclusively on human speech. This leaves a significant gap in real-time, sound-centric manipulation where fleeting environmental acoustics provide critical state verification during task execution. Consequently, key sounds are easily missed due to low-frequency updates or system latency. This problem is exacerbated by action chunking with open-loop execution, which creates a Blind Execution Interval where acoustic events are lost between discrete audio observation windows. Recognizing the necessity of continuous auditory awareness, we formalize Vision-Sound-Language-Action (VSLA) as a continuous control paradigm conditioned on vision, streaming audio, language, and proprioception under delayed decision loops. As an instantiation, we introduce HEAR, a VSLA framework integrating four components: (i) a streaming Historizer to maintain a compact, causal audio context across execution gaps; (ii) an Envisioner adapted from omni foundation models to reason over multi-sensory inputs; (iii) an Advancer, formulated as an audio world model, to learn temporal dynamics by predicting near-future audio codes; and (iv) a flow-matching Realizer policy to generate smooth action chunks. To address the scarcity of pretraining data and evaluations for VSLA, we construct OpenX-Sound for pretraining, alongside HEAR-Bench, the first sound-centric manipulation benchmark with strict causal timing rules. Our results suggest that robust sound-centric manipulation necessitates causal persistence and explicit temporal learning. This framework provides a practical step toward multi-sensory foundation models for embodied agents, enabling robots to perceive and interact with dynamic environments. Code and videos are available at https://hear.irmv.top.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of the HEAR framework, which integrates continuous auditory awareness into robotic manipulation, addressing critical limitations of existing VLA models and paving the way for more responsive and intelligent robotic systems. The comprehensive methodology and robust experimental validation position this work as a significant advancement in the field of multi-sensory robotics.
Methodology
The paper introduces a novel framework, HEAR, which addresses significant limitations in existing Vision-Language-Action (VLA) models by incorporating continuous auditory awareness into robotic manipulation. The methodology is well-structured, comprising four key components: the Historizer, Envisioner, Advancer, and Realizer, each designed to tackle specific challenges associated with audio processing in real-time environments. The use of a streaming causal audio memory and the introduction of a predictive learning signal for audio dynamics are particularly innovative, allowing the framework to maintain temporal context and improve reactivity to transient acoustic cues.
Results & evaluation
The authors present extensive evaluations on the HEAR-Bench, a newly constructed benchmark specifically designed for sound-centric manipulation tasks. The experiments demonstrate that HEAR significantly outperforms traditional methods, particularly in scenarios involving brief non-speech audio triggers. The results are quantitatively robust, showcasing an average success rate of 81%, which indicates the effectiveness of the proposed framework in real-world applications.
Reproducibility
The paper provides a clear description of the architecture and methodologies employed, along with a publicly accessible demo URL. However, the lack of a direct link to the code repository limits the reproducibility of the results. The authors should consider making their implementation available to facilitate further research and validation.
Limitations
While the framework shows promise, it may face challenges in scaling to more complex environments with diverse audio inputs. The reliance on synthetic audio data for training may also raise concerns about the generalizability of the model to real-world scenarios. Additionally, the system's performance in noisy environments or with overlapping sounds has not been thoroughly evaluated.
Broader impact
The HEAR framework has the potential to significantly advance the field of robotics by enabling more sophisticated interactions with dynamic environments. Its implications extend beyond robotics to areas such as human-robot interaction, assistive technologies, and autonomous systems, where real-time auditory feedback is crucial for effective operation. The main contribution of this paper is the introduction of the HEAR framework, which integrates continuous auditory awareness into robotic manipulation, addressing critical limitations of existing VLA models and paving the way for more responsive and intelligent robotic systems. The comprehensive methodology and robust experimental validation position this work as a significant advancement in the field of multi-sensory robotics.
Author of this paper? Send a correction.