Listen, Look, Drive: Coupling Audio Instructions for User-aware VLA-based Autonomous Driving
Abstract
Vision Language Action (VLA) models promise an open-vocabulary interface that can translate perceptual ambiguity into semantically grounded driving decisions, yet they still treat language as a static prior fixed at inference time. As a result, the model must infer continuously shifting objectives from pixels alone, yielding delayed or overly conservative maneuvers. We argue that effective VLAs for autonomous driving need an online channel in which users can influence driving with specific intentions. To this end, we present EchoVLA, a user-aware VLA that couples camera streams with in situ audio instructions. We augment the nuScenes dataset with temporally aligned, intent-specific speech commands generated by converting ego-motion descriptions into synthetic audios. Further, we compose emotional speech-trajectory pairs into a multimodal Chain-of-Thought (CoT) for fine-tuning a Multimodal Large Model (MLM) based on Qwen2.5-Omni. Specifically, we synthesize the audio-augmented dataset with different emotion types paired with corresponding driving behaviors, leveraging the emotional cues embedded in tone, pitch, and speech tempo to reflect varying user states, such as urgent or hesitant intentions, thus enabling our EchoVLA to interpret not only the semantic content but also the emotional context of audio commands for more nuanced and emotionally adaptive driving behavior. In open-loop benchmarks, our approach reduces the average L2 error by $59.4\%$ and the collision rate by $74.4\%$ compared to the baseline of vision-only perception. More experiments on nuScenes dataset validate that EchoVLA not only steers the trajectory through audio instructions, but also modulates driving behavior in response to the emotions detected in the user's speech.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of EchoVLA, a user-aware VLA model that couples audio instructions with visual inputs to improve autonomous driving behavior. This work represents a significant step towards creating more interactive and emotionally intelligent autonomous systems, though it requires further development in reproducibility and real-world applicability.
Methodology
The proposed methodology introduces EchoVLA, which integrates audio instructions with visual inputs to enhance user-aware driving behavior. The use of emotional speech-trajectory pairs and the synthesis of an audio-augmented dataset are innovative aspects that address the limitations of traditional VLA models. However, the methodology could benefit from a more detailed explanation of the data generation process and the specific architecture of the Multimodal Large Model (MLM) used.
Results & evaluation
The experiments conducted on the nuScenes dataset demonstrate significant improvements in driving performance metrics, such as a 59.4% reduction in average L2 error and a 74.4% decrease in collision rates compared to a vision-only baseline. While these results are promising, additional comparisons with other state-of-the-art methods could strengthen the findings. The benchmarks should also include a qualitative assessment of driving behavior to complement the quantitative metrics.
Reproducibility
The paper lacks detailed implementation specifics and code availability, which are crucial for reproducibility. Without access to the dataset augmentation process and the model training details, it would be challenging for other researchers to replicate the results or build upon this work.
Limitations
One limitation is the reliance on synthetic audio commands, which may not fully capture the variability and complexity of real-world user interactions. Additionally, the emotional context interpretation may be limited by the quality and diversity of the training data. The paper does not address potential biases in the emotional speech data or how they might affect driving behavior.
Broader impact
The integration of emotional intelligence into autonomous driving systems has the potential to enhance user experience and safety. By allowing vehicles to respond to user intentions and emotional states, this research could pave the way for more adaptive and user-friendly autonomous systems. However, ethical considerations regarding user privacy and the implications of emotional manipulation in driving contexts must be addressed. The main contribution of this paper is the introduction of EchoVLA, a user-aware VLA model that couples audio instructions with visual inputs to improve autonomous driving behavior. This work represents a significant step towards creating more interactive and emotionally intelligent autonomous systems, though it requires further development in reproducibility and real-world applicability.
Author of this paper? Send a correction.