DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression
Abstract
Omnimodal large language models (OmniLLMs) jointly process audio and visual streams, but the resulting long multimodal token sequences make inference prohibitively expensive. Existing compression methods typically rely on fixed window partitioning and attention-based pruning, which overlook the piecewise semantic structure of audio-visual signals and become fragile under aggressive token reduction. We propose Dynamic Audio-driven Semantic cHunking (DASH), a training-free framework that aligns token compression with semantic structure. DASH treats audio embeddings as a semantic anchor and detects boundary candidates via cosine-similarity discontinuities, inducing dynamic, variable-length segments that approximate the underlying piecewise-coherent organization of the sequence. These boundaries are projected onto video tokens to establish explicit cross-modal segmentation. Within each segment, token retention is determined by a tri-signal importance estimator that fuses structural boundary cues, representational distinctiveness, and attention-based salience, mitigating the sparsity bias of attention-only selection. This structure-aware allocation preserves transition-critical tokens while reducing redundant regions. Extensive experiments on AVUT, VideoMME, and WorldSense demonstrate that DASH maintains superior accuracy while achieving higher compression ratios compared to prior methods. Code is available at: https://github.com/laychou666/DASH.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of DASH, a training-free framework for dynamic audio-driven semantic chunking that enhances token compression in omnimodal large language models. This innovative approach effectively aligns compression with the inherent semantic structure of audio-visual signals, improving both accuracy and efficiency in processing multimodal data.
Methodology
The proposed DASH framework introduces a novel approach to token compression in omnimodal large language models by leveraging audio embeddings as semantic anchors for dynamic segmentation. This method addresses the limitations of existing compression techniques that rely on fixed window partitioning and attention-based pruning, which often fail to preserve the semantic structure of audio-visual signals. The use of cosine-similarity discontinuities for boundary detection and the tri-signal importance estimator for token retention are innovative contributions that enhance the model's ability to maintain critical information during aggressive compression. The training-free nature of DASH further simplifies its integration into existing systems, making it a practical solution for real-world applications.
Results & evaluation
The experiments conducted on multiple benchmarks (AVUT, VideoMME, and WorldSense) demonstrate the effectiveness of DASH in maintaining accuracy while achieving higher compression ratios compared to prior methods. The results indicate that DASH not only preserves critical information but also improves inference efficiency, which is crucial for deploying omnimodal models in resource-constrained environments. The comprehensive evaluation across different datasets strengthens the validity of the findings and showcases the robustness of the proposed method.
Reproducibility
The paper provides sufficient implementation details, including hyperparameters and the experimental setup, which enhances the reproducibility of the results. The authors have made the code available on GitHub, allowing other researchers to replicate their findings and further explore the DASH framework.
Limitations
One limitation of the proposed method is its reliance on audio embeddings as the primary source for boundary detection, which may not always capture the full complexity of the multimodal signals. Additionally, while the framework is training-free, its performance may vary depending on the quality of the audio input and the specific characteristics of the video content. Future work could explore the integration of more sophisticated learning-based approaches for boundary detection and token selection.
Broader impact
The DASH framework has significant implications for the field of multimodal machine learning, particularly in applications that require efficient processing of audio-visual data, such as video understanding, content creation, and interactive systems. By improving the efficiency of omnimodal models, DASH can facilitate advancements in real-time applications, enhance user experiences, and contribute to the development of more intelligent systems capable of understanding complex multimedia content. The main contribution of this paper is the introduction of DASH, a training-free framework for dynamic audio-driven semantic chunking that enhances token compression in omnimodal large language models. This innovative approach effectively aligns compression with the inherent semantic structure of audio-visual signals, improving both accuracy and efficiency in processing multimodal data.
Author of this paper? Send a correction.