AS$^2$D: Accelerating On-Demand Audio Understanding on Mobile Devices
Abstract
Speculative decoding accelerates autoregressive generation by using a smaller drafter to propose tokens for batched verification by a larger target. However, conventional speculative decoding couples drafting to the target's evolving verified prefix, serializing drafting and verification. We ask whether this dependency is necessary for source-conditioned generation. Our key observation is that, for audio language models, the input audio and user request can provide useful speculative candidates without following the target's evolving text prefix. We propose AS$^2$D (Audio Speculative Speculative Decoding), which enables target-decoupled drafting: an audio-conditioned drafter follows its own generation history while the target independently verifies and corrects ready candidates. Without usable candidates, the target advances alone. Thus, target feedback determines which candidates are committed but no longer determines when the drafter can make progress, enabling drafting and verification to proceed concurrently while retaining target-side verification and correction. We implement AS$^2$D in MNN for Android and evaluate two target models across four phones, seven datasets, and three tasks covering 12.2 hours of audio. Across four phones, AS$^2$D improves pooled ASR throughput by 42-76% over target-only decoding, while only 5.7% of evaluation windows are slower than target-only, compared with 58.1-63.0% for speculative baselines. For ASR, AS$^2$D reaches 97.33-98.20% of a hindsight per-window oracle's pooled throughput over the evaluated drafter/budget catalog. Native on-demand execution with a 7B target achieves up to 78% higher throughput than target-only. These results show that source-conditioned audio generation can relax the conventional dependence of speculative drafting on the target's evolving output prefix, exposing substantial parallelism for efficient inference.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces AS$^2$D, a target-decoupled speculative decoding framework for audio language models that enables concurrent drafting and verification on mobile devices, significantly improving throughput and latency consistency compared to conventional serial speculative methods.
Methodology
The paper proposes AS$^2$D (Audio Speculative Speculative Decoding), a method that decouples the drafting process from the target model's evolving verified prefix in speculative decoding. The core insight is that for source-conditioned tasks like ASR, the input audio and instruction provide sufficient context for a smaller drafter to generate candidates independently of the target's current output state. This allows drafting and verification to proceed concurrently on heterogeneous mobile processors, rather than serially. The methodology involves an audio-conditioned drafter that maintains its own generation history, publishing candidates to a buffer. The target model verifies these candidates against its own trajectory, using local token matching to align and reuse rejected candidates where possible. The approach is rigorously formulated with a problem definition that accounts for memory constraints and latency quantiles, and it includes a device-profiled configuration step to select optimal budget caps based on hardware characteristics.
Results & evaluation
The evaluation is extensive and well-suited to the mobile inference context. The authors test two target models (Qwen3-ASR-1.7B and Qwen2.5-Omni-7B) across four different Android phones with varying hardware capabilities (Snapdragon 750G to 8 Gen 3). They evaluate on seven datasets covering ASR, translation, and spoken QA, totaling 12.2 hours of audio. The results show significant throughput improvements (42-76%) over target-only decoding, with a crucial finding that AS$^2$D avoids the latency spikes common in standard speculative decoding (only 5.7% of windows are slower vs. 58-63% for baselines). The comparison against SpecASR and standard SD is fair and highlights the advantage of decoupled drafting on resource-constrained devices.
Reproducibility
The paper provides detailed implementation notes, specifying the use of the MNN framework for Android and the specific model versions used. The description of the buffer management, alignment rules, and profiling methodology is sufficient for a skilled engineer to reproduce the system. However, the specific code for the custom MNN modifications and the profiling scripts are not explicitly linked in the provided text, which slightly limits immediate reproducibility without access to the full repository.
Limitations
The primary limitation is the dependency on the specific hardware characteristics of the tested phones; the gains may vary on other architectures. The method relies on the assumption that the audio source provides enough signal for the drafter to be useful, which may not hold for all audio tasks or extremely noisy conditions. Additionally, the approach is tailored to autoregressive generation; it does not apply to non-autoregressive models. The paper focuses on throughput and latency but does not extensively discuss the energy consumption trade-offs of running two models concurrently, which is critical for mobile devices.
Broader impact
This work has significant implications for the deployment of large audio language models on edge devices. By enabling concurrent drafting and verification, it makes high-quality, real-time audio understanding feasible on smartphones without cloud dependency. This could enable new applications in privacy-preserving voice assistants, wearable memory aids, and offline translation tools. The technique of decoupling drafting from the target prefix could potentially be adapted to other source-conditioned generation tasks, such as video captioning or code completion with context. The paper introduces AS$^2$D, a target-decoupled speculative decoding framework for audio language models that enables concurrent drafting and verification on mobile devices, significantly improving throughput and latency consistency compared to conventional serial speculative methods.
Author of this paper? Send a correction.