Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper
Abstract
Whisper exposes speech through a fixed 1500-token encoder interface, now a default representation for ASR decoders and Whisper-based speech language models (SpeechLMs), yet its redundancy remains largely unexamined. We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer. Across five Whisper scales, k=2 preserves baseline WER at both positions, with CKA attributing this stability to acoustic overlap at the stem and attention-induced redistribution at the encoder output. Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones. The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%. Requiring no training or auxiliary computation, stride-k subsampling exploits Whisper's preprocessing redundancy, indicating that its audio-token interface carries more capacity than downstream tasks require.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
Methodology
The paper proposes "stride-k subsampling," a deterministic, train-free indexing operation to reduce the number of audio tokens in Whisper-based models. The method exploits the temporal redundancy inherent in Whisper's preprocessing pipeline (overlapping STFT windows and convolutional stem receptive fields). By retaining every k-th token either after the convolutional stem (input-side) or after the encoder transformer (output-side), the authors reduce sequence length without retraining. The methodology is grounded in a theoretical analysis of temporal overlap, showing that for k=2, the retained tokens still cover the full input span due to receptive field overlap. The authors further employ Centered Kernel Alignment (CKA) to diagnose why output-side subsampling is more stable than input-side, attributing it to attention-induced redistribution of redundancy in the encoder. The approach is simple, parameter-free, and easily integrable into existing pipelines.
Results & evaluation
The experimental evaluation is comprehensive, covering five Whisper scales (tiny to large-v3) and three ASR benchmarks (LibriTTS, ESD, Common Voice). The results demonstrate that k=2 preserves baseline WER on clean speech with minimal cost, while harder benchmarks show larger degradations. The paper extends the evaluation to three Whisper-based SpeechLMs (Audio Flamingo 3, Qwen2-Audio, LLaMA-Omni 2), showing modest accuracy drops on MMSU and MMAU benchmarks. The efficiency gains are significant, with 52-58% reduction in GFLOPs and 19-27% reduction in end-to-end latency. The inclusion of a noise robustness analysis (SNR sweep) and a comparison with other pruning methods (SpeechPrune) strengthens the evaluation. The "hurt/help" analysis provides insight into which types of tasks are most affected by temporal subsampling.
Reproducibility
The paper provides detailed implementation details, including the specific subsampling positions, FLOPs calculation formulas, and dataset preparation steps. The use of standard open-source models and benchmarks (HuggingFace checkpoints, LibriTTS, Common Voice) enhances reproducibility. The authors mention using a fixed random seed and provide code snippets for the subsampling operation. However, no explicit GitHub repository link is provided in the text, which slightly limits immediate reproducibility, though the method is simple enough to implement from the description.
Limitations
The primary limitation is that the method is specific to encoders with overlapping windowed frontends (like Whisper's log-mel). It does not generalize to raw-waveform encoders like wav2vec 2.0 or HuBERT, which collapse under subsampling. The method requires access to the encoder's internal computation, making it inapplicable to closed-API models. Additionally, the performance degradation on noisy or difficult benchmarks (Common Voice) is significant, limiting its utility in low-resource or noisy environments. The paper acknowledges that the redundancy exploited is partly shaped by training, and future work could explore training models to tolerate subsampling.
Broader impact
The paper contributes to the efficiency of speech recognition and speech language models, which are increasingly deployed in real-time applications. By reducing inference cost without retraining, the method offers a practical solution for deploying Whisper-based systems on resource-constrained devices. The insights into temporal redundancy in speech encoders may inform future architectural designs. The findings on the asymmetry between input-side and output-side subsampling provide valuable guidance for model optimization. The paper proposes a simple, train-free stride-k subsampling method to reduce audio tokens in Whisper, achieving significant efficiency gains with minimal performance loss on clean speech. It provides a thorough analysis of the underlying redundancy mechanisms using CKA and extensive experiments across multiple model scales and benchmarks, establishing a practical baseline for efficient speech processing.
Author of this paper? Send a correction.