Listen, Think, Transcribe: Continuous Latent Test-Time Scaling for ASR
Abstract
End-to-end ASR models transcribe in a single pass, leaving no room for the decoder to revisit hard inputs. We propose LatentASR, a parameter-efficient method that adds continuous latent test-time scaling to a frozen ASR backbone. Two small trainable modules drive it: a Latent Adapter that iteratively refines a few latent prefix positions through bounded, stabilized updates, and a Value Head that predicts whether extra computation will help and halts the loop early. The Qwen3-ASR-0.6B backbone stays fully frozen, and we train only ~4M extra parameters. We activate this loop with a deliberately small, diverse 500-utterance training set. Under this minimal-data regime, standard adaptation methods all regress: full fine-tuning, LoRA, and prompt tuning each increase WER. LatentASR is the only tested method that reduces WER on both clean benchmarks (FLEURS -2.54% and VoxPopuli -0.47% relative). The reductions are concentrated on intrinsically hard inputs. On accented and code-switched speech (ASCEND), LatentASR achieves a 16.0% relative CER reduction. Across 30 FLEURS languages (23,049 utterances), the multilingual WER decreases uniformly across resource tiers, confirming that the adapter generalizes without overfitting. Dynamic halting preserves most of the clean-set reduction at a fraction of the compute, skipping roughly half of all utterances at the entry gate. Our results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
LatentASR introduces a parameter-efficient, continuous latent test-time scaling mechanism for frozen ASR models, demonstrating that minimal-data activation can unlock significant robustness gains on challenging speech inputs without corrupting the backbone's zero-shot capabilities.
Methodology
The paper proposes LatentASR, a method for continuous latent test-time scaling in frozen ASR backbones. The core innovation lies in the "Latent Adapter," which iteratively refines a small number of latent prefix positions using bounded, stabilized updates (L2-normalization, sigmoid gating, fixed embedding anchor). This design is technically sophisticated, addressing the critical issue of representation collapse when injecting continuous states into a frozen decoder. The inclusion of a "Value Head" for dynamic halting adds a layer of efficiency and utility prediction. The methodology is well-grounded in the literature of latent test-time scaling (e.g., Coconut, Quiet-STaR) but adapts it uniquely for the direct transcription task where no intermediate reasoning trajectory exists. The stabilization mechanisms are novel and rigorously ablated, showing that each component addresses a distinct failure mode.
Results & evaluation
The experimental evaluation is comprehensive and convincing. The authors demonstrate that under a minimal-data regime (500 utterances), standard adaptation methods (LoRA, prompt tuning) regress, whereas LatentASR improves performance. This "minimal-data activation" hypothesis is supported by ablation studies on activation set size. Results on clean benchmarks (FLEURS, VoxPopuli) show modest but consistent WER reductions. More significantly, the method shows substantial gains on challenging data: ASCEND (accented/code-switched) and noise-corrupted speech. The dynamic halting mechanism effectively allocates compute, skipping easy utterances. The multilingual evaluation confirms generalization. The use of CER for ASCEND and WER for others is appropriate. The statistical significance of the small WER gains on clean data is not explicitly tested, but the consistency across benchmarks and the large gains on hard data provide strong evidence.
Reproducibility
The paper provides detailed hyperparameters, model architecture details, and training procedures. The use of a frozen backbone (Qwen3-ASR-0.6B) and specific training data sources (Common Voice, FLEURS, etc.) enhances reproducibility. The ablation studies are well-described. However, the exact composition of the 500-utterance training set is not fully public (only described as a mixture from specific sources with a seed), which might slightly hinder exact replication, though the general recipe is clear. The code is not linked, but the description is sufficient for a competent researcher to implement.
Limitations
The primary limitation is the modest gain on clean, high-resource benchmarks (e.g., -2.54% relative on FLEURS). While statistically significant, the absolute WER reduction is small. The method's effectiveness is highly dependent on the "minimal-data" activation set; deviating from this size or composition can lead to regression. The method adds inference latency for the subset of utterances that pass the Value Head gate, although dynamic halting mitigates this. The approach is specific to ASR and may not generalize directly to other modalities without adaptation.
Broader impact
This work contributes to the field of efficient and robust ASR. By enabling test-time scaling on frozen models, it reduces the need for expensive full fine-tuning and preserves zero-shot generalization. This is particularly impactful for low-resource and challenging acoustic conditions (accents, noise, code-switching). The method promotes sustainable AI by reducing the carbon footprint associated with training large models for every downstream task. It also opens new avenues for applying latent test-time scaling to other sequence-to-sequence tasks where intermediate reasoning is not natural. LatentASR introduces a parameter-efficient, continuous latent test-time scaling mechanism for frozen ASR models, demonstrating that minimal-data activation can unlock significant robustness gains on challenging speech inputs without corrupting the backbone's zero-shot capabilities.
Author of this paper? Send a correction.