Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing
Abstract
Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
Methodology
The paper proposes a novel framework for controlling transcription style (verbatim vs. intended) in Automatic Speech Recognition (ASR) models by treating the transcription policy as an explicit latent variable. The core technical contribution is the introduction of "mode tags" (discrete decoder-prefix tokens) that condition the model on the desired output style. This is complemented by a supervised cross-attention mechanism for word-level timing, which addresses the instability of emergent alignments in disfluent speech. Additionally, the authors introduce "verbatimize," a task for retroactively annotating verbatim transcripts from intended ones. The methodology is sound, leveraging the latent capabilities of pretrained Whisper models through efficient fine-tuning (Stage 1: embeddings only; Stage 2: decoder unfrozen). The approach to decoupling style control from acoustic modeling is innovative and addresses a significant gap in current ASR systems.
Results & evaluation
The experimental evaluation is comprehensive, covering disfluency detection, intended transcript quality, and word-level timing across English and German. The results demonstrate significant improvements: German disfluency F1 jumps from 10% to 79% in a zero-shot setting with only mode tag embeddings trained. The timing experiments show that the supervised cross-attention method outperforms strong baselines like MFA and WhisperX on disfluent speech (FluencyBank), achieving lower Mean Absolute Error (MAE). The ablation studies on head selection and sharpening techniques provide valuable insights into the mechanics of attention-based alignment. The use of a newly created German DisfluencySpeech dataset adds value, although the small size (202 utterances) limits the statistical power of the German results.
Reproducibility
The paper provides detailed descriptions of the training stages, hyperparameters (learning rates, batch size), and data preprocessing steps. The code repository is linked, which enhances reproducibility. The use of standard benchmarks (TIMIT, FluencyBank, AMI) allows for comparison with existing work. However, the reliance on GPT-4o for generating intended transcripts and the creation of the German dataset by non-trained speakers introduces potential biases that are not fully quantified. The specific implementation details of the "virtual blank" pause detection and Viterbi decoding are described but might require careful implementation to replicate exactly.
Limitations
The primary limitation is the reliance on paired verbatim/intended data for training, which is scarce. The authors mitigate this by using GPT-4o to generate intended transcripts from verbatim ones, which may not perfectly capture human annotation nuances. The German evaluation set is small and recorded by non-native speakers, potentially affecting the naturalness and generalizability of the results. The method is evaluated primarily on English and German; cross-lingual transfer to more distant languages is not thoroughly tested. Furthermore, the "verbatimize" task, while innovative, depends on the accuracy of the underlying intended transcript and the model's ability to correctly insert disfluencies, which may not always align with human perception of "verbatim" speech.
Broader impact
This work has significant implications for ASR systems used in clinical, legal, and linguistic contexts where verbatim transcription is crucial. By enabling controllable verbatim ASR, it opens up new possibilities for analyzing spontaneous speech, detecting speech disorders, and improving the quality of transcripts for accessibility. The "verbatimize" tool could facilitate the creation of large-scale annotated datasets, addressing a major bottleneck in speech research. However, the potential for misuse in generating realistic but fake verbatim transcripts (deepfakes) should be considered. The improvement in timing accuracy also benefits downstream tasks like speaker diarization and video captioning. The paper presents a significant advancement in controllable ASR by explicitly parameterizing transcription policy, demonstrating that pretrained models possess latent capabilities for verbatim transcription that can be activated through targeted fine-tuning and explicit conditioning.
Author of this paper? Send a correction.