DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection
Abstract
Variable frame rate (VFR) coding has recently emerged in neural speech codecs, allocating fewer frames to redundant regions and more frames to rapidly changing speech. VFR must transmit side information about retained time steps, but prior gains are either not rigorously addressed or often minor once these overhead bits are included in total bitrate. We present Dynamic Token Masking (DTM)-Codec, a neural speech codec that demonstrates clear gains over fixed-frame-rate baselines under a strict matched-total-bitrate protocol. DTM keeps selected encoder tokens, fills masked positions with a learned embedding, and transmits a binary keep-mask for position-aware decoding. We further introduce Path Length Equalization (PLE), a linear-time boundary selector for VFR coding that yields well-spread adaptive segments with negligible overhead. Across operating points, DTM-Codec broadly improves reconstruction quality and intelligibility over fixed-frame-rate baselines.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
DTM-Codec introduces a novel dynamic token masking mechanism and a linear-time boundary selector for variable frame rate speech coding, demonstrating significant reconstruction quality improvements over fixed-rate baselines under strict matched-total-bitrate evaluations. The paper makes a valuable contribution to the field of neural audio codecs by addressing the critical issue of fair bitrate comparison and providing a practical, efficient solution for adaptive temporal resolution in speech tokenization.
Methodology
The paper proposes DTM-Codec, a neural speech codec that integrates Variable Frame Rate (VFR) coding via Dynamic Token Masking (DTM) and a linear-time boundary selector called Path Length Equalization (PLE). The core methodological contribution is the combination of a masking-based token retention strategy (preserving original feature vectors rather than pooling/merging) with a computationally efficient, content-adaptive boundary selection algorithm. The approach addresses a specific gap in the literature: the lack of rigorous, matched-total-bitrate comparisons that account for side-information overhead in VFR codecs. The use of a learnable ` ` embedding for position-aware decoding is a practical and effective design choice that simplifies the decoder architecture compared to complex upsampling or interpolation methods used in prior works.
Results & evaluation
The experimental evaluation is a strong point of this paper. The authors conduct a comprehensive set of experiments on LibriSpeech and MLS, comparing DTM-Codec against several state-of-the-art baselines (FlexiCodec, VARSTok, BigCodec, etc.) under strict matched-total-bitrate protocols. They include both objective metrics (UTMOS, PESQ, STOI, WER) and subjective listening tests (MUSHRA). The results consistently show that DTM-Codec outperforms fixed-frame-rate baselines and competitive VFR baselines, particularly at lower bitrates. The ablation studies on the boundary selector (PLE vs. DP vs. Clustering) provide valuable insights into the trade-off between computational complexity and reconstruction quality. The inclusion of semantic evaluation (ARCH benchmark) adds depth, although the results there are mixed, highlighting that VFR benefits reconstruction more than global semantic retention.
Reproducibility
The paper provides sufficient implementation details, including model architecture (TAAE backbone, STFT/iSTFT front-end/back-end), training hyperparameters (AdamW, batch size, steps), and the specific VQ codebook size. The GitHub repository link is provided. The strict bitrate accounting methodology is clearly defined, which aids in reproducing the fair comparisons. The linear-time PLE algorithm is simple to implement.
Limitations
The primary limitation is that the model is evaluated primarily on English speech (LibriSpeech) and a small set of non-English utterances (MLS). Generalization to other languages or highly diverse acoustic environments is not thoroughly demonstrated. Additionally, while PLE is efficient, it is a heuristic; the paper acknowledges that Dynamic Programming (DP) yields slightly better quality but is slower. The semantic evaluation results suggest that for tasks requiring global context (like emotion classification), VFR might not always be superior to FFR with a larger codebook, which is an important nuance for downstream applications.
Broader impact
This work contributes to the efficient transmission and processing of speech data, which is crucial for low-bandwidth communication, streaming services, and efficient tokenization for Speech Language Models (SLMs). By demonstrating that VFR can provide clear gains even with side-information overhead, it encourages further research into adaptive-rate codecs for AI-driven audio applications. DTM-Codec introduces a novel dynamic token masking mechanism and a linear-time boundary selector for variable frame rate speech coding, demonstrating significant reconstruction quality improvements over fixed-rate baselines under strict matched-total-bitrate evaluations. The paper makes a valuable contribution to the field of neural audio codecs by addressing the critical issue of fair bitrate comparison and providing a practical, efficient solution for adaptive temporal resolution in speech tokenization.
Author of this paper? Send a correction.