wav2tok 2.0: Scalable Audio Tokenization Maintaining Explicit Pairwise Token Alignment for Efficient Audio Retrieval
Abstract
Learning discrete speech representations that preserve similarity across variable-length utterances is central to query-by-example spoken term detection (QbE-STD). While wav2tok introduced CTC-based sequence alignment to enforce token consistency, its tightly coupled clustering and alignment training recipe limits scalability. We propose wav2tok 2.0, a scalable alignment-aware speech tokenizer built on the BEST-STD backbone. wav2tok 2.0 employs staged training, first learning discriminative, speaker-invariant representations via contrastive learning and vector quantization, and then enforcing pairwise token consistency using a CTC alignment loss and a novel DTW-aligned framewise prediction objective with adaptive weighting. Experiments show that wav2tok 2.0 consistently outperforms BEST-STD and general-purpose tokenizers on QbE-STD while remaining efficient and scalable.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
wav2tok 2.0 introduces a scalable, alignment-aware speech tokenizer that combines contrastive learning with explicit CTC and DTW-aligned framewise alignment objectives, achieving state-of-the-art performance in QbE-STD tasks while maintaining computational efficiency.
Methodology
The paper proposes wav2tok 2.0, a scalable speech tokenizer for Query-by-Example Spoken Term Detection (QbE-STD). It builds upon the BEST-STD architecture by introducing a two-stage training process. Stage I uses contrastive learning and vector quantization to learn discriminative, speaker-invariant representations. Stage II enforces pairwise token consistency using a CTC-based alignment loss and a novel DTW-aligned framewise token prediction objective with adaptive weighting. The methodology addresses the scalability issues of the original wav2tok by decoupling representation learning from alignment constraints. The introduction of the DTW-aligned framewise prediction loss is a specific technical contribution aimed at fine-grained alignment, though it relies on existing DTW and CTC mechanisms.
Results & evaluation
The authors evaluate wav2tok 2.0 on LibriSpeech and TIMIT datasets using standard QbE-STD metrics (MAP, MRR, MTWV). They compare against general-purpose tokenizers (HuBERT, WavLM, SpeechTokenizer, EnCodec), conventional STD baselines (MFCC, BNF), and prior speech-specific tokenizers (BEST-STD, wav2tok). Results indicate that wav2tok 2.0 consistently outperforms these baselines across various codebook sizes and query types (IV/OOV). The ablation studies demonstrate the contribution of both the CTC alignment and the novel framewise prediction loss. The experiments are well-structured and provide a clear comparison, although the dataset scope is limited to English speech corpora.
Reproducibility
The paper provides detailed implementation details, including encoder architecture (Mamba-based), codebook sizes, loss weights, and training epochs. A GitHub repository link is provided. The use of standard libraries for CTC and DTW suggests high reproducibility. The staged training approach is clearly defined, facilitating replication.
Limitations
The primary limitation is the reliance on English-only datasets (LibriSpeech, TIMIT), which limits the assessment of multilingual generalization. The paper acknowledges this and suggests future work on multilingual settings. Additionally, while the method is more scalable than the original wav2tok, it still requires paired utterances for Stage II training, which may be a constraint for some retrieval scenarios. The performance gain, while consistent, is marginal in some metrics compared to the strong BEST-STD baseline, suggesting diminishing returns from the added complexity.
Broader impact
This work contributes to the field of efficient audio retrieval and spoken term detection. By improving the scalability and accuracy of discrete speech tokenizers, it facilitates more robust audio indexing and search applications. The techniques for explicit pairwise alignment could be relevant to other sequence modeling tasks in speech processing. However, the impact is somewhat niche, primarily benefiting researchers and practitioners in the specific domain of QbE-STD. wav2tok 2.0 introduces a scalable, alignment-aware speech tokenizer that combines contrastive learning with explicit CTC and DTW-aligned framewise alignment objectives, achieving state-of-the-art performance in QbE-STD tasks while maintaining computational efficiency.
Author of this paper? Send a correction.