Beyond Cross-Reconstruction: Probing-Based Disentanglement Evaluation for Acoustic Teleportation Codecs
Abstract
Some neural audio codecs disentangle speech into latent subspaces encoding content, speaker identity, and acoustics, enabling acoustic teleportation and voice conversion. Existing evaluations rely on cross-reconstruction quality, which cannot reliably detect leakage across partitions. We extend a probing based framework to assess disentanglement by regressing room-acoustic parameters (reverberation time, clarity, and direct-to-reverberant ratio) and classifying speaker identity, using the gap between intended and unintended partitions as the disentanglement measure. Applied to an acoustic teleportation codec, we find speaker identity is largely confined to its partition, while acoustics leak into the speech embeddings due to the training objective. Acoustic embeddings blindly estimate room parameters within 0.02 s of supervised baselines, indicating physically meaningful structure emerges without explicit supervision.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces a robust probing-based evaluation framework for neural audio codecs, revealing critical asymmetries in disentanglement that traditional metrics miss, thereby guiding the design of more effective audio representation learning models.
Methodology
The paper proposes a probing-based framework to evaluate disentanglement in Neural Audio Codecs (NACs), specifically targeting Acoustic Teleportation (AT) codecs. The core methodological contribution is the adaptation of the "informativeness" principle from the DCI metric to partition-level embeddings, using a gap between intended and unintended partition performance as the disentanglement measure. The authors employ lightweight MLP probes to regress continuous room-acoustic parameters ($T_{60}$, $C_{50}$, DRR) and classify speaker identity. This approach is technically sound and addresses a specific gap in evaluation methodologies where cross-reconstruction metrics fail to detect information leakage. The use of regression for physical parameters is a novel extension of existing classification-based probing in speech.
Results & evaluation
The experimental evaluation is rigorous and well-controlled. The authors test multiple model configurations varying training tasks, quantization levels, and temporal downsampling factors. They use established datasets (DNS5, GWA-small) and compare probe performance against supervised baselines (CRNN-MB, spectrogram CNN). The results clearly demonstrate that speaker identity is well-disentangled, while acoustic information leaks into the speech partition. The finding that acoustic embeddings can estimate room parameters competitively without explicit supervision is a strong empirical result. The statistical significance testing (Steiger test, z-test) adds robustness to the claims.
Reproducibility
The paper provides sufficient detail for reproduction, including probe architecture (MLP dimensions, layers), training hyperparameters (AdamW, learning rate, early stopping), and dataset preprocessing steps (RIR truncation, normalization). The use of fixed pre-trained encoders as feature extractors simplifies the experimental setup. However, the specific version of the AT codec and the exact code for the probes are not linked, which might require contacting authors for full reproducibility.
Limitations
The primary limitation is that the evaluation is restricted to a single codec architecture (EnCodec-based AT) and a specific set of room parameters. The probes are simple MLPs, which the authors acknowledge provides a lower bound on leakage; more complex probes might reveal higher leakage. The study focuses on time-invariant factors; dynamic aspects like linguistic content are not probed in depth, though mentioned as future work. The generalizability to other NAC architectures (e.g., those using different quantization schemes or hierarchical structures) is not empirically validated.
Broader impact
This work has significant implications for the development of neural audio codecs, particularly for applications like voice conversion, acoustic teleportation, and dereverberation. By providing a reliable method to detect information leakage, it enables researchers to design better training objectives (e.g., adversarial decorrelation) to achieve true disentanglement. This can lead to higher quality and more controllable audio generation systems. The finding that physically meaningful structures emerge without supervision also contributes to the broader understanding of latent space geometry in self-supervised audio learning. This paper introduces a robust probing-based evaluation framework for neural audio codecs, revealing critical asymmetries in disentanglement that traditional metrics miss, thereby guiding the design of more effective audio representation learning models.
Author of this paper? Send a correction.