UniSAE: Unified Speech Attribute Editing on Speaker, Emotion and Low-Level Content via Discrete Phonetic Posteriorgram Modelling
Abstract
Speech editing aims to modify specific portions of an utterance while preserving the remaining speech. Existing approaches primarily focus on word-level content modification and typically treat content, speaker, and emotion editing as separate tasks, limiting both editing granularity and flexibility. We propose UniSAE, a unified speech attribute editing framework which supports composable speaker, emotion and content editing from sub-phoneme to word level within a single architecture. UniSAE introduces a Discrete Phonetic PosteriorGram (DPPG) representation that factorizes speech content into discrete tokens encoding phoneme identity, pronunciation variants, and duration, enabling direct phoneme- and sub-phoneme-level editing. For higher-level modifications, an autoregressive content transformer predicts edited DPPG sequences for word-level content editing. The edited sequences are rendered into speech by a diffusion-based acoustic decoder, conditioned on disentangled speaker and emotion representations. Experimental results demonstrate that the proposed unified framework supports precise speaker and emotion control, content editing at multiple granularities, and joint modification of all three attributes within a single framework.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
[UniSAE introduces a unified speech attribute editing framework leveraging Discrete Phonetic PosteriorGrams to enable composable, fine-grained control over content, speaker, and emotion, addressing the limitations of existing word-level and single-attribute editing systems.]
Methodology
The paper proposes UniSAE, a unified framework for speech attribute editing that handles content, speaker, and emotion modifications. The core methodological innovation is the Discrete Phonetic PosteriorGram (DPPG), which factorizes speech content into phoneme identity, pronunciation variants, and duration. This allows for sub-phoneme level editing, a significant step up from the word-level editing dominant in recent works like VoiceCraft. The architecture employs a two-stage approach: a Content Transformer predicts edited DPPG sequences, and a diffusion-based acoustic decoder renders these into speech conditioned on disentangled speaker and emotion embeddings. The use of GE2E loss for dual-attribute disentanglement is a standard but effective technique in this domain. The novelty lies primarily in the granular control offered by the DPPG representation and the unification of these specific editing tasks, rather than in fundamentally new generative architectures.
Results & evaluation
The experimental setup is comprehensive, covering speaker-emotion editing, word-level content editing, phoneme/sub-phoneme editing, and joint editing. The authors construct a large synthetic corpus, UniEditCorpus, to address the scarcity of parallel emotional speech data with diverse speakers. Results show that UniSAE achieves competitive performance in content preservation (CER) and strong performance in speaker/emotion similarity and naturalness (UTMOS) compared to baselines like EmoConv-Diff and ZEST. The ablation studies effectively demonstrate the importance of the disentangled embeddings and the DPPG representation. However, the reliance on a synthetically generated corpus for training and evaluation (UniEditCorpus) raises questions about generalization to real-world, noisy, or out-of-distribution data, although some evaluation on the ESD dataset provides a partial check. The phoneme editing results show high Target Phoneme Detection (TPD) but also note failures in lexical plausibility, which is a realistic limitation.
Reproducibility
The paper provides detailed implementation details, including the structure of the Content Transformer, the diffusion model, and the DPPG construction process (K-means on PPGs). The alignment algorithm using DTW with phoneme-aware matching costs is described in the supplementary material. The code is not publicly available (Project URL is none), and the demo link is anonymous, suggesting the paper is under review. While the methodology is described well enough for a competent researcher to reproduce, the lack of public code and the specific hyperparameters for the synthetic corpus generation might pose minor hurdles.
Limitations
A primary limitation is the reliance on synthetic data (UniEditCorpus) for training the disentanglement components, which may not fully capture the complexity of real-world emotional speech variations. The performance on unseen speakers in the ESD dataset shows degradation, indicating challenges in generalizing speaker representations. Additionally, the phoneme editing task, while precise, can lead to unnatural sounding speech if the resulting word is lexically invalid or contextually incongruent, as noted in the analysis. The diffusion-based decoder, while high quality, is computationally more expensive than autoregressive vocoders used in some baselines.
Broader impact
This work contributes to the field of controllable speech synthesis and editing, enabling more nuanced and fine-grained manipulation of speech attributes. Applications could include accessibility tools, dubbing, voice acting assistance, and personalized speech interfaces. The ability to edit at the sub-phoneme level offers new possibilities for linguistic research and speech therapy tools. However, the ease of manipulating speaker and emotion attributes also raises concerns about deepfake audio and misinformation, necessitating robust detection and ethical usage guidelines. [UniSAE introduces a unified speech attribute editing framework leveraging Discrete Phonetic PosteriorGrams to enable composable, fine-grained control over content, speaker, and emotion, addressing the limitations of existing word-level and single-attribute editing systems.]
Author of this paper? Send a correction.