Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization
Abstract
Recent audio language models can follow long conversations. However, research on emotion-aware or spoken dialogue summarization is constrained by the lack of data that links speech, summaries, and paralinguistic cues. We introduce Spoken DialogSum, the first corpus aligning raw conversational audio with factual summaries, emotion-rich summaries, and utterance-level labels for speaker age, gender, and emotion. The dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with emotion, pitch, and speaking rate. Second, an expressive TTS engine synthesizes speech from the tagged scripts, aligned with paralinguistic labels. Spoken DialogSum comprises 13,460 emotion-diverse dialogues, each paired with both a factual and an emotion-focused summary. The dataset is available online at https://fatfat-emosum.github.io/EmoDialog-Sum-Audio-Samples/. Baselines show that an Audio-LLM raises emotional-summary ROUGE-L by 28% relative to a cascaded ASR-LLM system, confirming the value of end-to-end speech modeling.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Spoken DialogSum, a pioneering dataset that integrates emotional and factual summaries of spoken dialogues, significantly advancing the field of spoken dialogue summarization. The methodology is innovative, addressing a critical gap in emotion-aware dialogue processing, and the results demonstrate substantial improvements in summarization performance, making it a valuable resource for future research.
Methodology
The methodology presented in this paper is robust and innovative, involving a three-stage process to create a large-scale spoken dialogue summarization corpus. The use of LLMs for style transfer and backchannel insertion, followed by TTS synthesis, demonstrates a thoughtful integration of modern NLP techniques with audio processing. The approach to generating both factual and emotion-rich summaries is well-structured and addresses a significant gap in existing datasets. However, the reliance on LLMs for dialogue rewriting and emotion assignment may introduce biases inherent in the models used.
Results & evaluation
The experimental evaluation is thorough, with clear benchmarks against existing datasets and models. The paper reports substantial improvements in ROUGE-L scores for emotional summaries when using Audio-LLMs, which validates the effectiveness of their proposed method. The human evaluation metrics further support the claims of enhanced naturalness and emotion expressivity, showcasing the dataset's potential for future research. However, the paper could benefit from additional comparative analyses with more diverse models to strengthen its claims.
Reproducibility
The paper provides a clear description of the methods and datasets used, along with the evaluation framework. However, the reproducibility of results may be limited by the proprietary nature of some models (e.g., GPT-4o-mini) and the specific configurations of the TTS system. More detailed implementation guidelines or code availability would enhance reproducibility.
Limitations
The main limitations include potential biases introduced by the LLMs used for dialogue transformation and emotion labeling, which may not generalize across different contexts or speakers. Additionally, while the dataset is large, it may still lack diversity in terms of dialects and cultural contexts, which could affect the generalizability of the findings.
Broader impact
This research has significant implications for the fields of conversational AI, emotion recognition, and dialogue systems. The dataset can facilitate advancements in emotion-aware summarization and improve the quality of human-computer interactions. However, ethical considerations regarding the use of emotion recognition technologies must be addressed to prevent misuse in surveillance or profiling contexts. The paper introduces Spoken DialogSum, a pioneering dataset that integrates emotional and factual summaries of spoken dialogues, significantly advancing the field of spoken dialogue summarization. The methodology is innovative, addressing a critical gap in emotion-aware dialogue processing, and the results demonstrate substantial improvements in summarization performance, making it a valuable resource for future research.
Author of this paper? Send a correction.