Point of Order: Action-Aware LLM Persona Modeling for Realistic Civic Simulation
Abstract
Large language models offer opportunities to simulate multi-party deliberation, but realistic modeling remains limited by a lack of speaker-attributed data. Transcripts produced via automatic speech recognition (ASR) assign anonymous speaker labels (e.g., Speaker_1), preventing models from capturing consistent human behavior. This work introduces a reproducible pipeline to transform public Zoom recordings into speaker-attributed transcripts with metadata like persona profiles and pragmatic action tags (e.g., [propose_motion]). We release three local government deliberation datasets: Appellate Court hearings, School Board meetings, and Municipal Council sessions. Fine-tuning LLMs to model specific participants using this "action-aware" data produces a 67% reduction in perplexity and nearly doubles classifier-based performance metrics for speaker fidelity and realism. Turing-style human evaluations show our simulations are often indistinguishable from real deliberations, providing a practical and scalable method for complex realistic civic simulations.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of a novel framework for action-aware persona modeling in civic simulations, which significantly enhances the realism of multi-party deliberations through the integration of structured metadata and fine-tuning of large language models. This work addresses critical gaps in the field, providing both methodological advancements and practical datasets that can facilitate further research and applications in civic engagement and decision-making processes.
Methodology
The paper presents a novel multimodal speaker-linking framework that integrates visual, audio, and textual cues to generate speaker-attributed transcripts from public Zoom recordings. This approach addresses the significant challenge of speaker diarization in noisy environments, which is a common issue in real-world deliberations. The introduction of action-aware persona modeling, which augments transcripts with structured metadata, enhances the realism of simulations. The methodology is well-structured, with clear steps for data collection, metadata extraction, and fine-tuning of large language models (LLMs) using parameter-efficient techniques. The use of novel evaluation metrics (Classifier Fool Rate and Speaker Attribution Accuracy) is a strong point, providing a comprehensive assessment of the model's performance.
Results & evaluation
The experiments are robust, involving three different datasets of government deliberations, which allow for a thorough evaluation of the proposed methods. The results demonstrate significant improvements in perplexity and classification metrics, indicating that the fine-tuning with structured metadata leads to enhanced speaker fidelity and realism. The paper also includes human evaluations, which lend credibility to the findings, showing that the simulations are often indistinguishable from real deliberations. However, the paper could benefit from a more detailed statistical analysis of the results to further substantiate claims of improvement.
Reproducibility
The paper outlines a clear methodology for data collection and processing, including the use of specific tools like yt-dlp and Whisper for transcription. However, it lacks detailed information on the implementation of the models and the specific configurations used during fine-tuning. Providing access to code or a repository would greatly enhance reproducibility.
Limitations
One limitation is the reliance on Zoom recordings, which may not fully represent the diversity of deliberative contexts. Additionally, while the datasets are a significant contribution, their size and variety may still limit the generalizability of the findings. The paper does not address potential biases in the data or the implications of using LLMs trained on such datasets.
Broader impact
The work has significant implications for civic engagement and public decision-making, as it enables realistic simulations of multi-party deliberations. This can be useful for training decision-makers, analyzing institutional decision-making processes, and exploring the dynamics of public discourse. The release of speaker-labeled datasets also provides a valuable resource for future research in this area. The main contribution of this paper is the introduction of a novel framework for action-aware persona modeling in civic simulations, which significantly enhances the realism of multi-party deliberations through the integration of structured metadata and fine-tuning of large language models. This work addresses critical gaps in the field, providing both methodological advancements and practical datasets that can facilitate further research and applications in civic engagement and decision-making processes.
Author of this paper? Send a correction.