Reference-Guided Turn-Taking Style Transfer for Full-Duplex Spoken Language Models
Abstract
Full-duplex spoken language models decide at every frame whether to take the floor, defining a conversational personality -- yet each set of weights encodes exactly one, which human speakers bracket rather than share. Existing controls mostly act from outside; none identifies the quantity that sets how much the model speaks. We identify it: a single late-layer residual-stream direction, the initiative vector, extracted from the model's own conversations without training or annotation. The same direction reads the decision, anticipating onsets up to 800 ms ahead, and shifts it: scaled during generation, it moves speech ratio from 6-12% to 42-61% (no steering: 21-30%), consistently across conversations, weights, and corpora. A short reference recording suffices to transfer a speaker's turn-taking style to the model, and the transfer is reference-specific: swapping in another speaker's reference worsens the match in 93-98% of cases, whereas the best uniform dose is indistinguishable from no steering. Turn-taking style is not frozen into the weights but a direction that can be read, shifted, and calibrated per person. Audio examples are available at https://elpsykongloo.github.io/initiative-vector/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies a single residual-stream direction, the "initiative vector," that linearly encodes and controls turn-taking behavior in full-duplex spoken language models. By demonstrating that this vector can be extracted without training and used to steer speech ratio and transfer individual turn-taking styles from short reference recordings, the work provides a novel, efficient, and interpretable mechanism for personalizing conversational AI, bridging the gap between fixed model weights and dynamic human-like interaction styles.
Methodology
The paper proposes a method to identify and manipulate a specific "initiative vector" in the residual stream of full-duplex spoken language models (specifically the Moshi architecture). The methodology involves extracting this vector via class-mean contrasts of frames preceding speech onsets versus silent frames, requiring no additional training or annotation. The core technical contribution is the demonstration that this single direction linearly encodes the model's propensity to speak and can be scaled during generation to control turn-taking behavior. The authors further develop a reference-guided transfer mechanism that calibrates this steering dose based on a short reference recording of a target speaker's style, using descriptors like speech ratio and overlap onsets. The approach is elegant in its simplicity, leveraging linear interpretability techniques (similar to activation steering in LLMs) applied to the temporal dynamics of speech models.
Results & evaluation
The experiments are conducted on three distinct full-duplex weights (Moshika, Moshika-RL, PersonaPlex) using the Seamless Interaction and CANDOR corpora. The evaluation metrics include AUC for onset prediction, speech ratio changes, and distance metrics for style transfer. The results show that the initiative vector anticipates onsets with high AUC (up to 0.968) and that scaling it monotonically increases speech ratio. The style transfer experiments demonstrate that using a specific speaker's reference yields significantly better matches than uniform doses or other speakers' references. The statistical analysis includes confidence intervals and comparisons against baselines, providing robust evidence for the claims.
Reproducibility
The paper provides sufficient detail for reproducibility, including the specific layer (L30), the extraction method (class means), the dose scaling factor, and the descriptors used for reference matching. The use of public datasets (Seamless Interaction, CANDOR) and open-source models (Moshi) enhances reproducibility. However, the specific implementation details of the steering mechanism during generation (how exactly the residual is modified in the KV cache) could be more explicitly detailed in the main text, though the equations provided are clear.
Limitations
The study is limited to the Moshi architecture and its variants; generalizability to other full-duplex architectures (e.g., those with different temporal resolutions or attention mechanisms) is not tested. The steering is applied uniformly across all frames, which may not capture more nuanced, context-dependent turn-taking strategies. Additionally, the "style" is defined by a few acoustic descriptors, which may not fully capture the complexity of human conversational personality.
Broader impact
This work has significant implications for the development of more natural and personalized spoken dialogue systems. By enabling fine-grained control over turn-taking without retraining, it offers a practical path to adapting models to different user preferences or application contexts (e.g., tutoring vs. companionship). It also contributes to the interpretability of speech models, providing insight into how conversational decisions are computed within the network. The paper identifies a single residual-stream direction, the "initiative vector," that linearly encodes and controls turn-taking behavior in full-duplex spoken language models. By demonstrating that this vector can be extracted without training and used to steer speech ratio and transfer individual turn-taking styles from short reference recordings, the work provides a novel, efficient, and interpretable mechanism for personalizing conversational AI, bridging the gap between fixed model weights and dynamic human-like interaction styles.
Author of this paper? Send a correction.