HiPLEX: Hierarchical Policy Factorization for Full Duplex Speech Language Models
Abstract
As human--AI interactions become more conversational, full-duplex speech language models capable of natural real-time dialogue are growing in importance. Beyond generating appropriate responses, these models must coordinate turn-taking, backchanneling, and floor management in real time. Reinforcement learning (RL) provides a way to refine these behaviors through direct feedback on interaction outcomes. However, existing RL methods either apply timing feedback to a token policy or optimize semantic content, leaving the joint improvement of timing and content unresolved. We introduce HiPLEX, an RL framework that factorizes a pretrained full-duplex text policy into a control policy that decides when to emit content and a conditional content policy that decides what to emit. The first factor selects among 'pad', 'epad', and 'con'. The second selects a token only when 'con' is chosen. This hierarchy describes conditional actions within each frame and uses the model's existing text head. We route timing advantages to the token-group factor through event-causal masks derived from generated speech episodes, and route an LLM-judge semantic advantage to the conditional content factor. Across three Moshi seeds on Full-Duplex-Bench v1, HiPLEX reduces takeover rates during natural user pauses and backchannel opportunities, and shortens post-interruption response latency relative to GRPO, while maintaining comparable judged interruption-response quality. On Moshi and PersonaPlex, HiPLEX better matches pooled human turn-timing and backchannel-rate marginals than GRPO.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
HiPLEX introduces a parameter-free hierarchical policy factorization and event-causal credit assignment mechanism that enables simultaneous optimization of timing and semantic content in full-duplex speech language models, outperforming flat GRPO baselines in interaction naturalness and responsiveness.
Methodology
The paper proposes HiPLEX, a hierarchical policy factorization for full-duplex speech language models. The core methodological contribution is decomposing the single text policy head into a high-level "control" policy (deciding whether to emit content: pad, epad, or con) and a low-level "content" policy (deciding which token to emit, conditional on 'con'). This factorization is parameter-free, derived directly from the existing softmax distribution via log-sum-exp grouping. The innovation lies in the credit assignment mechanism: "event-causal masks" that route timing advantages to specific control decisions based on the model's generated speech episodes (e.g., penalizing waiting frames for late responses rather than the onset frame itself), while semantic advantages from an LLM judge are routed to the content policy. This addresses the credit assignment problem in streaming speech RL more precisely than flat GRPO or fixed-window approaches.
Results & evaluation
The experiments are conducted on Moshi and PersonaPlex models using the Full-Duplex-Bench v1. The paper compares HiPLEX against a reproduced GRPO baseline and published results from Ohashi et al. Results show improvements in pause restraint, backchannel takeover rates, and post-interruption latency. Ablations effectively demonstrate the necessity of event-causal masks (showing that random masking leads to silence collapse) and the benefit of semantic feedback. The use of Wasserstein-1 distance to human timing marginals is a strong evaluation choice that goes beyond binary pass/fail metrics.
Reproducibility
The paper provides detailed algorithmic descriptions, reward function definitions, and hyperparameters in the appendices. It includes a "Reproducibility Statement" and an "AI Use Statement." The factorization is mathematically verified with numerical checks. However, the specific implementation code is not linked in the provided text, and the reliance on specific proprietary models (Moshi, PersonaPlex) and benchmarks (Full-Duplex-Bench) may limit immediate reproducibility for those without access to these resources.
Limitations
The method is specific to Moshi-style architectures with a distinct text stream aligned to audio. The "event-causal masks" require accurate VAD and episode detection, which could be sensitive to noise or overlapping speech. The semantic reward relies on an external LLM judge (Gemini), introducing potential bias and cost. The paper notes that PersonaPlex requires careful learning rate tuning to avoid collapse, indicating fragility in the optimization landscape.
Broader impact
This work advances the field of full-duplex speech interaction by providing a principled way to optimize timing and content separately. It could influence the design of future conversational AI agents that require real-time floor management. The hierarchical credit assignment approach may be applicable to other sequential decision-making tasks in language models. HiPLEX introduces a parameter-free hierarchical policy factorization and event-causal credit assignment mechanism that enables simultaneous optimization of timing and semantic content in full-duplex speech language models, outperforming flat GRPO baselines in interaction naturalness and responsiveness.
Author of this paper? Send a correction.