Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
Abstract
Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands responses on a millisecond timescale. Smaller, real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, where a small talker model both immediately generates contextually grounded responses to hide the latency of an external reasoner model and fluently integrates streamed reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and demonstrate that this task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system implementation, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier reasoner performance. In a live user study (n=18) with talker deployments running on an Apple M2 SoC, participants rank ConvFill on par with frontier models overall, prefer it for retrieval-heavy tasks, and rate it significantly more responsive. These results show that conversational infill unlocks a new point on the latency-capability Pareto frontier, offering a practical path toward voice agents that are both responsive and highly capable. Code, models, and datasets are available at https://github.com/vysri/conversational-infill.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Conversational Infill, a novel architecture that enables small language models to generate immediate, contextually appropriate filler responses while streaming knowledge from a larger reasoning model, effectively bridging the latency-capability gap in voice agents. This approach offers a practical and scalable solution for building responsive, high-fidelity conversational AI systems that maintain user engagement without sacrificing factual accuracy or reasoning depth.
Methodology
The paper proposes "Conversational Infill" (ConvFill), a system architecture that decouples the latency-sensitive text-to-speech (TTS) generation from the high-latency reasoning/retrieval process. It employs a "Talker-Reasoner" paradigm where a small language model (SLM) acts as the Talker, generating immediate filler phrases to maintain conversational flow while a larger Reasoner model processes the query and streams knowledge chunks. The Talker is fine-tuned to seamlessly integrate these streamed chunks into its response. The methodology involves curating a large synthetic dataset (290k examples) with strict validation pipelines to ensure grounding and non-contradiction. The approach is technically sound and addresses a critical infrastructure problem in voice AI: the latency-capability trade-off. The use of control tokens and a queue-based inference pipeline is a practical engineering contribution.
Results & evaluation
The evaluation is comprehensive, spanning single-turn QA, multi-turn dialogue, and a live user study. The authors demonstrate that ConvFill systems (using SLMs like SmolLM2 and Llama 3.2) achieve accuracy within 6.3% of frontier models (GPT-4o, Claude Opus) while maintaining millisecond-level time-to-first response (TTFR). The user study (n=18) provides strong qualitative evidence that users perceive the system as significantly more responsive and prefer it for retrieval-heavy tasks. The inclusion of metrics like Entailment, Non-Contradiction, Coverage, and Faithfulness offers a nuanced view of system quality beyond simple accuracy. However, the user study sample size is small, and the live latency measurements, while impressive, are dependent on the specific hardware (Apple M2) and network conditions for the Reasoner.
Reproducibility
The authors release the dataset, code, and fine-tuned model weights, which significantly enhances reproducibility. The dataset generation pipeline is described in detail, including the validation steps. The training configurations are provided. The reliance on API-based Reasoners (GPT-4o, Claude) for the "ground truth" knowledge generation is a variable, but the Talker models are open-weight, allowing others to replicate the core inference mechanism.
Limitations
The paper acknowledges that the system relies on a large external model for reasoning, which incurs cloud costs and privacy implications, limiting its utility for fully on-device private applications. The user study was limited to 18 participants, which may not capture broader demographic variations or long-term usage fatigue. The synthetic dataset, while large, may not fully capture the edge cases of real-world human speech, such as interruptions, overlapping speech, or highly colloquial language, although the live study mitigates this to some extent. The performance gap with frontier models, while small, is still present, particularly on complex reasoning tasks.
Broader impact
This work has significant implications for the deployment of AI voice assistants, making them more usable and natural by removing the "thinking pause." It lowers the barrier for deploying capable voice agents on edge devices by offloading heavy reasoning to the cloud while keeping the interaction local and responsive. It also contributes to the field of model collaboration and latency optimization. The release of the dataset and code will facilitate further research in responsive conversational AI. The paper introduces Conversational Infill, a novel architecture that enables small language models to generate immediate, contextually appropriate filler responses while streaming knowledge from a larger reasoning model, effectively bridging the latency-capability gap in voice agents. This approach offers a practical and scalable solution for building responsive, high-fidelity conversational AI systems that maintain user engagement without sacrificing factual accuracy or reasoning depth.
Author of this paper? Send a correction.