An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation
Abstract
While Large Multimodal Models excel in comprehension, high-throughput inference engines lack native support for multimodal generation. This is severe in Speech Language Models, where generating multi-layered audio tokens via decoupled AR+NAR or synchronous Multi-Token Prediction (MTP) with delay-pattern interleaving conflicts with standard single-stream loops. We present a vLLM-based inference pipeline for unified speech understanding and generation. We extend autoregressive decoding to natively execute delay-pattern de-interleaving and coordinated multi-stream sampling, integrating an on-GPU acoustic decoder for end-to-end waveform synthesis. Crucially, we overcome the shared intuition that Classifier-Free Guidance (CFG) halves throughput. By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput, absorbing dual-request and logit merging overheads. We open-source our framework.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper presents a significant systems contribution to the deployment of Speech Language Models by extending the vLLM inference engine to natively support multi-token audio generation and efficient Classifier-Free Guidance. By introducing paired request co-scheduling, the authors overcome the traditional throughput penalty of CFG, enabling high-fidelity audio synthesis at scale, which is a critical step for making unified audio understanding and generation models practically viable in real-world applications.
Methodology
The paper addresses a critical infrastructure gap in Speech Language Models (SpeechLMs): the lack of high-throughput inference support for multi-token generation and Classifier-Free Guidance (CFG) in existing engines like vLLM. The proposed methodology involves extending the vLLM architecture to handle "delay-pattern" interleaving common in Residual Vector Quantization (RVQ) based audio codecs. The core technical innovation is the "Paired Request Co-Scheduling" mechanism for CFG. Instead of treating conditional and unconditional passes as separate requests (which fragments KV cache and prevents joint batching), the authors fuse them into a single scheduling unit. This allows the transformer backbone to process both branches in a single forward pass (sharing the KV cache computation for the shared prefix), significantly reducing overhead. The integration of an on-GPU acoustic decoder further unifies the pipeline. The approach is technically sound and leverages low-level engine optimizations (PagedAttention, continuous batching) effectively.
Results & evaluation
The evaluation is conducted on three representative SpeechLM architectures (Bagpiper, OpusLM, OpusLM-Dialogue) on a single H100 GPU. The authors demonstrate massive throughput gains (two orders of magnitude) over sequential PyTorch baselines. Crucially, they address the CFG bottleneck, showing that their co-scheduling method sustains 80% of non-CFG throughput, whereas naive implementations would suffer significant degradation. Numerical correctness is verified against reference implementations, showing strict alignment in FP32 and acceptable divergence in BF16 that does not impact end-to-end quality metrics (WER, UTMOS). The experiments are comprehensive for a systems paper, covering throughput, hardware utilization (MFU), and output quality.
Reproducibility
The authors state they open-source their framework, which is a strong positive for reproducibility. The paper provides detailed descriptions of the architecture, including the phase state machine for mixed-modality handling and the specific logit merging logic for CFG. The experimental setup is clearly defined (H100, FlashAttention-3, specific models). However, the lack of a provided URL in the metadata requires verification of the actual repository link, though the claim of open-sourcing is present in the abstract.
Limitations
The paper focuses heavily on the inference engine and does not propose new model architectures or training methods. The performance gains are specific to the vLLM architecture and RVQ-based SpeechLMs; generalization to other codec structures or non-autoregressive models is not discussed. The CFG optimization is specific to the "paired" request pattern and may not generalize to more complex guidance schemes (e.g., iterative refinement). The evaluation is limited to a single GPU (H100), so scalability to multi-GPU setups is not demonstrated.
Broader impact
This work significantly lowers the barrier to deploying high-fidelity SpeechLMs by making them computationally viable for real-time or high-concurrency applications. By enabling efficient CFG, it improves the quality of generated speech without prohibitive latency costs. This contributes to the broader field of multimodal AI by bridging the gap between model capability and system efficiency. This paper presents a significant systems contribution to the deployment of Speech Language Models by extending the vLLM inference engine to natively support multi-token audio generation and efficient Classifier-Free Guidance. By introducing paired request co-scheduling, the authors overcome the traditional throughput penalty of CFG, enabling high-fidelity audio synthesis at scale, which is a critical step for making unified audio understanding and generation models practically viable in real-world applications.
Author of this paper? Send a correction.