Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers
Abstract
Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple's most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design, a streaming encoder, a temporal decoder, and a depth decoder, that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10 ms per generation step, about 16x faster than real time, with a peak runtime memory of only 21 MB and 329 MB of on-device assets, enabling continuous streaming synthesis of 20-320 seconds of audio. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Ablation studies validate the key architectural components, and audio quality assessment confirms that the architecture maintains synthesis fidelity while achieving efficiency gains over existing methods. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a significant engineering achievement in deploying efficient, high-quality audio synthesis on-device, introducing architectural innovations that decouple temporal and depth processing to achieve constant memory complexity and superior efficiency compared to prior methods.
Methodology
The paper proposes a memory-efficient audio synthesis architecture designed for on-device deployment, specifically targeting the Apple Matrix Coprocessor (AMX). The core methodological contribution is a three-component design: a streaming encoder, a temporal decoder, and a depth decoder. The key innovation lies in the "depth decoder," which replaces the per-level input/output projections found in prior works like Moshi with a single, fully shared transformer body conditioned on the RVQ level index via DiT-style rotary positional encoding (RoPE). This eliminates parameter overhead associated with the number of quantization levels. Additionally, the architecture employs causal sliding window attention with a fixed-window key-value cache to ensure constant memory complexity independent of sequence length, a critical feature for real-time streaming. The temporal decoder bridges semantic tokens and RVQ tokens, incorporating a temporal lookahead mechanism and autoregressive feedback from the mean RVQ embeddings. The approach effectively decouples temporal and depth processing, allowing for efficient generation of high-fidelity audio from semantic tokens.
Results & evaluation
The evaluation is comprehensive, covering both algorithmic scaling and production deployment metrics. On a common GPU (NVIDIA H100), the proposed model demonstrates constant memory and generation time scaling with sequence length (up to 320 seconds), whereas baselines (Transformer Decoder, Autoregressive GAN) scale linearly or quadratically. On-device, the model achieves a peak runtime memory of 21 MB and sustains ~10 ms per generation step (16x real-time). Audio quality is assessed using objective metrics (UTMOS, SI-SNR, VisQOL, MOSNet) and subjective Mean Opinion Score (MOS) evaluations. The model outperforms the prior on-device system (+0.28 overall MOS, +0.42 on conversational speech) and shows competitive or superior performance against baselines like Moshi/Mimi and internal GAN/Transformer models on UTMOS and SI-SNR. Ablation studies validate the contributions of the unified depth decoder, DiT conditioning, fixed KV cache, and temporal lookahead.
Reproducibility
The paper provides detailed architectural descriptions, including layer configurations (4-6-2), attention mechanisms, and conditioning strategies. However, reproducibility is significantly hindered by the use of a proprietary audio corpus ("internally recorded voice assistant interactions") and the lack of open-source code. The hyperparameters and training details are described, but the specific data distribution and preprocessing pipelines are not disclosed due to confidentiality protocols. The comparison with baselines is conducted on a common GPU, which helps, but the lack of access to the training data and code limits independent verification of the exact performance claims.
Limitations
The primary limitation is the reliance on proprietary data and closed-source implementation, preventing independent replication and broader community adoption. The model is specifically optimized for Apple's AMX hardware, and while algorithmic scaling is reported on GPUs, the real-world performance on other platforms may vary. The "temporal lookahead" mechanism, while effective, introduces a slight latency overhead and requires access to future semantic tokens, which might be a constraint in strictly causal, low-latency streaming scenarios where future context is not immediately available. The comparison with Moshi is somewhat indirect, as Moshi's on-device performance metrics are not provided for direct comparison, although algorithmic scaling is compared.
Broader impact
This work demonstrates the feasibility of high-quality, real-time, on-device audio synthesis with minimal resource overhead, enabling privacy-preserving voice assistants and immersive audio applications without cloud dependency. The architectural innovations, particularly the decoupled temporal-depth processing with shared parameters and fixed-window attention, offer valuable design patterns for efficient transformer-based generative models in resource-constrained environments. It pushes the boundary of what is possible with on-device AI, potentially reducing reliance on cloud infrastructure for audio generation tasks. The paper presents a significant engineering achievement in deploying efficient, high-quality audio synthesis on-device, introducing architectural innovations that decouple temporal and depth processing to achieve constant memory complexity and superior efficiency compared to prior methods.
Author of this paper? Send a correction.