Sonic-AMG: Streaming Human Motion Generation from Spatial Audio
Abstract
Modeling human reactions to spatial audio requires understanding both what is heard and where it comes from. Prior work generates motion offline from complete acoustic and spatial input sequences. Humans, however, react to sounds without knowing what they will hear next. We present Sonic-AMG, to our knowledge the first framework for streaming human motion generation from spatial audio. The model encodes acoustic content and body-relative source geometry as separate tokens. Conditioned on these tokens, a causal transformer with a flow head generates continuous motion latents, from which a causal decoder reconstructs motion segments. Standard teacher-forced training derives motion histories and body-relative source locations from ground-truth motion, whereas streaming inference relies on generated motion for both, resulting in a training-inference mismatch. To reduce this history discrepancy, we introduce rehearsal training, which combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. To keep spatial conditioning consistent with generated motion, we propose spatial feedback, which recomputes body-relative source locations from generated root poses during both rehearsal training and inference. On SAM, Sonic-AMG achieves state-of-the-art FID and R-precision. It further supports real-time online generation of coherent reactions across sound events and source locations. Project page: https://jingchengni.com/sonic-amg/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Methodology
Sonic-AMG introduces a novel framework for streaming human motion generation from spatial audio, addressing key limitations of prior offline methods. The core methodology is built upon a causal transformer with a flow head, operating on continuous motion latents from a pretrained VAE. A significant methodological contribution is the separation of acoustic content ("what") and body-relative source geometry ("where") into distinct conditioning tokens. This design choice, supported by ablations, allows the model to better disentangle and utilize these modalities. The 11-dimensional spatial descriptor, inspired by perceptual research, is a thoughtful design for capturing relevant spatial cues like proximity, relative motion, and looming. The paper effectively tackles two critical training-inference mismatches inherent in autoregressive streaming generation. First, the "history mismatch" (teacher-forced training vs. generated histories at inference) is addressed by "rehearsal training." This innovative approach combines full-sequence teacher-forced supervision with supervised prediction along short autoregressive self-rollouts. This hybrid strategy allows the model to learn stable long-range dependencies while also gaining robustness to accumulated errors from its own predictions. Second, the "spatial conditioning mismatch" (ground-truth SSL vs. generated root poses) is resolved by "spatial feedback." This mechanism recomputes body-relative source locations from generated root poses during both rehearsal training and inference, ensuring consistency between the generated motion and the spatial conditions. The integration of these components within a causal transformer and flow-matching framework is technically sound and well-justified. The use of classifier-free guidance during inference further enhances generation quality.
Results & evaluation
The experimental evaluation is comprehensive and well-structured, primarily using the SAM dataset. The paper establishes state-of-the-art performance for Sonic-AMG on FID and R-precision metrics, demonstrating both distributional fidelity and conditioning alignment. A crucial aspect of the evaluation is the distinction between "GT SSL" (ground-truth spatial conditioning, for fair comparison with baselines) and "Online SSL" (spatial feedback from generated motion, reflecting real-world streaming). Sonic-AMG maintains strong performance under the more challenging Online protocol, which is a significant achievement. The ablation studies are particularly strong, providing clear evidence for the efficacy of the proposed components: 1. **Rehearsal training vs. two-forward training:** Rehearsal training consistently outperforms various two-forward schedules, validating the benefit of recursive history construction. 2. **Role of teacher-forced supervision:** Demonstrates that combining self-rollouts with full-sequence supervision is crucial for fidelity and alignment. 3. **Effect of spatial feedback:** Shows that training with spatial feedback dramatically improves performance under the Online protocol, confirming its necessity for consistent spatial conditioning. 4. **Separate spatial tokens:** Ablation against a "Fused" approach confirms the advantage of distinct acoustic and spatial tokens. 5. **Learned acoustic features:** Shows that simpler, learned log-mel features outperform MOSPA's high-dimensional audio descriptors, supporting the hypothesis that separating "what" from "where" helps the acoustic branch focus on content. Beyond quantitative metrics, the paper provides compelling qualitative results, including examples of recognizable and spatially appropriate reactions, and the ability to generate coherent motion across complete audio clips and successive sound events. The demonstration of real-time streaming efficiency, with significantly reduced initial response latency compared to MOSPA, is a strong practical contribution. The inclusion of physical simulation retargeting to a Unitree G1 robot further showcases the robustness and applicability of the generated motions. The "No-SSL" variant, which derives spatial tokens directly from binaural audio, is an excellent exploration of alternative inference scenarios, demonstrating the model's flexibility and potential for deployment without explicit source location inputs.
Reproducibility
The paper includes a dedicated "Reproducibility statement" and detailed appendices covering implementation details, training protocols, and runtime measurements. This includes specifics on architecture (transformer layers, heads, width, flow head details), optimization (AdamW, learning rate, batch size), training hardware and time, motion autoencoder pretraining, acoustic and spatial encoding, guidance, and training sequence construction for both teacher-forced and rehearsal training. The authors commit to releasing code and models upon acceptance, which, combined with the extensive details provided, suggests high reproducibility.
Limitations
The authors acknowledge several limitations: 1. **Known world-space source positions:** The main experiments assume known source positions, although the No-SSL variant explores deriving spatial conditions from binaural audio. A systematic evaluation of alternative spatial inference methods is left for future work. 2. **Dataset generalization:** Evaluation is limited to the SAM dataset, so generalization to broader acoustic environments and human reactions needs further assessment. 3. **Long-horizon coherence:** While qualitative evidence for long-horizon coherence and event transitions is provided, quantitative benchmarks for evaluating motion quality over longer audio streams with diverse event transitions are needed. Additional limitations could include: * The reliance on a pretrained causal temporal VAE, which
Author of this paper? Send a correction.