LiveBand: Live Accompaniment Generation in the Audio Domain

Marco Pasini, Javier Nistal, Mathias Rose Bjare ...Full author list in the original paper.

Johannes Kepler University +1 · arXiv

Affiliations: Johannes Kepler University, Sony Computer Science Laboratories

Abstract

We present LiveBand, a real-time system that generates high-fidelity music accompaniments to live audio input, respecting strict causal constraints. Our method trains a causal transformer generator in the continuous latent space of a pre-trained causal audio autoencoder, using adversarial sequence-level supervision from a discriminator. At each timestep, the generator receives only the causally available mix context and Gaussian noise, and predicts accompaniment latents without access to future mix frames or ground-truth target latents. Training is performed in a single parallel forward pass under causal masking, while streaming inference proceeds autoregressively with a rolling attention state. The model's training and inference computations are matched by design, eliminating teacher forcing and the associated exposure bias. On a multi-instrument music accompaniment benchmark, LiveBand improves over prior work on objective measures of audio quality, beat alignment, and mix adherence, while enabling real-time streaming generation without lookahead into the future on consumer hardware.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The experimental evaluation is comprehensive and well-structured, providing both objective and subjective evidence for LiveBand's superiority. **Datasets and Metrics**: Training and evaluation on Slakh2100, a standard multi-instrument dataset, ensures comparability. The use of an internal corpus for one variant (LiveBand$_int$) adds a dimension of real-world data. The chosen objective metrics (FAD$_vgg/CLAP$, BA$_F1$, COCOLA) are highly relevant for assessing audio quality, rhythmic accuracy, and mix adherence in music generation. Evaluating over two 10-second segments to measure drift is particularly insightful for streaming models. **Baselines**: Comparing against StreamMusicGen (SMG), a recent and relevant causal autoregressive model, provides a strong benchmark. The inclusion of a bidirectional upper bound (LiveBand$_bid$) and ground truth offers valuable context for potential performance ceilings. **Results**: LiveBand consistently outperforms SMG across all objective metrics (FAD, BA, COCOLA) and anticipation settings (0s, 0.1s, 1s). This is a significant finding, especially the ability to maintain strong performance even with 1-second anticipation, demonstrating that the model can generate meaningfully ahead of the

Methodology

LiveBand proposes a robust and well-thought-out methodology for real-time, causal music accompaniment generation. The core innovation lies in training a causal transformer generator in the continuous latent space of a pre-trained causal audio autoencoder, using adversarial sequence-level supervision. This design choice directly addresses the critical challenges of real-time systems: strict causality, low latency, and maintaining musical coherence without future lookahead. A key strength is the elimination of teacher forcing and its associated exposure bias. By feeding the generator only independent per-step Gaussian noise and causally available mix context, the training and inference conditions are matched by design. This allows for parallel computation during training while ensuring that the model's internal state accurately reflects its own generated history during autoregressive inference, avoiding the drift common in teacher-forced models. This is a significant improvement over prior autoregressive approaches that struggle with error accumulation. The use of a pre-trained *causal* audio autoencoder (a retrained CoDiCodec variant) is crucial, as it ensures that the latent representations themselves respect causality, which is fundamental to the real-time constraint. Operating in a continuous latent space, as opposed to discrete tokens, likely contributes to the high-fidelity claim and potentially smoother generation. Adversarial sequence-level supervision from a non-causal 1D convolutional discriminator is another strong methodological choice. This encourages the generator to produce holistically realistic and coherent sequences, rather than focusing on frame-level accuracy which can be overly rigid for musical tasks requiring continuous realignment. The discriminator's non-causal nature is justified by its training-only role, allowing it to provide a richer signal. Finally, the Adaptive Gradient Penalty (AdaGP) is a practical and elegant solution to stabilize adversarial training. By dynamically adjusting the gradient penalty weight to maintain a fixed discriminator advantage, it reduces the burden of hyperparameter tuning and improves training robustness, a common pain point in GANs. The generator's architecture, a pre-norm transformer with adaptive layer normalization, query-key normalization, Rotary Position Embeddings, and FlexAttention with scalar attention sink, incorporates modern best practices for stable and efficient transformer training.

Results & evaluation

Author of this paper? Send a correction.