AnyBand: Unified Multi-Bandwidth Speech Extension via Frequency-Aware In-Context Spectral Infilling
Abstract
Bandwidth extension (BWE) aims to recover missing high-frequency content from band-limited speech. Existing methods often formulate BWE as a fixed or predefined bandwidth conversion problem, potentially requiring cutoff-specific models or retraining when the input bandwidth changes. This assumption limits their applicability to practical scenarios where speech may arrive with diverse cutoff frequencies. We propose AnyBand, a unified BWE framework that recasts bandwidth extension as in-context spectral infilling. Motivated by prompt-based zero-shot speech generation, AnyBand conditions high-frequency generation on the observed low-frequency spectrum, using the available band as a frequency-domain prompt that conveys content, speaker, prosodic, and spectral-envelope cues. This formulation enables a single model to perform cutoff-conditioned generation over a continuous range of input bandwidths. AnyBand is trained with missing-band conditional flow matching and an Easy-to-Balanced cutoff curriculum over continuously sampled cutoff frequencies. To better exploit the spectral prompt, we introduce a frequency-aware Diffusion Transformer that models cross-frequency interactions and long-range temporal dependencies, followed by a physically motivated multi-view adversarial refinement stage to enhance spectral realism, envelope coherence, and harmonic consistency. Experiments on multiple datasets and bandwidth settings show that AnyBand consistently improves spectral reconstruction over existing baselines while achieving competitive perceptual quality across both standard and irregular input cutoffs. Audio samples are available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
Methodology
The paper proposes AnyBand, a unified framework for speech bandwidth extension (BWE) that treats the task as in-context spectral infilling. The core innovation lies in conditioning a Diffusion Transformer (DiT) on the observed low-frequency spectrum as a "frequency-domain prompt" to generate the missing high-frequency content. The architecture augments a temporal DiT backbone with explicit frequency encoders/decoders to model cross-frequency interactions. The training strategy employs a novel "Easy-to-Balanced" cutoff curriculum to handle continuously varying cutoff frequencies, avoiding the need for separate models per bandwidth. Additionally, a multi-view adversarial refinement stage is introduced to enhance spectral realism, envelope coherence, and harmonic consistency. The approach is technically sound, leveraging recent advances in flow matching and diffusion transformers, and applies the concept of in-context learning (typically seen in LLMs) to the spectral domain.
Results & evaluation
The authors evaluate AnyBand on VCTK and EARS datasets, comparing against baselines like NU-Wave 2, AudioSR, FLowHigh, and Fre-Painter. Metrics include LSD, LF-LSD, HF-LSD, NISQA, COL, and STOI. The results show consistent improvements in spectral reconstruction (lower LSD) and competitive perceptual quality (NISQA) across various input cutoffs (2kHz, 4kHz, etc.). The experiments are comprehensive, covering both in-domain and out-of-domain settings. However, the reliance on objective metrics like NISQA and LSD, while standard, lacks extensive subjective human evaluation (MOS/MUSHRA) which is crucial for perceptual quality claims in audio restoration. The comparison is fair, but the marginal gains in some perceptual metrics over strong baselines like AudioSR warrant caution in claiming a transformative leap.
Reproducibility
The paper provides detailed architectural descriptions, loss functions, and training procedures. The inclusion of a demo page suggests code or at least pre-trained models might be available, though no GitHub URL is explicitly provided in the text. The "Easy-to-Balanced" curriculum and specific discriminator designs are described with sufficient mathematical detail for reproduction. The use of standard solvers (Heun) and common datasets enhances reproducibility.
Limitations
The method assumes the input low-frequency spectrum is clean and correctly aligned; it does not address blind BWE where the cutoff frequency is unknown or the input is degraded by noise/compression beyond simple band-limiting (though BABE is cited as related work, AnyBand itself requires the mask/cutoff extent). The computational cost of the DiT backbone and adversarial refinement may be higher than lightweight CNN-based BWE systems. The paper does not provide extensive ablation studies on the contribution of each discriminator component or the specific impact of the frequency encoder/decoder versus a standard DiT.
Broader impact
AnyBand has significant potential for improving speech quality in legacy communication systems, audio restoration, and voice conversion where bandwidth constraints are dynamic. By unifying multiple bandwidth settings into a single model, it reduces the engineering overhead of deploying multiple specialized models. However, like all generative audio tools, it carries risks of misuse in deepfake generation or misrepresentation, although BWE is generally considered a benign enhancement task. AnyBand introduces a unified, in-context spectral infilling framework for speech bandwidth extension using a frequency-aware Diffusion Transformer and a novel cutoff curriculum. The paper presents a well-motivated and technically sophisticated approach to a long-standing problem in audio processing. By reframing BWE as a conditional generation task conditioned on a spectral prompt, it elegantly handles continuous bandwidth variations. The integration of frequency-aware modeling within a DiT architecture is a strong technical contribution that addresses the specific inductive biases required for spectral data. While the novelty is incremental in the context of the rapid evolution of diffusion models, the specific application to unified BWE with the proposed curriculum and adversarial refinement is valuable. The work is significant for its practical utility in creating flexible, single-model solutions for diverse bandwidth scenarios, potentially simplifying deployment in real-world applications.
Author of this paper? Send a correction.