Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Abstract
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
Methodology
The paper proposes NAPE, a self-supervised learning framework for audio that adapts the "next-token prediction" paradigm from Large Language Models (LLMs) to continuous audio patch embeddings. The core innovation is a minimalist causal Transformer that predicts the next patch embedding in a log-mel spectrogram sequence using only causal masking and a stop-gradient operator on the target, avoiding reconstruction decoders, tokenizers, or student-teacher setups. The methodology is theoretically sound and aligns with recent trends in joint-embedding predictive architectures (like JEPA) but applies them to the temporal domain of audio. The design choices regarding scanning orders (Raster, Time-major, Zigzag, Diagonal) for the 2D spectrogram grid are a significant technical contribution, as they address the inductive bias required for causal prediction in non-isotropic data. The use of negative cosine similarity with stop-gradient is a known technique (from SimSiam/JSSE), so the novelty lies primarily in the application and specific architectural adaptations (scanning orders, causal masking) rather than a fundamentally new loss function.
Results & evaluation
The experimental evaluation is comprehensive, covering six standard audio and speech benchmarks (AudioSet, ESC-50, Speech Commands V1/V2, IEMOCAP). The authors demonstrate state-of-the-art or competitive performance on several tasks, particularly in fine-tuning scenarios. They also provide linear-probing results, which is crucial for assessing the quality of the learned representations independent of the downstream task head. The scaling analysis (Small, Base, Large) shows consistent performance improvements, suggesting the method is scalable. The ablation studies on scanning orders and predictor heads provide valuable insights into the design space. However, the paper lacks direct comparison with other autoregressive audio models (if any exist in the immediate literature) or joint-embedding methods specifically tailored for audio (like WavJEP), which would strengthen the claim of being a "markedly different" and superior philosophy. The results are strong, but the "state-of-the-art" claim should be contextualized against the specific baselines provided.
Reproducibility
The paper provides a GitHub repository link and details the architectural components (Conv2d patch embedding, RoPE, LayerScale, etc.). The training objective is clearly defined with pseudocode. The pre-training data (AudioSet) and preprocessing steps (16kHz, 10s clips, 128 mel bands) are standard and well-documented. The inclusion of ablation studies on key hyperparameters (scanning order, predictor head) enhances reproducibility. The code availability is explicitly stated.
Limitations
The paper does not explicitly discuss the computational cost of the causal masking compared to masked modeling or contrastive learning. Causal Transformers can be more memory-intensive during training due to the attention mask. The paper also does not explore the robustness of the scanning order across different audio domains (e.g., music vs. speech vs. environmental sounds) in depth, although it mentions it. The reliance on AudioSet for pre-training might limit generalization to out-of-distribution domains not well-represented in AudioSet.
Broader impact
This work contributes to the democratization of audio representation learning by providing a simpler, more unified framework that aligns with the dominant paradigm in NLP and Vision. This could lead to more efficient pre-training pipelines and better transfer learning capabilities for audio tasks. It also opens up new avenues for research into autoregressive audio generation and understanding by establishing a strong baseline for next-embedding prediction. NAPE introduces a minimalist, causal next-patch-embedding prediction framework for audio self-supervised learning that achieves state-of-the-art performance by leveraging temporal structure and scanning order inductive biases. The paper presents a significant and timely contribution to audio representation learning by successfully adapting the autoregressive next-token prediction paradigm, dominant in LLMs and emerging in vision, to the audio domain. The key technical insight is the careful handling of the 2D-to-1D linearization of spectrograms through various scanning orders, which serves as a critical inductive bias for causal prediction. By eliminating the need for complex components like reconstruction decoders, tokenizers, or student-teacher networks, NAPE offers a cleaner, more scalable, and theoretically elegant alternative to existing SSL methods. The empirical results demonstrate that this simplicity does not come at the cost of performance, with NAPE achieving competitive or superior results across multiple benchmarks. The work is well-executed, with thorough ablations and a clear presentation. It has the potential to influence future audio foundation models by establishing a new standard for predictive pre-training objectives.
Author of this paper? Send a correction.