NSTR: Neural Spectral Transport Representation for Space-Varying Frequency Fields
Abstract
Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing signals such as images, audio, and 3D scenes. However, existing INR frameworks -- including MLPs with Fourier features, SIREN, and multiresolution hash grids -- implicitly assume a \textit{global and stationary} spectral basis. This assumption is fundamentally misaligned with real-world signals whose frequency characteristics vary significantly across space, exhibiting local high-frequency textures, smooth regions, and frequency drift phenomena. We propose \textbf{Neural Spectral Transport Representation (NSTR)}, the first INR framework that \textbf{explicitly models a spatially varying local frequency field}. NSTR introduces a learnable \emph{frequency transport equation}, a PDE that governs how local spectral compositions evolve across space. Given a learnable local spectrum field $S(x)$ and a frequency transport network $F_θ$ enforcing $\nabla S(x) \approx F_θ(x, S(x))$, NSTR reconstructs signals by spatially modulating a compact set of global sinusoidal bases. This formulation enables strong local adaptivity and offers a new level of interpretability via visualizing frequency flows. Experiments on 2D image regression, audio reconstruction, and implicit 3D geometry show that NSTR achieves significantly better accuracy-parameter trade-offs than SIREN, Fourier-feature MLPs, and Instant-NGP. NSTR requires fewer global frequencies, converges faster, and naturally explains signal structure through spectral transport fields. We believe NSTR opens a new direction in INR research by introducing explicit modeling of space-varying spectrum.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents NSTR, a novel framework for modeling spatially varying frequency fields in implicit neural representations, which enhances expressivity, stability, and interpretability in signal reconstruction. The innovative use of a learnable PDE for frequency transport represents a significant advancement in the field, addressing key limitations of existing INR methodologies.
Methodology
The proposed methodology introduces a novel framework, NSTR, which explicitly models spatially varying frequency fields through a learnable frequency transport PDE. This approach effectively decouples global frequency content from local spectral variation, allowing for adaptive representation of signals. The use of a PDE to govern the evolution of the local spectrum is particularly innovative, as it introduces a structured constraint that enhances interpretability and stability in the representation learning process. The parameterization of the local spectrum field using a coarse grid and a lightweight MLP is efficient, addressing the limitations of traditional INRs that rely on fixed global bases.
Results & evaluation
The experiments conducted across diverse tasks, including 2D image regression, audio waveform reconstruction, and implicit 3D geometry, demonstrate the effectiveness of NSTR in achieving superior accuracy-parameter trade-offs compared to existing methods like SIREN and Fourier-feature MLPs. The evaluation metrics used are appropriate for the tasks, and the results indicate significant improvements in fidelity and convergence speed. However, the paper could benefit from additional quantitative comparisons and visualizations to further substantiate its claims.
Reproducibility
While the paper provides a detailed description of the architecture and training setup, it lacks specific implementation details such as code availability or links to datasets used for experiments. This hinders reproducibility, as independent researchers may struggle to replicate the results without access to the exact configurations and data.
Limitations
One limitation is the lack of real-world application examples, as the experiments are primarily conducted on standard datasets. Additionally, the paper does not address potential computational overhead associated with the learnable PDE, which may impact scalability in more complex scenarios. The reliance on a fixed number of global frequencies may also limit the adaptability of the model in highly variable signal contexts.
Broader impact
The introduction of NSTR has the potential to significantly advance the field of implicit neural representations by providing a more flexible and interpretable framework for modeling complex signals. Its applications could extend to various domains, including graphics, audio processing, and scientific simulations, where understanding local frequency variations is crucial. The ability to visualize frequency flows could also enhance interpretability in machine learning models, fostering trust and understanding in AI systems. The paper presents NSTR, a novel framework for modeling spatially varying frequency fields in implicit neural representations, which enhances expressivity, stability, and interpretability in signal reconstruction. The innovative use of a learnable PDE for frequency transport represents a significant advancement in the field, addressing key limitations of existing INR methodologies.
Author of this paper? Send a correction.