FillGauss: Fine-Grained Filling-Aware Impact Sound Generation for 3D Gaussian Splatting
Abstract
Synthesizing physically plausible impact sounds from visual observations remains a great challenge in multi-modal AI. Existing 3D-aware audio generation methods primarily model the surface geometry of hollow rigid bodies. However, they fundamentally overlook internal filling states, a critical physical factor that drastically modulates acoustic resonance and damping. To address this issue, we have defined a new task called Fine-Grained Filling-Aware Impact Sound Generation. As a foundational step, we first introduce the fine-grained fill-aware dataset (FillImpact), a pioneering multi-modal collection comprising over 5,000 rigorous acoustic recordings from 88 diverse real-world objects. It captures impact interactions with varying internal contents (i.e., water, rice), a continuous range of fill levels, and distinct striker materials. Furthermore, comprehensive acoustic analysis confirms that the collected data closely aligns with established physical laws governing acoustic resonance and damping, indicating its suitability for physically grounded modeling. Building on this dataset, we propose a novel generative framework (FillGauss) that integrates 3D Gaussian Splatting (3DGS) with internal state conditioning for sound generation. By fusing 3DGS geometric features, precise 3D spatial strike coordinates, and fine-grained textual physical conditions within a latent diffusion architecture, FillGauss enables position-aware, striker-aware, and filling-aware audio generation. Extensive experiments demonstrate that our approach could generate high-fidelity impact sounds that adhere to underlying physical principles, establishing a new state-of-the-art for physically grounded cross-modal audio generation.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
Methodology
The paper proposes FillGauss, a framework for fine-grained, filling-aware impact sound generation using 3D Gaussian Splatting (3DGS). The methodology integrates three modalities: 3DGS geometric features, precise 3D strike coordinates, and textual descriptions of internal states (fill material and level). The core technical contribution lies in the cross-modal attention fusion mechanism, which entangles spatial strike context with internal physical semantics before conditioning a Latent Diffusion Transformer (DiT). The use of 3DGS as a geometric prior for audio generation is a novel application, moving beyond traditional mesh or point-cloud representations. However, the underlying diffusion architecture (TangoFlux) is a pre-existing model, and the primary innovation is in the conditioning pipeline and dataset construction rather than a new generative backbone. The approach of using text prompts to encode physical states (fill level, material) is effective but relies heavily on the semantic capabilities of the pre-trained language model, which may limit fine-grained control compared to continuous numerical conditioning.
Results & evaluation
The experimental setup is robust, centered on the introduction of the FillImpact dataset, which contains over 5,000 recordings from 88 objects with varying internal states. This dataset is a significant contribution, addressing a gap in existing literature where internal filling states are largely ignored. The evaluation includes objective metrics (FAD, KL, IS) and subjective human perceptual studies (MOS, Win Rate, Attribute Matching). The results demonstrate that FillGauss outperforms baselines (SonicGauss, TangoFlux) in generating physically plausible sounds, particularly in capturing the acoustic differences between liquid and granular fills. The ablation studies effectively validate the importance of the cross-attention modules and the text-prompting strategy. The sensitivity analysis regarding 3DGS sparsity provides valuable insights into the model's reliance on geometric detail.
Reproducibility
The paper provides detailed descriptions of the data collection pipeline, including the use of specific hardware (DJI Mic, Revo Scan) and software (Hunyuan 3D, Stable Audio Open VAE). The normalization procedures for 3DGS and strike coordinates are clearly defined. The training details, including the use of Huber loss and the specific architecture of the DiT, are described. However, as the code and dataset are not explicitly linked in the provided text (URLs are "none"), reproducibility depends on the authors' future release of these assets. The description of the dataset construction is sufficiently detailed for another group to replicate the data collection process.
Limitations
The paper acknowledges that extreme sparsification of 3DGS degrades performance, indicating a dependency on high-fidelity geometric reconstruction. The model's ability to handle complex internal dynamics like sloshing fluids or soft-body deformations is limited to static fill levels and materials. The reliance on text prompts for internal state conditioning might introduce semantic ambiguities or lack the precision of direct numerical inputs for continuous variables like fill level. Furthermore, the subjective evaluation, while rigorous, is limited by the number of participants (20) and the specific test set.
Broader impact
This work significantly advances the field of physically grounded multimodal AI, particularly for applications in virtual reality, gaming, and robotic simulation where accurate auditory feedback is crucial for immersion and task performance. By explicitly modeling internal states, it enables more realistic interactions with a wider variety of objects. The FillImpact dataset serves as a valuable resource for future research in acoustic physics and multimodal learning. The approach could be extended to other physical properties, such as temperature or material degradation, further enriching the realism of synthetic environments. This paper introduces FillGauss, a novel framework for fine-grained, filling-aware impact sound generation that leverages 3D Gaussian Splatting and latent diffusion to synthesize physically plausible audio conditioned on internal object states, supported by the pioneering FillImpact dataset.
Author of this paper? Send a correction.