MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
Abstract
Universal sound separation faces a fundamental misalignment: models optimized for low-level signal metrics often produce semantically contaminated outputs, failing to suppress perceptually salient interference from acoustically similar sources. To bridge this gap, we introduce MARS-Sep, a reinforcement learning framework that reformulates separation as decision making. Instead of simply regressing ground-truth masks, MARS-Sep learns a factorized Beta mask policy that is optimized by a clipped trust-region surrogate with entropy regularization and group-relative advantage normalization. Concretely, we sample masks from a frozen old policy, reconstruct waveforms, and update the current policy using clipped importance ratios-yielding substantially more stable and sample-efficient learning. Multimodal rewards, derived from an audio-text-vision encoder, directly incentivize semantic consistency with query prompts. We further propose a progressive alignment scheme to fine-tune this encoder, boosting its cross-modal discriminability and improving reward faithfulness. Extensive experiments on multiple benchmarks demonstrate consistent gains in Text-, Audio-, and Image-Queried separation, with notable improvements in signal metrics and semantic quality. Our code is available at https://anonymous.4open.science/r/MARS-Sep. Sound separation samples are available at https://mars-sep.github.io/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of MARS-Sep, a multimodal-aligned reinforced sound separation framework that significantly improves the semantic quality of separated audio through innovative reinforcement learning techniques. This work represents a meaningful advancement in the field of audio processing, addressing critical challenges in sound separation and offering a robust methodology for future research.
Methodology
The paper introduces MARS-Sep, a novel reinforcement learning framework that reformulates sound separation as a decision-making problem, optimizing a factorized Beta mask policy. The use of multimodal rewards derived from an audio-text-vision encoder is innovative, as it directly incentivizes semantic consistency with user queries. The progressive alignment strategy for fine-tuning the encoder enhances cross-modal discriminability, which is a significant methodological advancement. The approach effectively addresses the limitations of traditional signal-level metrics by incorporating semantic fidelity into the training process.
Results & evaluation
The experiments are comprehensive, utilizing multiple benchmarks (VGGSound and MUSIC) to validate the effectiveness of MARS-Sep. The results demonstrate consistent improvements across various metrics, including SDR, SIR, SAR, and CLAP scores, indicating that the proposed method not only enhances signal quality but also improves semantic alignment. The rigorous evaluation against strong baselines provides a solid foundation for the claims made in the paper.
Reproducibility
The paper provides detailed implementation information, including training parameters, dataset descriptions, and evaluation metrics. The availability of code and audio samples enhances reproducibility, although the lack of specific institutional affiliations may limit the accessibility of the research for some readers.
Limitations
One limitation is the potential complexity of the proposed method, which may hinder its application in real-time scenarios or on devices with limited computational resources. Additionally, while the paper addresses the metric dilemma, the subjective nature of semantic quality could lead to variability in user experiences across different contexts.
Broader impact
The advancements in sound separation have significant implications for various applications, including speech recognition, sound event detection, and augmented reality. By improving the semantic fidelity of separated audio, MARS-Sep can enhance user experiences in multimedia applications and contribute to the development of more robust audio processing systems. The main contribution of this paper is the introduction of MARS-Sep, a multimodal-aligned reinforced sound separation framework that significantly improves the semantic quality of separated audio through innovative reinforcement learning techniques. This work represents a meaningful advancement in the field of audio processing, addressing critical challenges in sound separation and offering a robust methodology for future research.
Author of this paper? Send a correction.