SelectTSL: Prompt-Guided Selective Target Sound Localization in Complex Scenarios
Abstract
Humans can selectively attend to a target sound and estimate its direction in complex scenarios, whereas such selective localization remains challenging for current deep learning-based systems. Sound source localization (SSL) has achieved remarkable success with deep learning, yet most methods localize all active sources without selectivity. Conversely, target sound extraction (TSE) extracts sources using multimodal prompts but typically fails to preserve the multichannel spatial information required for accurate localization. To bridge this gap, we formulate the task of prompt-guided selective target sound localization and propose SelectTSL, an end-to-end architecture that localizes only the user-specified target in multi-source acoustic scenes. Specifically, we design a target-aware selective localization strategy that employs a Prompt-Guided Selective Attention Module (PGSA) to generate prompt-informed embeddings. These embeddings guide an inter-channel phase difference (IPD) enhancer to refine raw phase cues, fusing with target magnitudes to jointly estimate direction of arrival (DoA) and target-source cardinality, i.e., the number of target sound sources. This coupled design effectively focuses on the user-specified target spatial cues for selective localization and also handles time-varying numbers of target sources. Extensive experiments on both synthetic data and real-world recordings demonstrate that our proposed method consistently outperforms other baselines and exhibits robust generalization to real acoustic environments.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces SelectTSL, an end-to-end framework for prompt-guided selective target sound localization that jointly estimates DoA and source cardinality using prompt-informed spatial cue refinement. The proposed method effectively addresses the challenge of localizing specific sound sources in complex multi-source environments by integrating semantic guidance from multimodal prompts with precise spatial cue enhancement. The novel combination of a PGSA module for target extraction and an IPD enhancer conditioned on extraction-informed embeddings allows for robust and selective localization, outperforming existing methods in both static and dynamic metrics. The joint prediction of DoA and cardinality provides a flexible solution for handling varying numbers of active sources, making it a significant advancement in the intersection of target sound extraction and sound source localization.
Methodology
The paper proposes SelectTSL, a novel framework for prompt-guided selective target sound localization (SSL). The core innovation lies in decoupling semantic selection from spatial estimation. It employs a Prompt-Guided Selective Attention (PGSA) module that uses multimodal prompts (text or audio) to generate extraction-informed embeddings (EIEs). These EIEs condition an Inter-Channel Phase Difference (IPD) enhancer to refine spatial cues, which are then fused with target magnitudes for Direction of Arrival (DoA) estimation. A key technical contribution is the joint prediction of DoA posteriorgrams and source cardinality, allowing the system to handle time-varying numbers of active target sources without relying on fixed track slots. The architecture integrates a DPRNN-based extraction network with a specialized DoA estimator using depthwise-separable convolutions and temporal modeling (TCN/BiGRU). The approach effectively bridges the gap between Target Sound Extraction (TSE) and SSL, addressing the "cocktail party problem" by enabling users to specify *which* source to localize.
Results & evaluation
The authors conduct extensive experiments on a large-scale synthetic dataset (288.9 hours of training data) generated using dynamic Room Impulse Responses (RIRs) and real-world recordings from the TAU-SRIR dataset. They compare SelectTSL against a comprehensive taxonomy of baselines, including track-wise SSL methods (IPDNet, EINV2), SELD methods (SELDnet, SELDT), pure DoA methods (SRP-DNN, FN-SSL), and prompt-based localization (SEL). Results demonstrate that SelectTSL significantly outperforms all baselines, achieving an MAE of 0.98° and a MOTA of 91.57% on the synthetic test set, compared to the next best prompt-based method (SEL) which achieved 2.78° MAE and 16.25% MOTA. The paper includes detailed ablation studies validating the contributions of the PGSA module, IPD enhancement, cardinality head, and training scheme. The robustness to noise and reverberation is also evaluated.
Reproducibility
The paper provides detailed implementation details, including STFT parameters, network architecture dimensions, loss function weights, and training hyperparameters (Adam optimizer, learning rate, early stopping). The dataset generation process is clearly described, including room dimensions, RIR simulation tools (GPURIR), and source/noise datasets. The authors state that the code and dataset will be released, which enhances reproducibility. The synthetic data generation protocol is sufficiently detailed for replication.
Limitations
The primary limitation is the reliance on synthetic data for training and primary evaluation. While real-world evaluation is performed, it is limited to a subset of TAU-SRIR rooms with fixed source positions (simulated movement via static RIRs), which may not fully capture the complexities of real-time moving source localization with dynamic acoustic changes. The dual-microphone setup limits the spatial resolution and front-back ambiguity handling compared to larger arrays. The method's performance on highly complex, multi-source scenarios with more than two concurrent targets is not explicitly evaluated (cardinality head is limited to 0, 1, or 2).
Broader impact
This work has significant implications for human-computer interaction, particularly in smart speakers, hearing aids, and robotics, where selective auditory attention is crucial. By enabling users to specify a target sound via text or audio, the system can improve speech enhancement, far-field ASR, and spatial audio perception in noisy environments. It advances the field of audio AI by integrating semantic understanding with precise spatial sensing. This paper introduces SelectTSL, an end-to-end framework for prompt-guided selective target sound localization that jointly estimates DoA and source cardinality using prompt-informed spatial cue refinement. The proposed method effectively addresses the challenge of localizing specific sound sources in complex multi-source environments by integrating semantic guidance from multimodal prompts with precise spatial cue enhancement. The novel combination of a PGSA module for target extraction and an IPD enhancer conditioned on extraction-informed embeddings allows for robust and selective localization, outperforming existing methods in both static and dynamic metrics. The joint prediction of DoA and cardinality provides a flexible solution for handling varying numbers of active sources, making it a significant advancement in the intersection of target sound extraction and sound source localization.
Author of this paper? Send a correction.