Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis
Abstract
Deep learning-based respiratory auscultation is currently hindered by two fundamental challenges: (i) inherent information loss, as converting signals into spectrograms discards transient acoustic events and clinical context; (ii) limited data availability, exacerbated by severe class imbalance. To bridge these gaps, we present Resp-Agent, an autonomous multimodal system orchestrated by a novel Active Adversarial Curriculum Agent (Thinker-A$^2$CA). Unlike static pipelines, Thinker-A$^2$CA serves as a central controller that actively identifies diagnostic weaknesses and schedules targeted synthesis in a closed loop. To address the representation gap, we introduce a Modality-Weaving Diagnoser that weaves EHR data with audio tokens via Strategic Global Attention and sparse audio anchors, capturing both long-range clinical context and millisecond-level transients. To address the data gap, we design a Flow Matching Generator that adapts a text-only Large Language Model (LLM) via modality injection, decoupling pathological content from acoustic style to synthesize hard-to-diagnose samples. As a foundation for these efforts, we introduce Resp-229k, a benchmark corpus of 229k recordings paired with LLM-distilled clinical narratives. Extensive experiments demonstrate that Resp-Agent consistently outperforms prior approaches across diverse evaluation settings, improving diagnostic robustness under data scarcity and long-tailed class imbalance. Our code and data are available at https://github.com/zpforlove/Resp-Agent.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Resp-Agent, a multimodal framework that effectively synthesizes respiratory sounds and integrates clinical context for improved disease diagnosis. This work represents a significant advancement in the application of machine learning to healthcare, particularly in addressing the challenges of data scarcity and class imbalance in respiratory sound analysis.
Methodology
The paper presents a novel agent-based framework, Resp-Agent, which integrates multimodal data (audio and EHR) for respiratory sound generation and diagnosis. The methodology is innovative, utilizing an Active Adversarial Curriculum Agent (Thinker-A$^2$CA) to dynamically identify weaknesses in diagnostics and schedule targeted synthesis. The Modality-Weaving Diagnoser and Flow Matching Generator are well-conceived to address the representation and data gaps in respiratory sound analysis. The use of large language models (LLMs) for generating clinical narratives and the careful design of the dataset (Resp-229k) enhance the robustness of the approach. However, while the methodology is sound, it heavily relies on the quality of the underlying data and the effectiveness of the LLMs used for synthesis.
Results & evaluation
The experiments are comprehensive, evaluating the proposed system against multiple benchmarks and comparing it with existing methods. The results demonstrate significant improvements in diagnostic accuracy and robustness, particularly in handling class imbalance and data scarcity. The use of a strict cross-domain evaluation protocol adds rigor to the assessment of generalization capabilities. The paper also includes detailed ablation studies that validate the contributions of various components of the system, further strengthening the findings.
Reproducibility
The authors have made significant efforts to ensure reproducibility by providing access to the code and dataset. The detailed descriptions of the architecture, training procedures, and evaluation metrics contribute to the transparency of the research. However, the reliance on LLMs and the complexity of the system may pose challenges for complete replication without adequate computational resources.
Limitations
One limitation is the dependency on the quality and diversity of the Resp-229k dataset, which may affect the generalizability of the findings. Additionally, while the paper addresses class imbalance, the performance on extremely rare conditions may still be limited. The complexity of the system could also hinder its practical deployment in clinical settings without further validation.
Broader impact
The proposed framework has the potential to significantly advance the field of respiratory sound analysis and diagnosis, offering a robust tool for clinicians to improve diagnostic accuracy and support medical education. The integration of generative modeling with diagnostic capabilities could lead to more effective training datasets and enhance the understanding of respiratory diseases. However, ethical considerations regarding the use of AI in clinical decision-making must be addressed. The main contribution of this paper is the introduction of Resp-Agent, a multimodal framework that effectively synthesizes respiratory sounds and integrates clinical context for improved disease diagnosis. This work represents a significant advancement in the application of machine learning to healthcare, particularly in addressing the challenges of data scarcity and class imbalance in respiratory sound analysis.
Author of this paper? Send a correction.