Fun-Audio-Chat Technical Report
Abstract
Recent advancements in joint speech-text models show great potential for seamless voice interactions. However, existing models face critical challenges: temporal resolution mismatch between speech tokens (25Hz) and text tokens (~3Hz) dilutes semantic information, incurs high computational costs, and causes catastrophic forgetting of text LLM knowledge. We introduce Fun-Audio-Chat, a Large Audio Language Model addressing these limitations via two innovations from our previous work DrVoice. First, Dual-Resolution Speech Representations (DRSR): the Shared LLM processes audio at efficient 5Hz (via token grouping), while the Speech Refined Head generates high-quality tokens at 25Hz, balancing efficiency (~50% GPU reduction) and quality. Second, Core-Cocktail Training, a two-stage fine-tuning with intermediate merging that mitigates catastrophic forgetting. We then apply Multi-Task DPO Training to enhance robustness, audio understanding, instruction-following and voice empathy. This multi-stage post-training enables Fun-Audio-Chat to retain text LLM knowledge while gaining powerful audio understanding, reasoning, and generation. Unlike recent LALMs requiring large-scale audio-text pre-training, Fun-Audio-Chat leverages pre-trained models and extensive post-training. Fun-Audio-Chat 8B and MoE 30B-A3B achieve competitive performance on Speech-to-Text and Speech-to-Speech tasks, ranking top among similar-scale models on Spoken QA benchmarks. They also achieve competitive to superior performance on Audio Understanding, Speech Function Calling, Instruction-Following and Voice Empathy. We develop Fun-Audio-Chat-Duplex, a full-duplex variant with strong performance on Spoken QA and full-duplex interactions. We open-source Fun-Audio-Chat-8B with training and inference code, and provide an interactive demo.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The Fun-Audio-Chat model represents a significant advancement in voice interaction systems, effectively addressing critical challenges in multimodal learning and demonstrating strong performance across various tasks. The innovative methodologies and comprehensive evaluations contribute to its potential impact in the field of machine learning and natural language processing.
Methodology
The methodology presented in Fun-Audio-Chat is innovative, particularly with the introduction of Dual-Resolution Speech Representations (DRSR) and Core-Cocktail Training. The DRSR architecture effectively addresses the temporal resolution mismatch between speech and text tokens, enhancing both computational efficiency and output quality. The Core-Cocktail Training strategy mitigates catastrophic forgetting, a common challenge in multimodal models. The multi-stage post-training approach, including Multi-Task DPO Training, further enhances the model's robustness and capability in real-world applications. Overall, the methodology is well-structured and demonstrates a clear advancement over existing models.
Results & evaluation
The experiments conducted are comprehensive, utilizing a variety of benchmarks for evaluation, including Spoken Question Answering, Audio Understanding, and Speech Function Calling. The results indicate that Fun-Audio-Chat achieves competitive to superior performance compared to existing models, particularly in the 8B and 30B parameter ranges. The extensive evaluation across multiple tasks provides strong evidence of the model's capabilities, although specific details on the experimental setup and statistical significance of the results could enhance the rigor of the findings.
Reproducibility
The paper includes links to the open-source model and provides training and inference code, which supports reproducibility. However, detailed hyperparameter settings and training configurations are not fully disclosed, which could hinder the ability of other researchers to replicate the results precisely. The availability of an interactive demo is a positive aspect, allowing users to experience the model's capabilities firsthand.
Limitations
The paper acknowledges several limitations, including issues with long-context comprehension in multi-turn conversations and variability in the model's expressiveness and empathy capabilities. These limitations highlight areas for future research, particularly in enhancing the model's ability to maintain context over extended interactions and improving the emotional responsiveness of generated speech.
Broader impact
Fun-Audio-Chat has significant implications for the development of advanced voice interaction systems, potentially improving user experiences in applications such as virtual assistants, customer service, and interactive entertainment. The model's capabilities in understanding and generating empathetic responses could lead to more natural and engaging human-computer interactions, fostering greater acceptance and reliance on AI technologies in everyday life. The Fun-Audio-Chat model represents a significant advancement in voice interaction systems, effectively addressing critical challenges in multimodal learning and demonstrating strong performance across various tasks. The innovative methodologies and comprehensive evaluations contribute to its potential impact in the field of machine learning and natural language processing.
Author of this paper? Send a correction.