Resonate: Reinforcing Text-to-Audio Generation via Online Feedback from Large Audio Language Models
Abstract
Reinforcement Learning (RL) has become an effective paradigm for enhancing Large Language Models (LLMs) and visual generative models. However, its application in text-to-audio (TTA) generation remains largely under-explored. Prior work typically employs offline methods like Direct Preference Optimization (DPO) and leverages Contrastive Language-Audio Pretraining (CLAP) models as reward functions. In this study, we investigate the integration of online Group Relative Policy Optimization (GRPO) into TTA generation. We adapt the algorithm for Flow Matching-based audio models and demonstrate that online RL significantly outperforms its offline counterparts. Furthermore, we incorporate rewards derived from Large Audio Language Models (LALMs), which can provide fine-grained scoring signals that are better aligned with human perception. With only 470M parameters, our final model, \textbf{Resonate}, establishes a new SOTA on TTA-Bench in terms of both audio quality and semantic alignment.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the introduction of Resonate, a novel text-to-audio generator that employs online reinforcement learning and LALMs to achieve state-of-the-art performance in audio quality and semantic alignment. This work represents a significant step forward in the field of generative audio models, combining innovative methodologies with rigorous experimental validation to address existing limitations in TTA generation.
Methodology
The paper presents a novel integration of online reinforcement learning (RL) into text-to-audio (TTA) generation, specifically through the Group Relative Policy Optimization (GRPO) framework. This approach addresses the limitations of offline RL methods by enabling more dynamic and responsive training that aligns better with human preferences. The use of Large Audio Language Models (LALMs) as reward models is particularly innovative, as it allows for fine-grained feedback that enhances the model's performance. The architecture leverages a Flux-style flow Transformer, which is well-suited for the generative tasks at hand. Overall, the methodology is robust, well-structured, and demonstrates a clear advancement over previous techniques.
Results & evaluation
The experiments are comprehensive, utilizing a large-scale audio-text dataset for pre-training and a well-defined evaluation benchmark (TTA-Bench) for assessing model performance. The results indicate that the proposed Resonate model achieves state-of-the-art performance in both audio quality and semantic alignment, outperforming existing models across various metrics. The inclusion of both objective and subjective evaluation methods strengthens the findings, providing a balanced view of the model's capabilities. The ablation studies further validate the effectiveness of the proposed methods, highlighting the advantages of online RL and LALM-based rewards.
Reproducibility
The authors have provided clear details regarding the model architecture, training procedures, and evaluation metrics, which supports reproducibility. The availability of code and model weights on GitHub enhances this aspect, allowing other researchers to replicate the study and build upon the work. However, specific hyperparameter settings and the rationale behind certain design choices could be elaborated further to aid in complete reproducibility.
Limitations
One limitation is the reliance on the quality of the datasets used for training and evaluation, which may affect the generalizability of the results. Additionally, while the model achieves state-of-the-art performance, the computational efficiency and scalability of the approach in real-world applications could be further explored. The paper does not address potential biases in the training data or the implications of using LALMs as reward models.
Broader impact
The advancements presented in this paper have significant implications for various applications, including automated content creation in filmmaking, gaming, and virtual reality. The integration of RL and LALMs in TTA generation could lead to more intuitive and human-aligned audio synthesis, enhancing user experiences across multimedia platforms. Furthermore, the open-sourcing of the model and code promotes collaboration and innovation within the research community. The main contribution of this paper is the introduction of Resonate, a novel text-to-audio generator that employs online reinforcement learning and LALMs to achieve state-of-the-art performance in audio quality and semantic alignment. This work represents a significant step forward in the field of generative audio models, combining innovative methodologies with rigorous experimental validation to address existing limitations in TTA generation.
Author of this paper? Send a correction.