MMAE: A Massive Multitask Audio Editing Benchmark

Ziyang Ma, Ruiqi Yan, Ruiyang Xu ...Full author list in the original paper.

Shanghai Jiao Tong University +8 · arXiv

Affiliations: Shanghai Jiao Tong University, Shanghai Innovation Institute, Nanyang Technological University, Hunyuan Team, Tencent, Tianjin University, ZODA, Peking University, Fudan University

Paper

Abstract

We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose instruction-based audio editing. Spurred by the shift toward intelligent creation, interactive editing has rapidly expanded from visual domains, pioneered by models like Nano-banana 2 for images and Gemini-Omni for video, into audio. However, the current evaluation infrastructure lags severely, remaining highly fragmented and restricted to specific subdomains or basic operations. Unlike existing benchmarks that are limited in scope, MMAE extends to a broad spectrum of real-world scenarios, encompassing 7 distinct audio modalities, including sound, speech, music, and their mixtures. Furthermore, we establish a comprehensive taxonomy spanning 6 levels of task complexity, from basic modifications to multi-hop reasoning and multi-round editing, 2 levels of granularity, and 8 distinct operation types. Meticulously curated through human-agent collaboration, MMAE comprises 2,000 high-fidelity samples paired with a pioneering rubric-based evaluation framework. By decomposing free-form tasks into 17,741 verifiable criteria, this robust rubric-based paradigm enables a precise, multi-dimensional assessment of both instruction following and context consistency. Our extensive evaluation of leading models reveals that current systems remain far from achieving reliable edits. Strikingly, the Exact Match Rate (EMR) consistently falls below 5% and plummets to an absolute 0% in complex, mixed-modality tasks, exposing critical bottlenecks in precise execution and structural robustness. We hope MMAE will serve as a catalyst for future advances in the intelligent creation community, providing a clear diagnostic roadmap and establishing a standardized, long-lasting evaluation paradigm for next-generation audio editing systems.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper benchmarks five leading audio editing models (Step-Audio-EditX, Ming-UniAudio, MMEdit, Audio-Omni, SmartDJ) along with two baselines (Identity, Noise). The inclusion of baselines provides essential context for interpreting model performance. A notable detail is the acknowledgment of input length constraints for some models, leading to evaluation on a subset of samples (801 out of 2,000) for MMEdit, Audio-Omni, and SmartDJ, while others are evaluated on the full set. This pragmatic approach ensures fair comparison given current model limitations. The results are striking and provide clear diagnostic insights. The consistently low Exact Match Rate (EMR) across all models (below 5%, and 0% for complex mixed-modality tasks) unequivocally demonstrates that current systems are far from achieving reliable, flawless edits. This highlights a significant challenge for the field. The detailed analysis by task complexity and modality reveals universal performance degradation for more complex and mixed-modality tasks, exposing a lack of structural robustness and cross-domain synchronization. The observed trade-off between Instruction Following Rate (IF

Methodology

MMAE introduces a meticulously designed benchmark for instruction-based audio editing, addressing a critical gap in the field. The core methodology revolves around a comprehensive, multi-dimensional taxonomy and a novel rubric-based evaluation paradigm. The taxonomy is a strong point, categorizing tasks along three orthogonal dimensions: Modality (7 types, including mixed modalities like sound-music-speech), Complexity (6 levels from 'Single' to 'Multi-hop' and 'Multi-round'), and Operation (8 types, local and global edits). This systematic classification allows for fine-grained analysis of model capabilities across a vast spectrum of real-world scenarios, moving beyond the fragmented and restricted scope of previous benchmarks. The distributions across these dimensions are presented, indicating a diverse dataset. The rubric-based evaluation paradigm is particularly innovative and robust for open-ended generative tasks. It decomposes free-form editing tasks into 17,741 atomic, verifiable criteria, assessed along two crucial dimensions: Instruction Following (IF) and Consistency (CR). The four principles guiding rubric design—Completeness, Atomicity, Orthogonality, and Objectivity—are well-articulated and essential for ensuring the reliability and interpretability of the evaluation. Using an external, high-performance audio language model (Qwen3-Omni) as a judger, with multiple queries and majority voting, adds a layer of objectivity and scalability that traditional human MOS ratings often lack for such complex tasks. The Exact Match Rate (EMR) serves as a stringent, holistic metric for perfect execution. The data curation pipeline is rigorous, involving five stages from expert brainstorming to iterative quality inspection. The human-agent collaborative annotation workflow, leveraging an agentic pipeline (Omni-Detective) for detailed audio captions and LLMs for initial rubric drafts, followed by human refinement, is a pragmatic approach to balance efficiency and quality for a large-scale, complex dataset. This hybrid approach is crucial for generating high-fidelity samples and rubrics.

Results & evaluation

Author of this paper? Send a correction.