CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
Abstract
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
One sentence main contribution: CrossEdit demonstrates that complex instruction-following for audio-visual editing can be acquired zero-shot through cross-modal transfer from synthetic audio data and self-supervised AV masked reconstruction. The paper presents a compelling argument for the modality-agnostic nature of instruction following, supported by rigorous experiments and a new benchmark. The use of synthetic audio data to bootstrap AV editing is a novel and effective strategy, addressing the data scarcity problem in multimodal editing. The introduction of AV-FES provides a more holistic evaluation metric for editing tasks. While the lack of open-source release is a drawback, the technical insights and results are highly relevant to the development of future multimodal generative systems.
Methodology
The paper proposes CrossEdit, a unified omni-modal editing model based on a DiT-MoE backbone (3B active, 12B total parameters) using flow matching. The core methodological contribution is the demonstration that complex instruction-following capabilities are modality-agnostic and can be transferred from audio to visual/AV domains. This is achieved by exploiting the physical additivity of acoustic signals to procedurally generate large-scale synthetic audio editing pairs with compositional instructions. Additionally, the authors introduce AV masked reconstruction as a self-supervised task to teach the model to generate new visual content consistent with existing AV scenes. The architecture leverages pre-trained VAEs for audio and video, along with Qwen3-VL for semantic embeddings. The approach is technically sound, leveraging the asymmetry between auditory (additive) and visual (non-additive) signals to overcome data scarcity in AV editing.
Results & evaluation
The evaluation is comprehensive, introducing CrossEditBench, a human-annotated benchmark of 100 movie scene clips, and AV-FES, a novel metric combining instruction following and consistency via a harmonic mean. Results show CrossEdit outperforming concurrent end-to-end models (MiniMax H3, InstructAV2AV) and cascaded systems on zero-shot AV scene editing. Ablations confirm that synthetic audio data improves instruction following across modalities, while AV masked reconstruction improves visual consistency and synchronization. The model also maintains or improves performance on lip-synced speech, video, and image editing benchmarks. The use of multiple LLM judges (Gemini, Qwen) and human ratings provides robust validation, though the small sample size for human evaluation (100 clips) is a minor limitation.
Reproducibility
The paper provides detailed descriptions of the architecture, training data mining pipelines, and training hyperparameters. However, the authors explicitly state that code and pre-trained checkpoints will not be released due to deepfake risks and proprietary data. While they commit to releasing the benchmark and annotations, the lack of open-source code and weights significantly limits full reproducibility for the broader community, despite the detailed technical description.
Limitations
The primary limitation is the lack of open-source release of code and models, which hinders independent verification and adoption. The evaluation is limited to short video clips (up to 10 seconds), and the model has not been tested on noisy, real-world home videos. The AV-FES metric relies on LLM judges, which may have inherent biases, although correlations with human ratings are reported. The synthetic audio data, while effective for transfer, may not fully capture the complexity of real-world AV interactions.
Broader impact
The work has significant implications for multimedia editing, enabling users to perform complex, compositional edits on AV content with natural language instructions. This could lower the barrier to entry for creative professionals and reduce production costs. However, the capability to generate indistinguishable lip-synced speech and video raises serious ethical concerns regarding deepfakes, fraud, and misinformation. The authors acknowledge these risks and justify their decision not to release the models, but the potential for misuse remains a critical concern for the field. One sentence main contribution: CrossEdit demonstrates that complex instruction-following for audio-visual editing can be acquired zero-shot through cross-modal transfer from synthetic audio data and self-supervised AV masked reconstruction. The paper presents a compelling argument for the modality-agnostic nature of instruction following, supported by rigorous experiments and a new benchmark. The use of synthetic audio data to bootstrap AV editing is a novel and effective strategy, addressing the data scarcity problem in multimodal editing. The introduction of AV-FES provides a more holistic evaluation metric for editing tasks. While the lack of open-source release is a drawback, the technical insights and results are highly relevant to the development of future multimodal generative systems.
Author of this paper? Send a correction.