MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model
Abstract
As an increasing majority of global video content is consumed on social platforms for interactive social purposes, video generation models built for social worlds are important but largely overlooked by previous studies. In this work, we define the position of social world models and build a prototype model as the first step towards this goal. While previous world models successfully simulate physical environments or gaming world exploration, they remain fundamentally detached from human-centric social dynamics. To bridge this gap as the first step to social world models, we present MaineCoon, the first real-time audio-visual autoregressive model that has 22B parameters and is capable of real-time streaming generation and sub-second interaction, with a record-breaking frame rate of up to 47.5 FPS, on a single GPU. To the best of our knowledge, MaineCoon is also the first real-time audio-visual generation model specifically optimized for social-interactive applications. To enable efficient and stable training, we introduce several novel techniques into MaineCoon, including self-resampling, cross-modal representation alignment, domain-aware preference optimization, and reinforced online-policy distillation (ROPD). We also design the first agentic streaming inference framework that supports thousand-second-scale or even longer generation while mitigating drift with agentic cache management and prompt planing. These innovations significantly accelerate training while optimizing real-time inference performance. We believe this work not only sets a new state-of-the-art (SOTA) performance benchmark for high-quality, low-latency, and long-horizon audio-visual autoregressive models, but also points out the paradigm shift desired for next-generation AI-native social platforms.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
MaineCoon presents a comprehensive engineering solution for real-time audio-visual autoregressive generation, effectively combining self-resampling, representation alignment, and reinforced policy distillation to achieve stable, long-horizon social video synthesis at high frame rates, marking a significant step toward practical AI-native social platforms.
Methodology
The paper proposes MaineCoon, a 22B-parameter autoregressive audio-visual model designed for real-time social interaction. The core methodological contributions lie in the training pipeline and inference framework rather than a novel architectural backbone. The authors introduce "self-resampling" to mitigate the train-test discrepancy in autoregressive generation by exposing the model to its own degraded histories during training. They also employ representation alignment (REPA) using a frozen V-JEPA teacher to accelerate convergence and improve semantic coherence. A significant portion of the technical depth is dedicated to post-training strategies: Domain-Aware Preference Optimization (DPO) to specialize the model in difficult social domains (e.g., wide shots, multi-person dialogue) and Reinforced Online-Policy Distillation (ROPD) to consolidate these specialized experts into a single streaming policy. The inference framework is agentic, utilizing a planner/observer LLM to manage prompt continuity and a cache manager to mitigate long-horizon drift. While the integration of these components is sophisticated, the individual techniques (self-resampling, REPA, DPO, ROPD) are adaptations of existing methods to the specific constraints of real-time audio-visual autoregressive generation. The novelty is moderate, lying primarily in the specific application and engineering orchestration rather than fundamental algorithmic breakthroughs.
Results & evaluation
The experimental evaluation focuses on demonstrating the feasibility of real-time generation (47.5 FPS on a single H100) and long-horizon stability. The authors introduce "SocialVideo Bench," a new benchmark tailored for social video content, evaluating visual quality, motion, audio quality, and audio-visual alignment. They compare MaineCoon against 7 representative open audio-visual generation models, claiming SOTA performance in terms of speed and quality metrics. The evaluation includes ablation studies on the training components (self-resampling, alignment, ROPD). However, the paper lacks extensive human preference studies (e.g., MOS comparisons) which are critical for social video quality assessment. The reliance on automated metrics (CLIP, SyncNet, etc.) is standard but less convincing for subjective qualities like "social harmony" or "emotional resonance." The claim of "record-breaking" performance is supported by the FPS metric, but the quality comparisons are somewhat qualitative or based on limited automated scores. The long-horizon evaluation (thousand-second scale) is a strong point, demonstrating the efficacy of the agentic cache management.
Reproducibility
The paper provides detailed descriptions of the data pipeline, including filtering criteria, domain balancing, and the synthetic data generation process. The training infrastructure details (FSDP2, sequence parallelism, mixed precision) are well-specified. The mathematical formulations for self-resampling, REPA, and ROPD are clear. However, the code and model weights are not mentioned as being publicly available (URLs are "none"), which hinders immediate reproducibility. The reliance on proprietary teacher models (LTX-2.3, Gemini-3.1-flash) for data generation and evaluation adds a layer of dependency that might make exact replication difficult for independent researchers.
Limitations
The paper acknowledges that the model is a "first step" and focuses on the generative core. Limitations include the potential for residual drift in very long generations despite the agentic repair mechanisms, the high computational cost of the pre-training and post-training phases (though claimed to be efficient relative to scale), and the dependency on high-quality social video data which may contain biases. The model is specialized for "social" videos (talking heads, single speakers), limiting its generalizability to cinematic or complex physical interaction scenarios. The "agentic" nature of the inference introduces latency variability depending on the LLM planner's speed, although the paper claims sub-second interaction.
Broader impact
This work has significant implications for the development of AI-native social platforms, virtual influencers, and interactive entertainment. By enabling real-time, long-horizon audio-visual generation, it lowers the barrier for creating interactive AI agents. However, it also raises concerns about deepfakes, misinformation, and the ethical use of AI-generated social interactions. The ability to generate realistic human speech and video in real-time could be misused for impersonation or harassment. The authors' focus on "social world models" points to a future where AI is an active participant in social dynamics, which has profound philosophical and societal implications. MaineCoon presents a comprehensive engineering solution for real-time audio-visual autoregressive generation, effectively combining self-resampling, representation alignment, and reinforced policy distillation to achieve stable, long-horizon social video synthesis at high frame rates, marking a significant step toward practical AI-native social platforms.
Author of this paper? Send a correction.