Source-linked AI summary
MaineCoon: Pursuing A Real-Time Audio-Visual Social World Model
Lichen Bai, Tianhao Zhang, Shitong Shao, Dingwei Tan, Qiyu Zhong, Zhengpeng Xie, Haopeng Li, Qinghao Huang, Dandan Shen, Tengjiao Ji, Wei Wang, Peicheng Wu, Yuxuan Zhao, Xiangyu Zhu, Welly Luo, Shurui Yang, Zeke Xie
TL;DR
Social world models need low-latency audio-visual generation for human-centric interaction, but existing video generators do not fundamentally support causal streaming or continuous user interaction. MaineCoon addresses this gap with a 22B-parameter streaming autoregressive model and significantly outperforms seven open audio-visual generation models in quality and speed.
Problem
Existing video generators remain limited in enabling causal streaming generation and continuous interaction for human-centric social worlds.
Method
MaineCoon is a 22B-parameter audio-visual autoregressive model that generates synchronized audio-video chunks causally for sub-second interaction and real-time single-GPU deployment.
Results
MaineCoon significantly outperforms seven representative open audio-visual generation models in quality and inference speed, establishing a new state-of-the-art standard.
Takeaways & Limitations
The work establishes real-time social audio-visual generation as a first technical step toward practical social world models and AI-native social platforms.
Takeaways & Limitations
MaineCoon remains an isolated, half-duplex generation system, while complete social world models require persistent multi-user ecosystems and real-time full-duplex interaction.
Abstract
from arXiv · showhide
As an increasing majority of global video content is consumed on social platforms for interactive social purposes, video generation models built for social worlds are important but largely overlooked by previous studies. In this work, we define the position of social world models and build a prototype model as the first step towards this goal. While previous world models successfully simulate physical environments or gaming world exploration, they remain fundamentally detached from human-centric social dynamics. To bridge this gap as the first step to social world models, we present MaineCoon, the first real-time audio-visual autoregressive model that has 22B parameters and is capable of real-time streaming generation and sub-second interaction, with a record-breaking frame rate of up to 47.5 FPS, on a single GPU. To the best of our knowledge, MaineCoon is also the first real-time audio-visual generation model specifically optimized for social-interactive applications. To enable efficient and stable training, we introduce several novel techniques into MaineCoon, including self-resampling, cross-modal representation alignment, domain-aware preference optimization, and reinforced online-policy distillation (ROPD). We also design the first agentic streaming inference framework that supports thousand-second-scale or even longer generation while mitigating drift with agentic cache management and prompt planing. These innovations significantly accelerate training while optimizing real-time inference performance. We believe this work not only sets a new state-of-the-art (SOTA) performance benchmark for high-quality, low-latency, and long-horizon audio-visual autoregressive models, but also points out the paradigm shift desired for next-generation AI-native social platforms.
1 Introduction
The introduction frames social world models as an overlooked extension of world modeling for human-centric, interactive video, requiring real-time synchronized audio-visual generation. MaineCoon addresses this need with a 22B streaming autoregressive model, specialized training and inference techniques, and a dedicated benchmark.
- Motivation: Conventional video diffusion models are slow and expensive, while efficiency techniques do not fundamentally enable causal streaming generation or continuous user interaction.Their costs arise from repeated spatiotemporal processing, many denoising steps, and attention that scales with video length and resolution.
- Motivation: Social video therefore requires low-latency, long-horizon, synchronized multimodal generation that captures human expression, speech rhythm, emotional resonance, conversational pacing, and audience feedback.The introduction identifies these capabilities as central to social interaction rather than merely visual video quality.
- Motivation: Social world models target human-centric interaction—expression, synchronized speech and motion, conversation, emotional resonance, pacing, and reaction prediction—beyond physical or gaming-world simulation.They are presented as a path from passive content generation toward active participation in human social interaction.
- MaineCoon: 22B parameters define MaineCoon as a real-time social audio-visual autoregressive model that generates synchronized audio and video chunk by chunk under causal streaming, supporting sub-second interaction.Its design is organized around deployment-time streaming, including data infrastructure, training, attention, context distribution, and KV-cache usage.
- MaineCoon: 47.5 FPS on a single H100 GPU and audio-visual generation cost below $0.001 per second establish the model’s reported real-time efficiency.The contribution passage describes 47.5 FPS as record-breaking and says the cost continues to fall.
- Methods: The work combines self-resampling, cross-modal alignment, preference optimization, ROPD, agentic cache management, chunk commitment, long-context rollout, and prompt planning for efficient training and long-horizon generation.These techniques support 22B-scale streaming training and thousand-second-scale or longer generation while mitigating drift.
- Evaluation: SocialVideo Bench evaluates visual quality, motion, audio quality, audio-visual alignment, and social-video harmony across 9 representative metrics, with MaineCoon outperforming 7 open audio-visual models and achieving the fastest generation speed.The benchmark is introduced specifically for audio-visual generation in social videos.
2 Data
MaineCoon’s data infrastructure is designed around the liveness, human-centricity, audio-visual synchronization, and temporal coherence required for social-world modeling. It combines synthetic streaming-oriented supervision with curated real social video, domain balancing, and the SocialVideo Bench evaluation benchmark.
- Data principles: Social videos are prioritized over cinematic footage because they provide liveness and human-centric content that requires synchronized speech, lip motion, and facial expression.The dataset also targets temporal consistency and coherence over each clip’s duration.
- Data sources: The dataset combines synthetic audio-visual videos from a teacher model with real social-media videos, standardizing both into balanced training-ready representations.The shared representation includes clips with text, audio, and visual conditioning.
- Synthetic data: 225 scenes across ten thematic groups, 15 visual styles, and 12 camera shots define synthetic scenarios decomposed into three to four linked clips with fixed character identity.The clips form roughly twenty-second stories in which shot, action, dialogue, and sound advance across segments.
- Real data: Tens of millions of raw social-media videos are filtered with a speech-strict, person-centric cascade that retains single-shot windows showing a clearly visible person speaking on camera.Low-level screening rejects unusable footage and removes internal cuts before more costly speech and lip-sync processing.
- Balancing and evaluation: Rare and demanding domains such as wide-shot, high-motion, multi-person, and first-person clips are up-weighted, while the dominant static talking-head bulk is set aside for post-training.The infrastructure also introduces SocialVideo Bench as a domain-focused benchmark for audio-visual generation on representative social-video content.
3 Training
MaineCoon uses a forcing-free training framework that matches chunk-by-chunk causal streaming inference, combining native autoregressive training, self-resampling, representation alignment, domain-aware preference optimization, and ROPD. This framework addresses deployment context mismatch and consolidates heterogeneous social-video preferences into a streaming policy.
- Native streaming AR training: MaineCoon trains directly under the same chunk-by-chunk causal regime used at inference rather than distilling from a non-causal teacher.Its attention mask, chunk ordering, and KV-cache updates already match streaming inference.
- Training framework: The training recipe combines native streaming AR with self-resampling, pretrained-encoder representation alignment, domain-aware preference optimization, and reinforced online-policy distillation.These components are organized into four parts covering causal chunk generation, audio-visual correspondence, heterogeneous social-video domains, policy consolidation, and efficient streaming infrastructure.
- Self-resampling: Self-resampling exposes training to model-generated degraded histories, reducing the context-distribution gap between clean training histories and streaming deployment.Short autoregressive rollouts use the same causal interface and KV-cache as inference, with gradients stopped through the generated history.
- Domain-aware preference optimization and ROPD: Domain-aware optimization addresses conflicting quality criteria across social-video domains, including full-body stability, speaker identity and turn-taking, and coherent high-motion movement.ROPD combines candidate outcomes with domain-expert information, while its posterior weight is jointly determined by candidate outcome and the student’s empirical success rate.
- Step distillation: Nearly-lossless four-step distillation is obtained by combining DMD with two variants, while softened real-teacher predictions stabilize early step-distillation gradients.The resulting step-distilled weight bias can also be loaded into other posttrained MaineCoon checkpoints.
4 Agentic Streaming Inference
The training-free agentic streaming inference framework wraps frozen MaineCoon with controllers for planning, memory, and look-ahead pacing, enabling open-ended interactive generation while preserving continuity and correcting degradation forward. On one GPU, MaineCoon generates about 32 FPS versus 25 FPS playback, creating a generation lead for non-disruptive observation and interaction.
- Framework overview: Three agentic controllers separately govern prompt planning and observation, KV-cache memory, and generation look-ahead without retraining the frozen generator.The planner incorporates viewer interaction, the cache manager preserves stream context, and the buffer controller regulates generation relative to playback.
- Prompt planning and interaction: The planner and observer uses a locally deployed Gemma 4 26B mixture-of-experts model to author structured prompts, monitor the generation head, and incorporate viewer input.It maintains bounded planning history and spoken-line transcripts, while typed instructions and optional camera feeds enter at natural switch points.
- Cache management: A persistent KV-cache commits each chunk’s clean keys and values, retains a bounded keep-set, and preserves original positions so throughput remains constant over stream duration.The keep-set combines scene memory, anchors, recent chunks, subject anchors, and restored chunks, while bounded epoch slots keep positions within the trained range.
- Drift control: Drift corrections modify only cache-committed copies rather than published output, breaking feedback amplification while preserving the emitted stream.The framework detects appearance drift with a five-metric photometric score and semantic defects with periodic vision-language checks, then applies graded forward repair without hard resets.
5 Experiments
MaineCoon is evaluated on SocialVideo-Bench for audio-visual quality and single-GPU streaming throughput. It achieves the strongest aggregate quality, improves joint audio-visual metrics, and reaches up to 47.5 FPS without observable visual degradation.
- Benchmark: SocialVideo-Bench evaluates 700 prompts across seven social-video domains using two consecutive 10-second segments with updated conditioning.The protocol measures visual and audio quality, temporal consistency, audio-visual coherence, and changing-condition following over 20 seconds.
- Quality Results: MaineCoon achieves the best overall performance, with an aggregate score of 0.934 versus 0.895 for the strongest baseline, ranking first on six of nine metrics.Reported strengths include visual quality, audio quality, text-audio alignment, audio-video semantic consistency, AVH, and JAVIS.
- Quality Results: MaineCoon improves AVH from 0.291 to 0.308 and JAVIS from 0.247 to 0.272, corresponding to relative improvements of 5.8% and 10.1%.It also achieves the highest visual and audio quality scores, indicating improved joint consistency without sacrificing either modality’s fidelity.
- Quality Results: Despite using causal streaming rather than the bidirectional formulation of most competing text-to-audio-video models, MaineCoon surpasses bidirectional generators and specialized streaming avatar models in aggregate quality.The comparison spans visual fidelity, audio quality, semantic alignment, and cross-modal performance.
- Qualitative Results: Representative examples span dense speech, two-person interaction, music, emotional performance, dance, and social memes, exhibiting TikTok-style social-media nativity and realistic human portraits.The examples are presented in Figure 7 and throughout Section A.
- Efficiency Results: 47.5 FPS is achieved for 480P video on a single H100 GPU by increasing inference chunk size to 6, up from 31 FPS with training chunk size 2.The higher throughput has no observable degradation in visual quality and is the highest among evaluated systems.
6 Position and Outlook: Social World Models
Social world models shift world modeling from rigid physical environments toward non-deterministic, human-centric social physics represented through synchronized multimodal behavior. The proposed roadmap advances MaineCoon toward persistent multi-user ecosystems through real-time dual-system full-duplex interaction.
- Position of Social World Models: Social world models prioritize psychological, communicative, and interpersonal forces over the spatial geometry and physical dynamics emphasized by traditional world models.They model complex, non-deterministic behavioral laws governing human-centric interaction, termed social physics.
- Position of Social World Models: The formalization autoregressively predicts joint visual and acoustic states from historical outputs and real-time user perturbations.Visual tokens capture facial expressions, gestures, and framing, while acoustic tokens capture phonemes, tone, and ambient noise.
- Next Step Towards Social World Models: Real-Time Dual-System Full-Duplex Interaction is identified as the next key step beyond MaineCoon’s current baseline for real-time human-centric generation.The roadmap calls for evolution from isolated, half-duplex generation toward scalable, persistent multi-user ecosystems.
- Next Step Towards Social World Models: Full-duplex interaction must stream interlaced Audio, Video, Text, and Gesture inputs and outputs chunk-by-chunk without blocking.This avoids the turn-based constraint in which users wait passively while systems process and respond.
- Next Step Towards Social World Models: The dual-system architecture pairs a sub-second reactive System 1 with an asynchronous strategic System 2 for long-horizon planning and persona management.System 1 handles synchronization, denoising, backchanneling, viseme matching, and gestures; System 2 handles semantic comprehension, preference-cache management, and conversation guidance.
7 Conclusion
The conclusion frames social world models as an overlooked direction requiring real-time audio-visual generation for human-centric interaction. MaineCoon establishes a 22B-parameter, single-GPU prototype with sub-second interaction, up to 47.5 FPS streaming, and reported SOTA quality and speed.
- Conclusion: Social world models target human-centric interactive audio-visual simulation, requiring active user observation, internal social-dynamics simulation, and real-time reactions.Traditional world models instead focus on physical environments or gaming mechanics and remain decoupled from conversational flow, micro-expressions, and high-engagement pacing.
- Conclusion: 22B parameters define MaineCoon as a pioneering real-time audio-visual autoregressive model optimized for social-interactive applications and single-GPU generation and interaction.The conclusion presents it as the first step toward social world models.
- Conclusion: 47.5 FPS is MaineCoon’s record-breaking streaming frame rate on a single GPU, alongside sub-second interaction enabled by its forcing-free streaming training paradigm.The paradigm integrates self-resampling, cross-modal representation alignment, data-domain-aware preference optimization, and ROPD.
- Conclusion: 7 representative open audio-visual generation models are significantly outperformed by MaineCoon in evaluations covering generation quality and inference speed.The conclusion characterizes this result as a new state-of-the-art standard.
- Conclusion: Future work will add active observation and internal social simulator modules, then jointly optimize them with MaineCoon’s reactive module.The stated aim is to develop the three core modules of a social world model.
Contributions
The Contributions section credits three core contributors and lists fourteen additional contributors. Together, the passages identify the paper’s credited contributor roster.
- Core Contributors: Three core contributors are credited: Lichen Bai, Tianhao Zhang, and Zeke Xie.The passage marks Bai and Zhang with ⋆ and Xie with †.
- Contributions: The credited contributor roster comprises the three core contributors and fourteen additional contributors.This combines the two contributor passages without adding roles beyond those stated.
- Contributors: Fourteen additional contributors are listed, including Shitong Shao, Dingwei Tan, Qiyu Zhong, and Zhengpeng Xie.The full passage also names Haopeng Li, Qinghao Huang, Dandan Shen, Tengjiao Ji, Wei Wang, Peicheng Wu, Yuxuan Zhao, Xiangyu Zhu, Welly Luo, and Shurui Yang.
A More Qualtiative Cases
MaineCoon produces qualitative examples across all seven SocialVideo-Bench domains under the same streaming inference setup, preserving temporal coherence and audio-visual synchronization. The examples demonstrate stable speech, interaction, music, motion, emotion, and social-media dynamics over streaming rollouts.
- Qualitative Coverage: Qualitative examples span seven SocialVideo-Bench domains, including dense speech, two-person interaction, music and vocal, emotional performance, dance, and social memes.Each domain is shown through several consecutive frames under the streaming inference setup described in Section 4.
- Speech and Interaction: Dense speech preserves natural lip synchronization, facial expression, and speech clarity throughout the streaming rollout.
- Speech and Interaction: Music and vocal generation consistently maintains vocal timbre and rhythmic motion, while two-person interaction preserves speaker identity, turn-taking, lip motion, and facial expressions.
- Motion and Emotion: Dance generation keeps large body movements temporally coherent without tearing or ghosting artifacts, while emotional performance smoothly transitions facial emotions and prosodic changes.
- Social Memes: Social memes capture humorous timing, narrative reversals, and expressive reactions while maintaining engaging social-media-style dynamics over the streaming rollout.