PaperlayerView exact revision on arXiv

Source-linked AI summary

JoyAI-Image: Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation

Lin Song, Wenbo Li, Guoqing Ma, Wei Tang, Bo Wang, Yuan Zhang, Yijun Yang, Yicheng Xiao, Jianhui Liu, Yanbing Zhang, Guohui Zhang, Wenhu Zhang, Hang Xu, Nan Jiang, Xin Han, Haoze Sun, Maoquan Zhang, Haoyang Huang, Nan Duan

arXiv:2605.04128v2cs.GRcs.AIcs.CLcs.CVcs.LG

TL;DR

Unified multimodal models still weakly connect understanding, generation, and editing, and remain limited in fine-grained spatial intelligence. JoyAI-Image couples a spatially enhanced MLLM with an MMDiT and spatially grounded training, achieving strong performance across visual tasks while strengthening spatial intelligence.

  • Problem

    Existing unified models weakly connect understanding, generation, and editing and remain limited in fine-grained spatial understanding and geometrically precise manipulation.

  • Method

    JoyAI-Image couples a spatially enhanced MLLM with an MMDiT and injects spatially grounded data, task design, and supervision throughout unified training.

  • Results

    JoyAI-Image achieves strong performance across broad visual tasks, including spatial understanding, long-text rendering, controllable editing, and multi-view generation.

  • Takeaways & Limitations

    The results support spatial intelligence as a first-class property of unified visual modeling, with implications for visual-language-action systems, robotics, and world models.

  • Takeaways & Limitations

    The model’s reinforcement-learning evaluation remains limited by the vision-language model judge’s capabilities and may not align with human preferences.

Abstract

from arXiv · show

We present JoyAI-Image, a unified multimodal foundation model for visual understanding, text-to-image generation, and instruction-guided image editing. JoyAI-Image couples a spatially enhanced Multimodal Large Language Model (MLLM) with a Multimodal Diffusion Transformer (MMDiT), allowing perception and generation to interact through a shared multimodal interface. Around this architecture, we build a scalable training recipe that combines unified instruction tuning, long-text rendering supervision, spatially grounded data, and both general and spatial editing signals. This design gives the model broad multimodal capability while strengthening geometry-aware reasoning and controllable visual synthesis. Experiments across understanding, generation, long-text rendering, and editing benchmarks show that JoyAI-Image achieves state-of-the-art or highly competitive performance. More importantly, the bidirectional loop between enhanced understanding, controllable spatial editing, and novel-view-assisted reasoning enables the model to move beyond general visual competence toward stronger spatial intelligence. These results suggest a promising path for unified visual models in downstream applications such as vision-language-action systems and world models.

1 Introduction

JoyAI-Image is a unified multimodal framework that couples a spatially enhanced MLLM with an MMDiT to support image understanding, generation, and instruction-based editing. Its unified training and bidirectional understanding-generation design targets weak cross-task interaction and stronger spatial intelligence.

  • Introduction: Current unified models integrate understanding, generation, and editing but retain weak cross-task interaction and limited spatial intelligence.Understanding is not fully exploited to guide grounded generation and editing, while generative transformations rarely provide feedback for perception and reasoning.
  • Introduction: JoyAI-Image combines a spatially enhanced MLLM with an MMDiT through a shared interface for image understanding, text-to-image generation, and instruction-based editing.The MLLM provides scene understanding, instruction parsing, and semantically rich, spatially grounded conditioning for downstream generation and editing.
  • Introduction: Spatial intelligence is trained throughout JoyAI-Image’s unified pipeline using spatially grounded data, task design, and supervision alongside understanding, generation, and editing.The framework treats spatial capability as a jointly developed property rather than an isolated module or late-stage extension.
  • Introduction: The framework’s bidirectional loop strengthens spatial understanding, controllable spatial editing, and novel-view-assisted reasoning, with implications for robotic systems and world models.The contribution emphasizes spatial intelligence beyond general-purpose visual performance.
  • Introduction: JoyAI-Image uses unified instruction following and a multi-stage curriculum to harmonize understanding, generation, and editing objectives across general visual tasks and specialized spatial operations.The data suite spans fine-grained spatial reasoning, high-fidelity synthesis, long-text typography, and versatile content editing.

2 Model

JoyAI-Image unifies image understanding, generation, and instruction-based editing by coupling a spatially enhanced MLLM with a VAE and MMDiT. Its pipeline moves from scene comprehension and intent parsing through latent encoding to conditional generation, with progressive training across these capabilities.

  • Architecture: JoyAI-Image combines a spatially enhanced MLLM, VAE, and MMDiT in one framework for image understanding, high-fidelity synthesis, and instruction-based editing.The MLLM provides understanding and intent mediation, the VAE maps pixels to latents, and the MMDiT performs conditional generation.
  • Multimodal Understanding: The MLLM acts both as a standalone engine for scene parsing and spatial reasoning and as an intent mediator for interleaved instructions and reference signals.For understanding tasks it decodes representations into text; for synthesis and editing it parses inputs into structured semantic representations and visual-attribute mappings.
  • Training: Training first fine-tunes the MLLM for visual-spatial understanding, then trains the MMDiT from scratch with MLLM-derived priors, and finally optimizes precise instruction-based editing.For synthesis and editing, final-layer MLLM hidden states provide high-level semantic-spatial conditioning features for the MMDiT.
  • Multimodal Understanding: The comprehension module is built on Qwen3-VL-8B-Instruct and is further strengthened for geometric awareness and multi-view structural consistency.The supplied passage introduces this spatial-reasoning enhancement but is truncated before specifying its implementation.
  • Latent Encoding and Conditional Generation: Wan-2.1-VAE provides causal 3D-convolutional spatio-temporal compression, while a 16B-parameter MMDiT jointly models MLLM multimodal representations and VAE latents.The VAE preserves fine-grained structures and high-frequency details, including small text, during reconstruction; the MMDiT uses a dual-stream design for multimodal fusion.

3 Advanced Spatial Understanding

The section introduces OpenSpatial, a 3D box-centric data engine and 3-million-entry dataset for spatially grounded supervision across five capabilities. Spatial-specialized training combines this data with broader multimodal sources, yielding a 64.4 average on spatial-understanding benchmarks, 5.3 points above the base model.

  • OpenSpatial data engine: OpenSpatial uses 3D lifting to transform unconstrained web videos into high-fidelity spatial training data and curates OpenSpatial-3M with 3 million entries.The dataset covers Spatial Measurement, Spatial Relationship, Camera Perception, Multi-view Consistency, and Scene-Aware Reasoning.
  • Geometric representation: 3D oriented bounding boxes provide metric scale, centroids, and orientations, while a unified object-frame index synchronizes 3D/2D boxes, masks, point clouds, and metadata.The workflow projects scene-level primitives to frames, applies visibility filtering and mask refinement, and uses multi-view cycle consistency to validate lifted boxes.
  • Training corpus: The training corpus contains approximately 11.3M samples, with 6.1M general-understanding samples (54.25%) and 3.4M spatial-understanding samples (29.65%) mixed using per-dataset sampling ratios.Prompt enhancement adds Instruction Rewriting (1.4M, 11.98%) and Spatial Editing (137.4K, 1.21%) sources for generation and editing.
  • Spatial training data: The spatial branch spans single-view, multi-view, and video supervision for distance, size, depth, position, correspondence, scene captioning, camera motion, grounding, orientation, and camera-pose estimation.OpenSpatial contributes 3.3M samples, while the VST Subset contributes 49.4K multi-view camera-motion samples.
  • Results: 64.4 average score on spatial understanding marks a new state-of-the-art and improves the base model by 5.3 points, matching Gemini-2.5-Pro.The comparison covers 13 benchmarks across proprietary and open-source VLMs.

4 Text-to-Image: JoyAI-Image · 4.1 Data Pipeline

JoyAI-Image’s data pipeline progressively improves training-data quality and coverage through filtering, captioning, semantic rebalancing, human curation, and approximately 1M-scale multi-view generation. These modules provide multi-level and OCR-aware supervision, address long-tail concepts, and align data with controllable viewpoint-conditioned generation.

  • 4.1 Data Pipeline: The five-module pipeline jointly optimizes data quality and distributional coverage through progressive filtering, multi-level captioning, semantic rebalancing, human annotation, and multi-view generation.Filtering thresholds become increasingly stringent across training stages, while the other modules expand textual, semantic, curated, and viewpoint-conditioned supervision.
  • 4.1.1 Data Filtering: 90% human-verification accuracy supports the in-house IQA operator’s cascaded decisions for accepting, recovering, or rejecting images.The operator combines statistical and perceptual indicators; extremely low brightness and severely abnormal statistics trigger rejection.
  • 4.1.1 Data Filtering: Caption-based keyword and pattern matching uses the deployed VLM captioner to identify collages, watermarks, logos, screenshots, and memes at billion-scale.The approach incurs no additional cost, is orders of magnitude faster than image classification, and can be updated by modifying keyword lists.
  • 4.1.2 Captioning: OCR-aware captioning combines OCR-extracted tokens with visual features in an MLLM to produce text-grounded descriptions, complementing multi-level captions for holistic understanding and fine-grained control.Both captioning strategies use Qwen3-VL-8B-Instruct [3] as the unified captioning backbone.
  • 4.1.3 Rebalancing: 285K leaf-node categories and top-K (K=1000) cosine-similarity tagging enable scalable semantic rebalancing, with 86% human-verification accuracy in embedding matching tests.Adaptive diversity sampling reduces semantic redundancy, while inverse-logarithmic sampling retains tail categories and downsampling head categories.
  • 4.1.4 Annotating: Human-in-the-loop annotation establishes higher-quality SFT curation through multi-dimensional scoring of aesthetics, information density, and style purity.The scoring dimensions use weights of 50%, 30%, and 20%, respectively.
  • 4.1.5 Multi-view Generation: Approximately 1M-scale multi-view generation data covers object-centric and scene-centric settings with a dominant subject for controllable generation under explicit viewpoint constraints.The corpus is designed for multi-view collage generation and practical inference requests for target or coherent supporting views.
  • 4.1.5 Multi-view Generation: Blender 4.5 rendering uses one main view and supporting sub-views, while dual dense and schema-constrained JSON captions encode layout, subject, rotation, anchoring, and per-view details.Cameras are oriented toward the target object’s center, with intrinsics and object-relative distances sampled to maintain consistent framing.

4.2 JoyAI-Image Model

JoyAI-Image is trained through flow matching and a progressive multi-resolution text-to-image schedule spanning 208P, 512P, and 1024P. Subsequent continued training, supervised fine-tuning, and multi-reward reinforcement learning target generation quality, complex text rendering, multi-view generation, perceptual quality, and prompt faithfulness.

  • Flow-matching objective: The model uses flow matching to predict the velocity field along a linear interpolation path between Gaussian noise and clean image latents.Image latents are obtained with WanVAE, and text provides the conditioning signal.
  • Progressive pre-training: Pre-training follows a progressive multi-resolution schedule with low-resolution training (208P), mid-resolution training (512P), and high-resolution training (1024P).The schedule establishes basic text-conditioned image generation rather than directly optimizing target-resolution quality.
  • Progressive pre-training: Multi-view data is introduced during the high-resolution (1024P) stage to encourage consistent 3D-aware understanding and improve controllability across viewpoints.For each prompt, the model generates images from different viewpoints.
  • Quality refinement: Continued training narrows the data distribution through strict filtering and reweighting, emphasizing visually appealing, compositionally coherent, and semantically accurate samples.The high-quality subset is constructed from the original training corpus to improve generation quality after large-scale pre-training.
  • Capability enhancement: Supervised fine-tuning uses thousands of human-annotated samples to strengthen complex text rendering and multi-view generation, while Flow-GRPO optimizes aesthetic quality and text-image alignment.The reinforcement-learning stage combines aesthetic and text-image alignment reward models for perceptual quality and prompt faithfulness.

4.3 Evaluation

JoyAI-Image achieves highly competitive image-generation performance across long-text rendering, general generation, composition, reasoning, and stylistic synthesis. Its strongest results include bilingual text accuracy, leading word accuracy, strong compositional performance, and improved visual coherence and aesthetic quality.

  • Overall evaluation: JoyAI-Image performs highly competitively across image-generation settings, with particularly strong and consistent bilingual English–Chinese text generation.The evaluation covers long-text rendering, instruction following, and stylistic quality.
  • LongText-Bench: 0.963 on both LongText-Bench-EN and LongText-Bench-ZH, outperforming existing methods while maintaining balanced bilingual accuracy.Prior models exhibit performance gaps across languages, whereas JoyAI-Image remains consistently strong in English and Chinese.
  • CVTG-2k: 0.8739 Word Accuracy, exceeding Z-Image (0.8671) and GPT Image 1 (0.8569), while achieving a competitive NED of 0.9369.Word Accuracy reflects character-level generation precision, while NED measures normalized edit distance.
  • OneIG and DPG: JoyAI-Image scores 0.542 on OneIG English, ranking second, 0.521 on OneIG Chinese, and 88.05 overall on DPG.These results demonstrate competitive bilingual generation and strong general text-to-image quality across diverse prompts and settings.
  • CoreBench: On T2I-CoReBench, JoyAI-Image achieves 68.7 overall and a best composition mean of 94.2 across four composition-related dimensions.The benchmark evaluates compositional and reasoning capabilities across diverse tasks.
  • Qualitative comparison: Qualitative comparisons show more accurate text rendering, stronger multilingual consistency, improved layout and visual coherence, and higher aesthetic quality in stylized scenarios.The examples cover artistic Chinese text rendering, structured bilingual typography, and stylized editorial synthesis.

5 Image Editing: JoyAI-Image-Edit · 5.1 Image Editing Data

JoyAI-Image-Edit is trained for instruction-following edits that modify targets while preserving fidelity, identity, and non-target regions. Its data pipeline combines broad open-domain coverage with spatial, text, multi-image, and other specialized supervision, followed by unified curation, refinement, and balancing.

  • 5 Image Editing: JoyAI-Image-Edit: JoyAI-Image-Edit targets diverse instruction-following edits while preserving visual fidelity, content identity, and non-target regions.The training pipeline combines broad-coverage general editing data with specialized supervision for challenging settings.
  • 5.1.1 Data Distribution: The capability-oriented mixture combines nearly half open-domain editing, substantial spatial editing, and specialized general editing for long-tail controllability.These branches balance broad visual changes, geometry-aware transformations, and challenging specialist tasks.
  • 5.1.2 General Editing Data Engine: General editing data covers IP-preserving edits, text rendering, in-context generation, portrait retouching, motion-aware modification, and localized appearance or content transformations.Sources include open-source datasets, expert-model distillation with multimodal filtering, dedicated text editing, and multi-image reference data.
  • 5.1.3 Open-domain Editing Data Engine: Open-domain editing uses video-derived pairs to provide broad instruction following, semantic diversity, naturally occurring changes, and robust preservation of non-target regions.Shots are segmented, frames uniformly sampled, and blur- or defocus-affected frames filtered before forming candidate pairs.
  • 5.1.4 Spatial Editing Data Engine: A scalable 3D-driven engine creates geometrically unambiguous spatial edits through fixed-camera object transformations and dynamic-camera viewpoint transformations.Object edits include translation, scaling, and rotation; viewpoint changes vary yaw, pitch, and distance while controlling scene geometry and camera intrinsics.
  • 5.1.5 Data Curation and Quality Control: Unified curation removes low-fidelity, artifact-prone, ambiguous, negligible-change, or instruction-misaligned samples and applies branch-specific validity checks.Model-based assessment and targeted manual inspection monitor semantic faithfulness, content preservation, visual plausibility, and recurring failure modes.
  • 5.1.6 Instruction Refinement and Unified Data Representation: All sources are standardized into shared samples containing source images, optional references, natural-language instructions, target images, and optional structured metadata.Noisy or underspecified descriptions are refined into explicit prompts aligned with source-target differences, including normalized templates and visual cues for structured tasks.
  • 5.1.7 Data Balancing and Training Mixture: The final corpus uses a balanced training mixture to prevent abundant generic editing pairs from overwhelming spatial, text, and multi-image tasks.Balancing considers task category, edit granularity, input condition, language, and semantics.

5.2 JoyAI-Image-Edit Model

JoyAI-Image-Edit uses an MMDiT denoising framework to condition editing on text, source images, and noisy latents while preserving non-target content. Its staged training progressively develops coarse editing, controllable specialized editing, targeted refinement, and preference-aligned fidelity using multimodal reward models.

  • Architecture: JoyAI-Image-Edit jointly models text conditions, source-image conditions, and noisy latent inputs within a unified MMDiT denoising framework.Given a source image and editing instruction, it predicts an edited target while preserving non-target content.
  • Training Stages: Pre-training establishes source-image perception, source–target difference understanding, and natural-language instruction association through generation, reconstruction, and coarse editing supervision.The supervision draws on broad image-text and video-related data, with final editing quality deferred to later stages.
  • Training Stages: Continue training converts coarse editing ability into more practical control using general, spatial, text-centric, and multi-image editing data.This stage improves instruction following, content preservation, and other controllable editing behaviors.
  • Training Stages: Optional SFT refines instruction fidelity, local controllability, text accuracy, spatial precision, reference consistency, robustness, and difficult long-tail cases.It targets sensitive editing dimensions that may remain underrepresented after earlier training stages.
  • Post-training and Rewards: Post-training uses DiffusionNFT and reward models to improve edit fidelity, visual quality, naturalness, consistency, usability, and alignment with human preferences.Gemini-3-Flash and HPSv3 replace limited LLM-as-a-Judge evaluation, with Gemini supporting instruction-following and consistency scoring.

5.3 Model Performance

JoyAI-Image-Edit maintains competitive general-purpose editing while substantially improving fine-grained spatial manipulation across quantitative, human, and qualitative evaluations. Its strongest advantages are instruction following, content preservation, geometric faithfulness, and spatial consistency, with perceptual naturalness remaining a relative weakness against Qwen-Image-Edit-2511.

  • General Editing: Evaluation on GEdit and ImgEdit shows competitive general editing performance without noticeable regression from spatial-editing training.These benchmarks assess instruction following, semantic consistency, editing quality, and preservation of irrelevant content.
  • Spatial Editing: JoyAI-Image-Edit exceeds leading approaches on SpatialEdit-Bench, improving faithful geometric compliance across object transformation and camera control.The benchmark emphasizes geometrically correct edits rather than merely visually plausible results.
  • Human Evaluation: Against Flux.2 [DEV], JoyAI-Image-Edit leads on Semantic Following (40.1% vs. 13.8%), Consistency (56.3% vs. 18.5%), Overall (60.8% vs. 23.2%), and Naturalness (34.8% vs. 34.0%).The largest margins are on Consistency and Overall, reflecting stronger preservation of identity, structure, and scene coherence.
  • Qualitative Results: Qualitative comparisons show stronger localized precision and content preservation in general editing, including attribute modification, style transfer, enhancement, and restoration.The model handles multi-subject scenarios and low-level vision tasks more effectively than several baselines.
  • Qualitative Results: Camera-control and object-transformation examples demonstrate viewpoint manipulation, target selection, non-target preservation, and stronger 3D consistency without geometric collapse.These results indicate early spatial reasoning related to viewpoint change and scene structure.

6 Applications

JoyAI-Image applies spatial editing to strengthen spatial reasoning through Thinking with Novel Views and to improve downstream 3D reconstruction from a single image. These applications show that generated viewpoints can expose spatial relations while preserving meaningful cross-view geometry.

  • Thinking with Novel Views: TwNV uses an MLLM Planner, generative Synthesizer, and MLLM Reasoner to explore novel viewpoints for answering spatial questions.The Planner predicts a 6-DOF camera motion, the Synthesizer renders the target view, and the Reasoner infers from the original and novel views.
  • Thinking with Novel Views: JoyAI-Image-Edit raises GPT-5’s overall accuracy from 68.8% to 71.7% when novel-view generation supports spatial reasoning.The evaluation suite contains 695 samples spanning Orientation, Location, and Multi-Object Relationship.
  • Thinking with Novel Views: TwNV improves diverse reasoners by 1.7 to 4.4 pp, with relative gains of 7.8% for Qwen3-VL-32B, 5.5% for Qwen3-VL-235B, and 2.3% for Gemini-3-Flash.The larger relative gains for smaller models indicate that explicit view synthesis can compensate for constrained spatial modeling capacity.
  • Thinking with Novel Views: Compared with Qwen-Image-Edit and Nano Banana Pro, JoyAI-Image-Edit executes camera motions more faithfully and exposes target spatial relations more clearly.Visualization covers representative height-comparison and vertical-relationship tasks.
  • Novel-View-Assisted 3D Reconstruction: Generated novel views improve VGGT-based 3D reconstruction from a single image, producing more complete geometry than reconstruction from the input alone.The improvement indicates preservation of camera geometry, object placement, scene structure, and cross-view correspondence to a meaningful extent.

7 Conclusion

JoyAI-Image unifies image understanding, text-to-image generation, and instruction-based editing through a spatially enhanced MLLM and large-scale MMDiT. The model is positioned as a step toward visual foundation models with grounded reasoning about structure, transformation, and physical-world geometry.

  • 7 Conclusion: JoyAI-Image brings image understanding, text-to-image generation, and instruction-based image editing into a shared framework centered on a spatially enhanced MLLM and large-scale MMDiT.The framework tightly couples understanding, generation, and editing rather than treating them as isolated capabilities.
  • 7 Conclusion: The unified design achieves strong performance across broad visual tasks, including spatial understanding and bilingual long-text rendering.
  • 7 Conclusion: The work targets visual systems that reason about structure, transformation, and physical-world geometry for grounded reasoning under change in visual-language-action systems, robotics, and world models.The authors frame this as progress from broad competence toward genuinely spatially intelligent visual foundation models.

8 Authors

Qualitative comparisons show JoyAI-Image’s strengths in Chinese and bilingual text rendering, aesthetic generation, attribute editing, restoration, and spatial viewpoint control. The examples also include multi-view try-on applications.

  • Text rendering: JoyAI-Image produces the most faithful and complete Chinese text rendering while maintaining strong visual coherence.
  • Text rendering: On bilingual layouts, JoyAI-Image generates more accurate Chinese-English content, clearer typography, and better overall layout fidelity.
  • Aesthetic generation: For stylized fashion editorials, JoyAI-Image delivers stronger aesthetic appeal through polished composition, richer color harmony, and high-fashion visual style.
  • Image editing: In attribute editing and restoration, JoyAI-Image better preserves identity, scene structure, lighting, and visual consistency while producing accurate, natural results.It removes degradation effectively during image restoration.
  • Spatial control: JoyAI-Image accurately follows camera and object viewpoint instructions while preserving scene content and visual consistency, and the paper illustrates multi-view try-on applications.
Loading 2605.04128v2…