PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstruction Alignment Improves Unified Multimodal Models

Ji Xie, Trevor Darrell, Luke Zettlemoyer, XuDong Wang

arXiv:2509.07295v4cs.CVcs.AIcs.LG

TL;DR

Conventional caption-based training leaves fine-grained visual details underspecified. RECA post-trains UMMs to reconstruct images from their own visual understanding embeddings, providing dense supervision without captions. The authors report improvements in generation and editing, with gains varying across architectures.

  • Problem

    Conventional image-text training relies on captions that omit fine-grained visual details such as spatial layout, geometry, and attributes.

    Source: §1 · p. 2

  • Method

    RECA conditions a UMM on its own visual understanding embeddings and trains it to reconstruct the input image with a self-supervised loss.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    RECA improves generation and editing benchmarks; with GPT-4o-Image data, the reported scores are GenEval 0.90 and DPGBench 88.15.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RECA is a lightweight post-training method that improves image generation and editing across architectures without requiring extra caption data.

    Source: §5 · p. 10

    Gains vary by architecture: Harmon and OpenUni improve, while BAGEL and Show-o show limited or no gains; reasoning ability remains an open problem.

    Source: §3.3 · p. 8

Abstract

from arXiv · show

Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture. However, conventional training relies on image-text pairs (or sequences) whose captions are typically sparse and miss fine-grained visual details, even when they use hundreds of words to describe a simple image. We introduce Reconstruction Alignment (RECA), a resource-efficient post-training method that leverages visual understanding encoder embeddings as dense "text prompts", providing rich supervision without captions. Concretely, RECA conditions a UMM on its own visual understanding embeddings and optimizes it to reconstruct the input image with a self-supervised reconstruction loss, thereby realigning understanding and generation. Despite its simplicity, RECA is broadly applicable: across autoregressive, masked-autoregressive, and diffusion-based UMMs, it consistently improves generation and editing fidelity. With only 27 GPU hours, post-training with RECA substantially improves image generation performance on GenEval (0.73 $\rightarrow$ 0.90) and DPGBench (80.93 $\rightarrow$ 88.15), while also boosting editing benchmarks (ImgEdit 3.38 $\rightarrow$ 3.75, GEdit 6.94 $\rightarrow$ 7.27). Notably, RECA surpasses much larger open-source models and applies broadly across diverse UMM architectures, establishing it as an efficient and general post-training alignment strategy for UMMs.

1 INTRODUCTION

Conventional caption-based training leaves fine-grained visual details underspecified, motivating RECA’s use of visual understanding embeddings as dense prompts for image reconstruction. RECA improves generation and editing, with results reported across diverse UMM architectures.

1 INTRODUCTION

  • Captions can omit spatial layout, geometry, and fine-grained attributes, while visual understanding embeddings provide denser semantic supervision for generation.The introduction illustrates the gap with uncommon prompts such as “a yellow broccoli,” which models may generate as green despite recognizing the concept.Source: §1 · p. 2, Figure 2 · p. 2, Figure 3 · p. 2
  • RECA conditions a UMM on its own visual understanding embeddings and trains it to reconstruct the input image, supplying supervision without paired captions.The pipeline uses a self-supervised loss between original and generated images or image latents; the encoder is frozen except when understanding and generation share it.Source: §1 · p. 2, Figure 4 · p. 3
  • With only 27 A100 GPU hours, RECA post-trains a 1.5B-parameter UMM without GPT-4o-Image distillation or reinforcement learning to GenEval 0.86 and DPGBench 87.21; GPT-4o-Image data further raises these to 0.90 and 88.15, while ImgEdit rises from 3.38 to 3.75 and GEdit from 6.94 to 7.27.Source: §1 · p. 3
  • RECA improves diverse UMM families, including discrete, masked-autoregressive, and continuous-generation models.The contribution summary also characterizes the method as applicable from autoregressive models to hybrid frameworks.Source: §1 · p. 3

2 RECONSTRUCTION ALIGNMENT

RECA uses a UMM’s own visual understanding embeddings as input to a self-supervised image reconstruction objective. The method is motivated by a gap between recognizing visual content and generating it, and differs from prior reconstruction approaches by making semantic reconstruction a native post-training objective.

2 RECONSTRUCTION ALIGNMENT

  • RECA trains a UMM to reconstruct images from its visual understanding embeddings, providing dense supervision for fine-grained details often omitted by captions.The paper presents this as a self-supervised reconstruction objective.Source: §2 · p. 3

2.1 MOTIVATION AND SETUP

  • When understanding embeddings are paired with a reconstruction instruction, baseline generations preserve the main subject but scramble spatial layout and composition.The examples motivate training the image itself as a dense “text prompt” to improve the generation branch’s use of LLM features.Source: §2.1 · p. 4, Figure 5 · p. 4

2.2 RECA TRAINING PARADIGM

  • RECA replaces text-to-image supervision with a reconstruction loss that conditions the UMM on its own visual understanding embeddings and compares its output with the ground-truth image.The formulation uses a reconstruction-triggering prompt template; the training paradigm includes text-to-image and image-to-text objectives, with losses such as cross-entropy or diffusion loss depending on the model.Source: §2.2 · p. 4
  • For UMMs with shared understanding and generation parameters, image-to-text training is retained and the shared encoder is not frozen; otherwise, the understanding component is frozen.Inputs are resized to the understanding encoder’s minimum accepted resolution, and post-trained models use standard inference inputs without additional visual embeddings.Source: §2.2 · p. 4

2.3 DIFFERENCE BETWEEN RECA AND PREVIOUS WORKS

  • Unlike prior reconstruction methods, RECA uses semantic-level reconstruction as a native UMM post-training objective without auxiliary modules or text-to-image data.The paper distinguishes its methodology, architecture, motivation, and task from earlier reconstruction approaches.Source: §2.3 · p. 5

3 EXPERIMENTS

RECA improves image generation and editing across unified multimodal models, while comparisons and ablations identify when it works best. The experiments also report that visual-understanding performance is preserved and that RECA works best after SFT.

3.2 RECA ACHIEVES SOTA PERFORMANCE ON IMAGE GENERATION AND EDITING

  • RECA achieves state-of-the-art generation results and scores 3.75 on ImgEdit and 7.27 on GEdit-Bench-EN, outperforming the cited editing baselines.On ImgEdit, the baseline scores 3.38 and FLUX-Kontext scores 3.60; on GEdit-Bench-EN, RECA closes the gap between open-source models and GPT-4o-Image.Source: §3 · p. 5, Table 3 · p. 6, §3.2 · pp. 6–7

3.1 EXPERIMENT SETUP

  • Experiments span discrete, masked-autoregressive, and continuous-diffusion UMMs, evaluating generation and editing on four benchmarks with open-source training data.The benchmarks are GenEval, DPGBench, ImgEdit, and GEdit-Bench-EN; the main experiments exclude GPT-4o-Image distillation data.Source: §3.1 · p. 6

3.3 RECA IS EFFECTIVE ACROSS DIFFERENT UMM FRAMEWORKS

  • Across evaluated UMM frameworks, RECA delivers consistent significant improvements; for largest variants, Harmon-1.5B achieves 85.7 (+12.8) on GenEval and 87.21 (+6.28) on DPGBench.RECA improves Harmon and OpenUni, while BAGEL and Show-o show limited or no gains.Source: §3.3 · pp. 7–8

3.4 MORE RESULTS

  • Qualitative comparisons show better preservation of multiple objects, complex attributes, spatial layouts, and fine details in generation, plus improvements across editing tasks.Editing examples cover object addition, replacement, style transfer, and color modification.Source: Figure 6 · p. 7, Figure 7 · p. 7, §3.4 · p. 8
  • For UMMs sharing understanding and generation parameters, RECA improves generation fidelity without degrading visual understanding; decoupled understanding components remain unchanged.The evaluated understanding benchmarks include POPE, MME, GQA, MMMU, and SEED.Source: §3.4 · p. 8, Table 4 · p. 8

3.5 EMPIRICAL STUDIES OF RECA AS A POST-TRAINING METHOD

  • RECA outperforms SFT across the reported comparisons, including when template-leakage data is excluded, while denser captions do not improve SFT results.On MidjourneyV6, SFT scores 74.76 on GenEval; removing template-leakage data lowers SFT GenEval by 4.07, while RECA maintains strong performance.Source: §3.5 · p. 8, Table 5 · p. 8
  • Sequential SFT→RECA performs best: reversing the order degrades MidjourneyV6 results by 3.09/1.83, while mixed training is unstable across datasets.The conclusion recommends SFT on high-quality paired data for coarse alignment, followed by RECA for self-supervised fine-grained refinement.Source: §3.5 · pp. 8–9, Table 6 · p. 8
  • Understanding-encoder embeddings outperform generation-encoder embeddings, and 224 × 224 inputs outperform 512 × 512 inputs across BAGEL benchmarks.The authors associate higher-resolution embeddings with greater pixel-level detail and a shift toward low-level cues rather than the semantic features RECA emphasizes.Source: Table 7 · p. 9, §3.5 · p. 9

4 RELATED WORK

The related work situates UMMs among several image-generation architectures and reviews approaches to improving their generation capabilities. It also illustrates post-trained generation and editing outputs.

4.1 UNIFIED MULTIMODAL MODELS (UMMS)

  • UMMs combine multimodal understanding and generation, with discrete-token, continuous-diffusion, and masked-autoregressive model families.Examples include Chameleon and Show-o, Transfusion, and Harmon, respectively.Source: §4.1 · p. 9

4.2 POST-TRAINING STRATEGIES FOR UMMS

  • Prior post-training strategies include reasoning or verification with external models, preference-based reinforcement learning, and supervised fine-tuning on synthetic image-text pairs.The cited discussion notes that reasoning and verification do not improve native UMM generation, preference methods require curated paired data and tuned advantage functions, and synthetic data carries generation cost and potential distribution shift.Source: §4.2 · pp. 9–10
  • Qualitative examples show post-trained BAGEL generation at 1024 × 1024 and Harmon generation at 512 × 512, alongside BAGEL image edits.The editing figure places original images on the left and edited images on the right.Source: Figure 8 · p. 10, Figure 9 · p. 10

5 CONCLUSION

RECA is a lightweight post-training method that uses visual-understanding embeddings from a model’s own encoder instead of sparse text supervision, improving image generation and editing across architectures.

5 CONCLUSION

  • RECA replaces sparse text-to-image supervision with dense features from the model’s own visual-understanding encoder, without requiring extra caption data.The conclusion reports improved image generation and editing across architectures.Source: §5 · p. 10

A.1 DISCRETE PARADIGM

  • Discrete image generation predicts sequences of image-token codes, either autoregressively or through masked parallel prediction.Masked prediction trains on masked token positions and iteratively fills them during inference.Source: §A.1 · pp. 19–20

A.2 CONTINUOUS PARADIGM

  • Diffusion generates images by progressively corrupting clean images with noise and learning to reverse that process.Continuous paradigms also include flow matching and rectified flow, which define trajectories between noise and data.Source: §A.2 · p. 20

A.3 MASKED AUTOREGRESSIVE (MAR)

  • Masked autoregressive generation uses a diffusion decoder to generate continuous image features and stochastically samples denoised patches at each step.Unlike MaskGIT, it predicts continuous features rather than logits over discrete codes.Source: §A.3 · p. 21

B DIFFERENCE BETWEEN RECA AND CLASSIFIER-FREE GUIDANCE (CFG)

  • Classifier-free guidance combines conditional and unconditional predictions, while RECA uses visual-understanding embeddings for reconstruction alignment; the methods are compatible.The guidance scale controls the combination of conditional and unconditional predictions.Source: B DIFFERENCE BETWEEN RECA AND… · p. 21

C.1 VISUAL EMBEDDING INTEGRATION

  • RECA concatenates visual embeddings with a text template and trains the model to reconstruct the original image rather than generate text.The embeddings can function as dense, language-aligned prompts because visual and text features share a semantic manifold in deep transformer layers.Source: §C.1 · pp. 21–22, Figure 11 · p. 22

C.2 PROMPT TEMPLATE DIVERSIFICATION

  • RECA diversifies prompts using 360 templates that vary in linguistic structure, complexity, and prompting style while preserving the goal of eliciting detailed image descriptions.The templates are generated from the seed prompt “Describe the image in detail.”Source: §C.2 · p. 22

D MECHANISTIC ANALYSIS OF RECA

The analysis finds that visual embeddings act as semantic prompts, and that reconstruction capability is strongly correlated with generation capability.

Visual-Textual Feature Isomorphism

  • In transformer layers 3–20, text and visual features align, supporting the use of visual embeddings as dense prompts for generation.The example compares the word “black” with an image of a black dog.Source: D MECHANISTIC ANALYSIS OF RECA · p. 23

Internal Semantic Projection

  • For an anomalous pink-banana input, the model maps the visual features to its learned “pink shoe” concept and reconstructs a shoe.The analysis describes this as internal semantic projection rather than direct reproduction of raw pixels.Source: D MECHANISTIC ANALYSIS OF RECA · p. 23

Correlation Between Reconstruction and Generation

  • The model’s failure to reconstruct a microwave from a dense image prompt coincides with failure to generate it from a sparse text prompt across multiple seeds.The authors report a strong empirical correlation between reconstruction and generation, suggesting that refining the mapping may improve generation capability.Source: D MECHANISTIC ANALYSIS OF RECA · p. 23

E EXPERIMENTAL SETUP

Across evaluated architectures, RECA improves performance, particularly semantic alignment; the additional analyses examine evaluation leakage and reconstruction settings that affect results.

E EXPERIMENTAL SETUP

  • The experimental setup uses one A100 for post-training and evaluates generation and editing with established benchmarks across multiple model architectures.Training data vary by model: Show-o, Harmon, and OpenUni use MidjourneyV6, while BAGEL uses 10,000 FLUX-generated images; shared-parameter models also include understanding data to preserve visual understanding.Source: E EXPERIMENTAL SETUP · pp. 23–24

F MORE EXPERIMENTAL RESULTS AND ANALYSIS

F.1 GENEVAL TEMPLATE LEAKAGE

  • With the full BLIP3o-60k dataset, SFT scores 84.95 on GenEval but falls to 80.88 after removing 7,000 near-duplicate template pairs; DPGBench changes by only +0.05.RECA reaches 85.21 on GenEval with the full dataset and surpasses SFT on DPGBench, 86.50 versus 85.19.Source: §F.1 · p. 25

F.2 RECA ON SHOW-O WITH IMAGE TOKENIZATION

  • For Show-o’s VQGAN variant, 512 × 512 reconstruction causes token-copying collapse, while blurred inputs yield the highest GenEval and DPGBench scores but slightly lower WISE reasoning accuracy.Resizing to 256 × 256 gives the best balance across metrics; 128 × 128 inputs significantly reduce all reported scores.Source: §F.2 · pp. 25–26, Figure 14 · p. 26

F.3 COMPREHENSIVE QUANTITATIVE RESULTS

  • RECA consistently improves performance across evaluated architectures, primarily enhancing semantic alignment and showing more modest gains on reasoning-intensive tasks.The analysis reports notable gains in positional understanding and color attribution.Source: §F.3 · p. 26

F.4 ADDITIONAL COMPOSITIONAL EVALUATION ON T2I-COMPBENCH

RECA is evaluated on broader compositional attributes, shows gains across tested UMMs, and reveals distinct dynamics across GenEval subtasks and architecture-specific constraints.

F.4 ADDITIONAL COMPOSITIONAL EVALUATION ON T2I-COMPBENCH

  • T2I-CompBench evaluates 2,670 samples across 10 compositional dimensions, including shape and texture, numeracy, 3D spatial reasoning, and complex scenes.Its 3D spatial tests distinguish depth and occlusion from 2D spatial relations.Source: §F.4 · pp. 26–27
  • RECA improves T2I-CompBench across tested UMM architectures, with reported gains including +15.75 for OpenUni 3D Spatial, +10.50 for Harmon 3D Spatial, and +6.46 for Harmon-1.5B Complex Scenario.OpenUni also gains +10.38 on Shape; Show-o-512 has a −0.25 fluctuation in Non-Spatial.Source: §F.4 · pp. 27–28, Table 12 · p. 28

F.5 GENEVAL SUBTASK DYNAMICS

  • GenEval gains occur mainly within the first 3k–5k training steps; Color continues rising, while Counting shows no clear upward trend.Single Object and Two Objects quickly reach their ceiling; Color Attributes peaks around 5k steps and then gently drops.Source: §F.5 · p. 28, Figure 15 · p. 29

F.6 FURTHER DISCUSSION ON EXPERIMENT RESULTS

  • Show-o’s RECA gains are limited by its CLIP encoder’s insufficient high-level semantic information and its 4096-token codebook.Source: §F.6 · p. 29
  • BAGEL’s generation improvement is constrained by available data and compute, while its pre-existing semantic reconstruction training limits the remaining generation headroom and supports editing gains.Source: §F.6 · p. 29

F.7 MORE VISUALIZATION OF RECONSTRUCTION RESULTS

Visualizations show reconstruction improving during RECA training and across the evaluated UMM architectures, with model-specific changes in recovered detail.

F.8 MORE VISUALIZATION OF RECONSTRUCTION RESULTS

  • RECA improves reconstruction quality across all evaluated architectures, with clearer semantic content and object structures in Show-o, and better geometry, color, and texture in OpenUni and Harmon.Source: §F.7 · p. 29
  • For BAGEL, RECA further improves already-existing reconstruction ability, especially fine-grained semantic details and spatial relationships.Source: §F.7 · p. 29
  • Harmon’s early reconstructions are blurry and poorly structured, but become sharp and semantically faithful as training progresses.Later outputs recover object shapes, colors, and layouts more accurately.Source: §F.7 · p. 29, Figure 16 · p. 30

F.8 MORE ABLATION STUDIES

Ablations report that RECA gains persist across training-data scales and sources, including lower-quality real-world images.

F.8 MORE ABLATION STUDIES

  • RECA shows consistent performance gains as Harmon-1.5B training data increases from 10,000 to 240,000 MidjourneyV6 images.Source: §F.8 · p. 30, Table 13 · p. 30
  • RECA maintains stable performance across real-world and synthetic training-data sources of different quality levels.The evaluated sources are COCO2014, JourneyDB, BLIP3o-60k, and MidjourneyV6.Source: §F.8 · p. 30
  • Training on lower-quality real-world COCO images yields a GenEval score of 85.3, which the authors describe as a competitive improvement.Source: §F.8 · p. 30

G LIMITATIONS AND FUTURE WORK

The reported limitations concern modest gains on counting and limited effectiveness on BLIP-3o, whose pretraining already includes reconstruction objectives.

G LIMITATIONS AND FUTURE WORK

  • RECA’s counting gains are minor; the authors attribute this to current MLLMs’ difficulty extracting object number as a mid-level visual feature.Source: G LIMITATIONS AND FUTURE WORK · pp. 30–31
  • RECA has limited effectiveness on BLIP-3o, where pretraining already includes strong reconstruction objectives and additional reconstruction training is reported as counterproductive.The authors state that RECA fails to improve BLIP-3o reconstruction or text-to-image generation quality.Source: G LIMITATIONS AND FUTURE WORK · p. 31, Figure 18 · p. 31

I MORE QUALITATIVE RESULTS ON IMAGE EDITING

The editing comparisons report stronger instruction following than ICEdit and FLUX-Kontext, and better identity preservation and background fidelity than GPT-4o-Image.

I MORE QUALITATIVE RESULTS ON IMAGE EDITING

  • RECA shows stronger instruction following than ICEdit and FLUX-Kontext, including rainbow-dome and star-addition edits.The comparisons also report consistent gains in semantic consistency, instruction following, and visual quality preservation across background modification, style transfer, and object manipulation.Source: I MORE QUALITATIVE RESULTS ON IMAGE… · p. 35, Figure 21 · p. 36
  • Against GPT-4o-Image, RECA exhibits better identity preservation and background fidelity, including for pet and human faces.The qualitative comparison also notes stronger background consistency.Source: I MORE QUALITATIVE RESULTS ON IMAGE… · p. 35, Figure 21 · p. 36

J MORE QUALITATIVE RESULTS ON TEXT-TO-IMAGE GENERATION

The section presents uncurated text-to-image examples comparing Harmon-1.5B with its post-trained model across varied prompts, plus examples from BAGEL, OpenUni, and Show-o-512x512.

J MORE QUALITATIVE RESULTS ON TEXT-TO-IMAGE GENERATION

  • Harmon-1.5B and its post-trained model are compared on prompts involving unusual colors, objects, and spatial relations.Examples include a white and black banana, yellow broccoli, orange snowboard and green cat, skis right of a zebra, microwave and truck, and jewels in specified positions.Source: Figure 22 · p. 37, Figure 23 · p. 37, Figure 24 · p. 37
  • Further Harmon-1.5B comparisons use detailed scene prompts for lighting, animals, produce, and everyday objects.The examples include a bright shell-and-bear scene, a calico cat with sunglasses, tomatoes spilling from a basket, a briefcase with a red fedora, and balloons near a ceiling fan.Source: Figure 28 · p. 39, Figure 29 · p. 39, Figure 30 · p. 39
  • Additional single-object examples compare the post-trained model with Harmon-1.5B, BAGEL, OpenUni, and Show-o-512x512.The prompts depict a fire hydrant, surfboard, suitcase, and cup, respectively.Source: Figure 33 · p. 41, Figure 34 · p. 41, Figure 35 · p. 42
Loading 2509.07295v4…