PaperlayerView exact revision on arXiv

Source-linked AI summary

Factuality Matters: When Image Generation and Editing Meet Structured Visuals

Le Zhuo, Songhao Han, Yuandong Pu, Boxiang Qiu, Sayak Paul, Yue Liao, Yihao Liu, Jie Shao, Xi Chen, Si Liu, Hongsheng Li

arXiv:2510.05091v2cs.CV

TL;DR

Structured-image generation and editing remain difficult because they require factual accuracy, yet existing models struggle with these visuals. The paper builds a code-aligned dataset, trains a unified model, and introduces StructBench and StructScore. Even leading models remain far from satisfactory, while inference-time reasoning yields gains across models.

  • Problem

    Current models struggle to generate and edit structured visuals, which require factual accuracy, composition planning, precise text rendering, and multimodal reasoning.

    Source: §1 · p. 2

  • Method

    The paper constructs a code-aligned dataset with reasoning annotations, trains a unified model, and introduces StructBench with a fine-grained multi-round Q&A metric.

    Source: §1 · p. 2, §2.2 · p. 4, §3.2 · p. 4

  • Results

    Even the best models achieve only about half accuracy on generation or editing, while inference-time reasoning improves performance across models.

    Source: §1 · p. 3, §5.2 · p. 9

  • Takeaways & Limitations

    The authors conclude that structured visuals remain a substantial challenge and that inference-time reasoning benefits their generation and editing.

    Source: §1 · p. 3, §6 · p. 10

Abstract

from arXiv · show

While modern visual generation models excel at creating aesthetically pleasing natural images, they struggle with producing or editing structured visuals like charts, diagrams, and mathematical figures, which demand composition planning, text rendering, and multimodal reasoning for factual fidelity. To address this, we present the first comprehensive, systematic investigation of this domain, encompassing data construction, model training, and an evaluation benchmark. First, we construct a large-scale dataset of 1.3 million high-quality structured image pairs derived from executable drawing programs and augmented with chain-of-thought reasoning annotations. Building on it, we train a unified model that integrates a VLM with FLUX.1 Kontext via a lightweight connector for enhanced multimodal understanding. A three-stage training curriculum enables progressive feature alignment, knowledge infusion, and reasoning-augmented generation, further boosted by an external reasoner at inference time. Finally, we introduce StructBench, a novel benchmark for generation and editing with over 1,700 challenging instances, and an accompanying evaluation metric, StructScore, which employs a multi-round Q\&A protocol to assess fine-grained factual accuracy. Evaluations of 15 models reveal that even leading closed-source systems remain far from satisfactory. Our model attains strong editing performance, and inference-time reasoning yields consistent gains across diverse architectures. By releasing the dataset, model, and benchmark, we aim to advance unified multimodal foundations for structured visuals.

1 INTRODUCTION

Structured visuals require factual accuracy through composition planning, precise text rendering, and multimodal reasoning, yet current generation and editing models struggle with them. The paper introduces a dataset, unified model, and benchmark, and reports strong editing performance from its model and consistent gains from inference-time reasoning.

1 INTRODUCTION

  • Structured visuals pose challenges beyond aesthetics: they require factual accuracy, including composition planning, precise text rendering, and multimodal reasoning.The paper describes a gap between visual generation and understanding and notes that existing work and evaluation largely focus on natural images.Source: §1 · pp. 2–3
  • The 1.3M image-pair dataset aligns images with source code and editing actions, and adds GPT-5-generated reasoning annotations for generation and editing.The code-driven edits yield precise, verifiable state transitions.Source: §1 · p. 2
  • The unified model aligns Qwen-VL multimodal features with FLUX.1 Kontext through a lightweight MLP connector and uses a three-stage curriculum.The stages progressively support feature alignment, knowledge infusion, and reasoning-augmented generation.Source: §1 · p. 2
  • The paper introduces StructBench, with over 1,700 samples across six structural categories, and StructScore, which assesses fine-grained visual details through multi-round Q&A.The protocol compares model-generated answers with ground-truth responses to produce an overall score.Source: §1 · p. 2
  • Across evaluations of 15 open- and closed-source models on text-to-image and editing benchmarks, closed-source models substantially outperform open-source ones, while even state-of-the-art systems remain far from satisfactory.Source: §1 · p. 2
  • Inference-time reasoning improves other models as well, and the authors identify reasoning compute as a key bottleneck in current unified systems.The paper reports that adding explicit reasoning trajectories improves its editing performance and yields consistent gains when the same procedure is applied to other models.Source: §1 · pp. 2–3

2 STRUCTURED IMAGE DATASET

The dataset uses executable drawing programs to create code-aligned structured image pairs, with GPT-5 generating matched visual and program-level editing instructions. Filtering and reasoning annotations produce 1.3M examples intended to support structured image generation and editing.

2.1 DATA CURATION PIPELINE

  • The dataset challenge is the lack of open-source resources tailored to generative modeling that also support precise editing pairs.The paper proposes using source code as seed prompts, code-editing operations to obtain target code, and code-based graphics libraries to render images and pairs.Source: §2.1 · p. 3
  • Approximately 2M programs from diverse sources are collected and executed, retaining those that render valid structured images.The programs cover mathematics, charts, puzzles, scientific figures, graph diagrams, and tables, primarily in Python and LaTeX.Source: §2.1 · pp. 3–4
  • GPT-5 analyzes each source image, then generates paired image-editing and code-editing instructions before code edits are rendered into aligned target images.Image instructions refer to perceptible visual elements, while code instructions specify precise program changes, creating verifiable state transitions.Source: §2.1 · p. 4

2.2 POST-PROCESSING PIPELINE

  • Filtering removes invalid renders, visually null edits, and low-information images, yielding 1.3M examples with source–target pairs, captions, and editing instructions.The final examples span diverse categories.Source: §2.2 · p. 4
  • Each text-to-image example receives a dense caption, while each editing example receives a three-step reasoning chain covering image analysis, instruction interpretation, and target prediction.GPT-5 generates the annotations, which provide semantic and analytical signals for structured generation and editing.Source: §2.2 · p. 4

3 STRUCTBENCH

StructBench is curated across six structured-image domains and uses a fine-grained Q&A evaluation protocol designed for factual details and domain differences. Its editing score emphasizes instruction following, and its refined Q&A set exceeds 95% accuracy on ground-truth images.

3.1 DATA SELECTION

  • The benchmark selects images from six domains—Math, Graph, Chart, Puzzle, Science, and Table—with stratified sampling and dual GPT-5 and human review.The selection process aims to ensure diversity and balance and checks image quality and instruction correctness.Source: §3.1 · p. 4

3.2 EVALUATION PROTOCOL

  • Structured-image evaluation must account for global layout, fine-grained details, lower emphasis on aesthetics, and differences between domains.These demands make CLIP, aesthetic, and naive VLM-based scores poorly suited to the setting; StructScore uses a controlled multi-turn workflow for instance-specific attributes.Source: §3.2 · p. 4
  • StructScore uses open-ended VLM answers to atomic questions and compares predicted answers with ground truth, rather than relying on yes/no judgments.The paper notes that yes/no answers have a 50% random-guessing ceiling; Q&A pairs are derived from descriptions of salient visual elements and attributes.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
  • Editing accuracy weights instruction-following accuracy at 0.9 and visual-consistency accuracy at 0.1 to counter the imbalance between those Q&A types.The protocol labels Q&As correct on both source and target as visual-consistency related; those incorrect on the source but correct on the target are instruction-following related.Source: §3.2 · p. 5
  • Human audits prompted refinement of compound Q&As into atomic questions with concise answers, nearly doubling the number of Q&A pairs.The paper attributes evaluator answer stochasticity to questions that bundle multiple attributes or relations.Source: §3.2 · pp. 5–6
  • The refined Q&A set yields over 95% StructScore accuracy on ground-truth images, compared with approximately 80% for the initial set.GPT-5 rewrites failing questions to make them clear and unambiguous.Source: §3.2 · p. 6

3.3 NUMERICAL STATISTICS

  • StructBench contains 1,714 items, with 32,031 editing and 37,941 generation Q&A pairs; over 87% of samples have more than ten questions.Fourteen percent of samples include more than 26 questions; the chart test set is further stratified into five categories by editing type.Source: §3.3 · p. 6

4 MODEL TRAINING

The model adapts FLUX.1 Kontext with a lightweight connector for higher-level multimodal understanding, then uses progressive training to introduce structured-visual knowledge and explicit reasoning.

4.1 MODEL ARCHITECTURE

  • A lightweight MLP connector aligns Qwen-VL multimodal features with FLUX.1 Kontext, adding higher-level semantic information needed for structured edits such as chart conversion.The design reduces training overhead and stabilizes optimization without sacrificing performance compared with learnable-query transformer-based connectors.Source: §4.1 · pp. 6–7

4.2 TRAINING PIPELINE

  • The three-stage curriculum progresses from feature alignment, through hybrid visual learning, to thinking enhancement as training difficulty increases.Stage 1 freezes the backbone and trains only the connector; Stage 2 fine-tunes the backbone and connector on structured and general-domain data; Stage 3 injects explicit reasoning.Source: Figure 5 · p. 7, §4.2 · p. 7
  • Stage 2 uses adaptive masking to downweight background and unchanged regions, which otherwise dominate structured images and edits.The stage mixes structured-visual data with high-quality general-domain datasets to preserve general capabilities.Source: §4.2 · p. 7
  • Stage 3 uses chain-of-thought annotations and long-context inputs so a VLM analyzes the image–text pair and predicts target content before guiding generation.This addresses plausible-looking semantic errors on complex tasks by making multimodal reasoning explicit at inference time.Source: §4.2 · p. 7

5 EXPERIMENTS

Across generation and editing evaluations, even leading systems remain far from satisfactory, while the paper’s model performs strongly on editing and explicit reasoning improves results. The experiments also assess StructScore’s alignment with human preferences.

5.2 RESULTS AND ANALYSIS

  • 55.98 is the highest StructEditBench score, while GPT-Image’s 49.58 exceeds Qwen-Image’s 41.03 on StructT2IBench; even the best models reach only about half accuracy.The comparisons cover editing and text-to-image benchmarks, respectively.Source: Table 1 · p. 8, Table 2 · p. 8, §5.2 · p. 9

5.1 EXPERIMENTAL SETUP

  • Across 15 models and six subtasks, the evaluation covers open- and closed-source text-to-image, editing, and unified systems.Models use official default settings; outputs with custom sizing are matched to ground-truth image proportions.Source: §5.1 · p. 8, §5.2 · p. 8

5.2 RESULTS AND ANALYSIS

  • Architecture variants do not yield a consistent winner, while training on the curated dataset produces strong gains over the original FLUX.1 Kontext.The performance gap between closed- and open-source models is larger on text-to-image than editing; the authors suggest text-to-image is more challenging because it must synthesize fine-grained attributes from scratch.Source: §5.2 · p. 9
  • Reasoning variants substantially outperform non-reasoning baselines, while complex chart-type conversions are harder than color changes or adding auxiliary lines.The more complex edits require understanding input proportions and ordering before generating the structurally specified target.Source: §5.2 · p. 9

5.4 HUMAN ALIGNMENT STUDY

  • StructScore correlates with human Elo preferences at r > 0.9 and substantially exceeds traditional metrics such as PSNR.The study uses a large-scale Elo rating study and Pearson correlation to compare metrics against human rankings.Source: Figure 6 · p. 10, §5.4 · p. 10

5.3 SCALING COMPUTE WITH EXPLICIT REASONING

  • 38.44 vs. 33.34: Bagel with the designed reasoning trajectories outperforms Bagel-Think, and most models improve with explicit inference-time reasoning.Unified models benefit more than specialized expert models such as HiDream-E1.1.Source: §5.3 · pp. 9–10, Figure 7 · p. 10, Figure 8 · p. 10

6 CONCLUSION

The conclusion presents a code-aligned dataset, a progressively trained unified model, and a fine-grained benchmark-metric suite for structured visuals.

6 CONCLUSION

  • The experiments show that leading models still struggle with structured visuals, while the approach sets a new standard for open-source models and benefits from inference-time reasoning.Future work could extend code-renderable coverage to molecular formulas, musical staff, and structured videos such as educational videos.Source: §6 · p. 10

A.1.1 GENERATIVE MODELS FOR VISUAL CONTENT

The related work traces advances in text-to-image generation and instruction-based editing, including the emergence of unified systems.

A.1.1 GENERATIVE MODELS FOR VISUAL CONTENT

A.1.2 DATASET AND BENCHMARKS FOR GENERATION AND EDITING

Prior image datasets often rely on synthetic or video-derived editing pairs that can be noisy. This dataset instead uses code–image alignment and explicit code-level edits for faithful, verifiable supervision.

A.1.2 DATASET AND BENCHMARKS FOR GENERATION AND EDITING

  • Prior editing datasets can be imprecise because expert-generated content and video-derived pairs introduce noise.Source: §A.1.2 · p. 16
  • The dataset aligns code with images and uses explicit code-level editing operations to enforce precise state transitions.Source: §A.1.2 · p. 16
  • This alignment provides faithful, verifiable supervision for both generation and editing.Source: §A.1.2 · p. 16

A.2 TRAINING DETAILS

The model remains competitive on general image editing, while the supplementary evaluations examine training, evaluator agreement, human alignment, weighting choices, generalization, and data-generation prompts.

A.2 TRAINING DETAILS

  • Training progresses through three stages, mixing natural and structured data and replacing short prompts with detailed chain-of-thought instructions in the final stage.Source: §A.2 · p. 16, Figure 9 · p. 17

A.3 ADDITIONAL EVALUATION RESULTS

  • GPT-5 and Qwen2.5-VL rankings generally align, while PSNR and SSIM are unreliable for semantic correctness in structured visuals.Source: §A.3 · p. 17
  • Our model is competitive with state-of-the-art systems on ImgEdit and outperforms the original FLUX.1 Kontext.Source: §A.3 · p. 17

A.4 DETAILED HUMAN ALIGNMENT STUDY

  • StructScore Accuracy correlates more strongly with human evaluation overall (Pearson r = 0.87) than PSNR (r = 0.69) and SSIM (r = 0.70), and reaches r = 0.84 on chart-specific cases.Source: §A.4 · p. 19

A.5 ABLATION ON DIFFERENT WEIGHT SETTING

  • The 1:9 weighting achieved the highest correction coefficient for alignment with human preference, followed by 2:8.Source: §A.5 · p. 20

A.6 CASE STUDY

  • Qualitative examples show generalization to structurally complex images generated with Nano Banana 2, despite training only on code-rendered images.Source: §A.6 · p. 20

A.7 PROMPTS

  • The prompt appendix covers editing-pair construction, caption and reasoning generation, benchmark descriptions and Q&A, refinement, and ground-truth judgments.Source: §A.7 · p. 24
Loading 2510.05091v2…