PaperlayerView exact revision on arXiv

Source-linked AI summary

Recursive Self-Improvement in Unified Multimodal Models

Huijuan Wang, Chufan Shi, Cheng Yang, Yaokang Wu, Taylor Berg-Kirkpatrick, Xuezhe Ma

arXiv:2610.03002v1cs.CLcs.CV

TL;DR

Existing UMM self-improvement uses visual understanding to reward or filter image generation, without involving the model’s text abilities. RSI has the model diagnose generated images, write programs targeting its shortcomings, and use execution-verified data to train generation, understanding, and program writing. After four RSI rounds, chart-generation scores on unseen request wording rise from 45.7% to 60.2%, while continued training on the original data stays near 46%.

  • Problem

    Existing UMM self-improvement uses visual understanding to reward or filter image generation, without involving the model’s text abilities.

    Source: §1 · p. 1

  • Method

    RSI has the model diagnose generated images, write programs targeting its shortcomings, and use execution-verified data to train generation, understanding, and program writing.

    Source: §1 · pp. 1–2

  • Results

    After four RSI rounds, chart-generation scores on unseen request wording rise from 45.7% to 60.2%, while continued training on the original data stays near 46%.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Verified construction carries most of the chart-generation gain, and targeting model failures adds 3.5%.

    Source: §1 · p. 2, §6 · p. 9

    On the chart subset of StructT2IBench, RSI raises the score only from 11.0% to 11.9%, and after four rounds the reader is lower by 2.6% on MMBench and 3.0% on MMMU.

    Source: §4.5 · p. 8

Abstract

from arXiv · show

Unified multimodal models (UMMs) understand and generate both text and images, which lets a model produce its own training data. Existing self-improvement in UMMs keeps supervision on the visual side, where image understanding judges image generation. We propose recursive cross-capability self-improvement (RSI), a training loop in which the text and visual abilities of a UMM supply training data for one another. In each round, the model generates images and reads them to find where it falls short. It then writes programs aimed at these shortcomings, and execution verifies every result against its specification. Verified renders train image generation, while labeled renders and the model's own correct programs train visual understanding and program writing. Program execution thus acts as a source of truth outside the model, so errors do not accumulate across rounds. We study RSI on charts and build BasicChartBench to evaluate open models early in training. On requests worded differently from training, four rounds of RSI raise the score from 45.7% to 60.2%, while continued training stays at 46.3%. Verified construction carries most of the gain, and targeting the model's failures adds 3.5%. Along the way, the share of verified programs rises from 48.9% to 95.2%, and the reader's accuracy on edited renders rises from 55.6% to 87.4%.

1 INTRODUCTION

RSI lets a UMM’s text and visual abilities supply training data for one another, using program execution as an independent check. On chart requests with wording unseen in training, four rounds raise the score from 45.7% to 60.2%, while continued training stays at 46.3%.

Motivation

  • Existing UMM self-improvement uses visual understanding to reward or filter image generation, leaving text abilities out of the supervision loop.The paper asks whether text and visual abilities can instead improve one another in a closed loop.Source: §1 · p. 1

Approach

  • RSI diagnoses the model’s generated images, writes programs targeting identified shortcomings, and uses execution to verify the resulting training data.Verified renders train generation; labeled renders and correct programs train visual understanding and program writing.Source: §1 · pp. 1–2, Figure 1 · p. 2

Evaluation

  • BasicChartBench evaluates chart generation at two difficulty levels because existing benchmarks are too difficult to distinguish open models early in training.Charts require exact text, values, and structure.Source: §1 · p. 2

Results

  • 45.7% to 60.2% on BasicChartBench L0 requests with unseen wording after four RSI rounds, versus 46.3% with continued training.Targeting model failures adds 3.5% over spreading the same data evenly.Source: §1 · p. 2
  • Verified construction carries most of the gain; the share of verified programs rises from 48.9% to 95.2%, and reader accuracy on edited renders rises from 55.6% to 87.4%.Training on the model’s own best-rated images gains only 0.5%, compared with 8.6% for verified data.Source: §1 · p. 2

2 METHOD

Each RSI round diagnoses generated charts, uses that diagnosis to plan verified examples, and updates the model’s generation and understanding abilities. Execution supplies images and exact records for checking programs and labels.

2.1 SETUP

  • The model has image-generation, visual-understanding, and program-writing abilities; the reader and writer share understanding parameters, while generation uses separate expert parameters.The execution environment runs programs and records what they draw.Source: §2.1 · p. 3

2.2 EXECUTION AS A SOURCE OF TRUTH

  • Execution runs a program to produce an image and record, then verifies the record against the specification before accepting generation targets.It checks data, text, colors, axes, layout, and readability; labeled altered renders and verified programs also train understanding and writing.Source: §2.2 · p. 3

2.3 DIAGNOSING THE MODEL’S OWN GENERATIONS

  • The current generator produces diagnostic images, and the reader marks each requirement met, violated, or unresolved to estimate generation accuracy.Correct diagnostic renders calibrate the reader, while held-out edited renders track whether it reports image content rather than request intent.Source: §2.3 · p. 3

2.4 FROM DIAGNOSIS TO VERIFIED DATA

  • The diagnosis sets a sampling plan that allocates examples across chart types and weak capabilities, with fixed shares, capped error-based weights, and difficulty levels adjusted at accuracy thresholds.Attribute-level rules introduce rare values, add examples for weak common values, and pair errors with weaker request wordings.Source: §2.4 · pp. 3–4
  • The writer produces two programs per sampled specification; verified candidates are retained, and low writer pass rates can trigger extra specifications and program examples for understanding data.Extra writer-focused specifications are capped at 20% of a round’s data.Source: §2.4 · p. 4

2.5 UPDATES AND RECURSION

  • RSI alternates generation and understanding updates, then uses the updated model’s generator, reader, and writer to build the next round’s examples.At inference, images are drawn directly; programs are used only to build training data.Source: §2.5 · p. 4

3 BASICCHARTBENCH

BasicChartBench is designed to distinguish open chart-generation models early in training, when existing benchmarks are often too difficult. It uses atomic questions and a fixed reader for scoring.

Motivation

  • BasicChartBench targets early training because existing chart benchmarks rank strong commercial models but separate early-stage open models poorly.The starting model answers about 11% of StructT2IBench chart questions and 4% of BizGenEval chart questions correctly.Source: §3 · p. 4

Levels

  • The benchmark has L0 with 336 charts and 10,140 questions, and L1 with 140 charts and 4,300 questions; L1 adds rare-topic vocabulary and number formats beyond the curriculum.L0 uses familiar vocabulary with request wording absent from training; benchmark requests do not share content with training specifications.Source: §3 · p. 4

Questions and scoring

  • Each request is scored through atomic questions covering chart structure, exact text, and value-color-order binding, with both levels using the same specification-derived scoring rules.The initial reader U0 scores all models by fixed criteria; an external judge and OCR character similarity provide checks independent of that reader.Source: §3 · p. 4

4 EXPERIMENTS

The experiments test RSI across chart-generation accuracy, its component mechanisms, changes in program writing and visual reading, and remaining limitations. Across four rounds, generation accuracy rises, while controls and component analyses probe the contribution of verification, diagnosis, and recursion.

4.1 EXPERIMENTAL SETTING

  • The setup uses a curriculum-trained BAGEL-7B-MoT, seven verifiable chart types, and 336 diagnostic requests; controls isolate extra training, self-filtering, fixed data allocation, and non-updating diagnosis or reader.All strategies use the same training budget; the self-filtered control trains on the higher-scored of two generated images.Source: §4.1 · p. 5

4.2 DOES THE LOOP KEEP IMPROVING?

  • 59.0% pooled accuracy after round 4, up from 45.5% at the start; continued training reaches 46.3%, while RSI reaches 60.2% on L0 and 56.2% on L1.The trend holds across both BasicChartBench difficulty levels, and the external judge reports rising scores on each level.Source: §4.2 · p. 5, Figure 3 · p. 6, Table 1 · p. 6

4.3 WHICH PARTS DRIVE THE GAIN?

  • Verified examples drive most of the gain: the first-round verified-data generator gains 8.6% on L0 and 4.2% on L1, while self-filtering gains 0.5% and 1.2%.Targeted diagnosis adds to verified construction: RSI leads fixed-distribution data by 2.8–4.1% on L0 and 1.7–5.5% on L1; on L1, RSI also leads single diagnosis by 3.1 and 3.7% in rounds 3 and 4.Source: §4.3 · p. 6

4.4 HOW DO THE TEXT AND VISUAL ABILITIES CHANGE?

  • Verified-program coverage rises from 48.9% of specifications at the start to 95.2% after one round, and stays between 95% and 97% later.The first-round writer passes verification on second-round specifications at the same 90.8% rate whether its correct programs are sampled uniformly or more often for weak values.Source: §4.4 · p. 7
  • On held-out edited renders, accuracy on questions whose answers change rises from 55.6% to 86.5% after one round; on the model’s own generations, later readers’ violation-rate rankings correlate 0.86–0.91 with human rankings.Despite improved ranking, every reader still accepts about a third of requirements humans mark as not shown.Source: §4.4 · pp. 7–8, Table 2 · p. 8

4.5 WHERE DOES THE LOOP FALL SHORT?

  • On BasicChartBench’s unseen wording, structure accuracy reaches 86.1% after round 4, but chart type and panel count still drift; the external judge sees structure recover fully.On the diagnostic set with training-matched wording, structure accuracy rises from 82.5% to 90.3%.Source: §4.5 · p. 8
  • Replacing the vocabulary in round 3 drops old-vocabulary category-text accuracy from 72% to 60%; retaining half the earlier vocabulary in round 4 brings old words back the next round.The authors conclude that the curriculum should widen training data over time rather than replace it.Source: §4.5 · p. 8
  • On existing chart benchmarks, RSI raises StructT2IBench from 11.0% to 11.9%, while general understanding falls 2.6% on MMBench and 3.0% on MMMU after four rounds.GenEval stays between 83.7 and 84.5 after each round, compared with 82.9 at the start; the authors report only 5% general visual data in each reader-data round.Source: §4.5 · p. 8, Table 3 · p. 9

5 RELATED WORK

Unified multimodal models connect visual understanding and generation, including through generated images used as intermediate representations. Prior work on self-improvement, cross-modal feedback, and synthetic data motivates studying how understanding can shape later training examples while execution verifies them.

Unified Multimodal Models

  • Unified multimodal models link visual understanding and generation, and interleaved reasoning can use generated images as intermediate representations.The paper examines how understanding a model’s own visual outputs informs subsequent training data.Source: §5 · p. 9

Recursive Self-Improvement

  • Prior self-improvement work uses verification or learned evaluation, while multimodal approaches add supervision from cross-modal consistency and visual experience.This paper focuses on how learning to recognize generation errors changes later examples.Source: §5 · p. 9

Synthetic Data Generation

  • Programmatic construction supports verification, but verifying examples alone does not establish their usefulness to the current model.The paper combines model-based error diagnosis with executable records to select needs and verify examples.Source: §5 · p. 9

6 CONCLUSION

The paper concludes that recursive cross-capability self-improvement can improve chart generation while program execution keeps training data correct. Its experiments start from a staged BAGEL-7B-MoT checkpoint, M0.

6 CONCLUSION

  • 45.7% to 60.2% on requests worded differently from training after four rounds, while continued training on original data stays near 46%.Verified construction carries the gain, targeting model failures adds 3.5%, and program writing and reading also improve.Source: §6 · p. 9
  • RSI has the text and visual abilities produce training data for one another, with program execution keeping that data correct.The paper presents this as a broader goal for models whose abilities continue supplying one another with correct training data.Source: §6 · p. 9

A.1 STARTING MODEL

  • M0 starts from BAGEL-7B-MoT and is built through staged training of program writing, generation, and visual understanding.The selected checkpoint follows 14,000 updates of the generation-training stage; the reader is U0.Source: §A.1 · p. 12

A.2 TRAINING IN EACH ROUND

Each RSI round updates generation and understanding from the previous merged model, using verified examples and replay alongside other data. Program construction verifies outputs against specifications, and decision-rule thresholds are fixed before training rounds begin.

A.2 TRAINING IN EACH ROUND

  • Generation training uses verified round targets alongside replay, general images, and editing pairs; understanding training combines questions, error reports, code, and general visual pairs.The generation update runs 400 steps; the understanding update runs 704 steps with generation parameters frozen, and half its code examples are the round’s verified programs.Source: §A.2 · p. 12

A.3 CONSTRUCTION AND VERIFICATION

  • Each round plans about 36,000 specifications and keeps the first verified candidate per specification after sandbox execution checks chart content and layout.Low writer success can trigger up to 20% more specifications; verification checks data, text, colors, order, axes, labels, legends, layout, and readability.Source: §A.3 · p. 12

A.4 DECISION RULES

  • The decision-rule thresholds were all fixed before the first round.The passage refers to Table 4 for the thresholds but does not state their values.Source: §A.4 · p. 12

B REQUEST SETS AND BASICCHARTBENCH

BasicChartBench and a diagnostic set use specification-derived questions to assess chart requirements across structure, text, and binding. The sets vary wording, vocabulary, chart complexity, and number formats, with no training specification sharing a content fingerprint with the three test sets.

Request sets

  • The diagnostic set, BasicChartBench L0, and BasicChartBench L1 contain 336, 336, and 140 requests, respectively, with no test request sharing a training content fingerprint.Each set derives atomic questions from specifications and applies the same scoring rules.Source: B REQUEST SETS AND BASICCHARTBENCH · p. 12

Diagnostic set

  • The diagnostic set uses first-stage vocabulary and varies chart properties, wording, and number formats across its requests.It has 9,636 questions, and after round four adds a second-vocabulary portion with 4,814 questions for the data plan and vocabulary gate.Source: B REQUEST SETS AND BASICCHARTBENCH · p. 13

BasicChartBench L0

  • BasicChartBench L0 uses familiar first-stage vocabulary but request wording absent from training, and includes charts with more categories than the curriculum trains.It contains 10,140 questions.Source: B REQUEST SETS AND BASICCHARTBENCH · p. 13

BasicChartBench L1

  • BasicChartBench L1 uses rarely seen topic words and number formats not yet introduced by the curriculum.It includes 14 topics, one to four series, and 4,300 questions.Source: B REQUEST SETS AND BASICCHARTBENCH · p. 13

Question coverage

  • Atomic questions cover chart structure, visible text, and the bindings between marks, values, colors, and series or category order.Binding questions also include value labels and simple reasoning over the data.Source: B REQUEST SETS AND BASICCHARTBENCH · p. 13

C EVALUATION PROTOCOL AND EXTERNAL JUDGE

The evaluation compares the frozen initial reader with an external judge using the same deterministic scoring rules. They agree on most trends across BasicChartBench levels, though their structural scores differ and the judge is stricter on text.

C EVALUATION PROTOCOL AND EXTERNAL JUDGE

  • The external judge answers each atomic question separately, with thinking disabled, and its answers use the reader's deterministic scoring rules.OCR uses PaddleOCR with rotated reading for vertical axis titles.Source: C EVALUATION PROTOCOL AND EXTERNAL JUDGE · p. 13
  • The external judge shares no weights or training data with the model, providing a comparison with the frozen initial reader U0.Figure 7 plots both scorers on the two BasicChartBench levels; scores are satisfaction rates on atomic questions.Source: C EVALUATION PROTOCOL AND EXTERNAL JUDGE · p. 14, Figure 7 · p. 14
  • On L0, both scorers raise overall, text, and binding scores over four rounds, but they differ on structure; on L1, both raise the same three score groups.For example, L0 overall rises from 45.7% to 60.2% for U0 and from 37.4% to 57.9% for the judge.Source: C EVALUATION PROTOCOL AND EXTERNAL JUDGE · p. 14
  • The judge is stricter on text, so the authors compare trends rather than absolute score levels.On structure, the judge sees scores exceed baseline by round 3, while U0 remains below baseline on L0 after round 4.Source: C EVALUATION PROTOCOL AND EXTERNAL JUDGE · p. 14

D HUMAN AUDIT OF NATIVE GENERATIONS

The human audit tests how well readers judge the model's own generated charts, using blinded labels kept separate from RSI training. Reader accuracy improves modestly, while its ranking of weaknesses aligns well with human rankings.

D HUMAN AUDIT OF NATIVE GENERATIONS

  • Human labels are collected after the final round and kept outside diagnosis, specification construction, training updates, and the thresholds.RSI runs identically whether or not the audit exists.Source: D HUMAN AUDIT OF NATIVE GENERATIONS · p. 14
  • The audit uses 392 questions on 196 native images generated by M0 through the round-3 model, with each model contributing 49 images.Humans label requirements as shown, not shown, or unreadable.Source: D HUMAN AUDIT OF NATIVE GENERATIONS · p. 14
  • Reader accuracy on native generations rises from 78.5% for U0 to 82.4% for U3, but the 3.9% gain has a 95% interval of [0.0, 7.9].The gain comes from fewer false rejections; false acceptances do not fall, and binding accuracy stays near 75%.Source: D HUMAN AUDIT OF NATIVE GENERATIONS · p. 15
  • Reader-human correlations in weakness rankings rise from 0.82 for U0 to 0.86–0.91 for U1–U3, supporting the allocation rules' ranking-based diagnosis.About a third of requirements humans mark as not shown are still accepted by U0 and U3.Source: D HUMAN AUDIT OF NATIVE GENERATIONS · p. 15
  • Before RSI, the initial reader agreed with 76.2% of 210 labels and mostly accepted text labeled not shown, prompting the authors to report OCR alongside reader text results.These development-set labels are not part of any reported result.Source: D HUMAN AUDIT OF NATIVE GENERATIONS · p. 15

E ADDITIONAL RESULTS

The additional-results tables provide reading-quality comparisons across render types and a chart-type breakdown of program construction.

E ADDITIONAL RESULTS

  • Table 6 breaks program-writing results down by chart type.Table 5 provides the full reading results behind the left and middle panels of Figure 6.Source: E ADDITIONAL RESULTS · p. 15
  • Table 5 reports reading quality for each reader on correct, edited, and native renders.Correct renders contain 9,636 questions; edited renders are held-out first-round families, and native renders cover 392 questions on generations from M0 to M3.Source: Table 5 · p. 15
  • Edited renders are shared across readers, and the table identifies the best value in each column.The edited images come from held-out families on which no round trains.Source: Table 5 · p. 15

F DATA DISTRIBUTION CHANGES

The study changed some data-distribution rules during the experiment, documenting when and why. Later rules addressed a vocabulary-related loss but affect rounds not reported in the paper.

F DATA DISTRIBUTION CHANGES

  • The study lists every reported-round-affecting rule change, when it occurred, and why.Source: F DATA DISTRIBUTION CHANGES · p. 15
  • Before round 1 training, prompt-format pairing was changed after the fixed probability sent 92% of prompts to one format.The mixed-format share was then held between 40% and 60%; the content checker was also fixed for line charts plotting category names directly.Source: F DATA DISTRIBUTION CHANGES · p. 15
  • Each specification receives two candidate programs, and verification counts a specification when at least one candidate matches it.Each round uses about 35,000–43,000 specifications; round 1 adds about 30,000 more for pairs that fell short, excluded from the table.Source: Table 6 · p. 16
  • After round 3 replaced the old vocabulary, exact category text on the old vocabulary fell from 72% to 60% on the diagnostic set.From round 4, half of specifications retained an earlier vocabulary when a new one opened.Source: F DATA DISTRIBUTION CHANGES · p. 16
  • Additional vocabulary diagnostics and a floor for the old vocabulary were introduced after round 4, affecting only later, unreported rounds.Source: F DATA DISTRIBUTION CHANGES · p. 16

Related papers

Loading 2610.03002v1…