PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniTaskonomy: When Does Visual Generation Improve Visual Understanding?

Jiaxin Ge, Yiming Qin, Ji Xie, Haozhe Jiang, Xiaochuang Han, Junyi Zhang, Andrew Dai, Yinfei Yang, Jitendra Malik, Ranjay Krishna, Sewon Min, Haiwen Feng, Le Xue, Baifeng Shi, Trevor Darrell, XuDong Wang

arXiv:2609.38079v1cs.CV

TL;DR

The paper asks when visual generation supervision improves visual understanding, a transfer direction whose benefits remain unclear. It tests controlled paired tasks, maps transfer across generation and understanding capabilities, and analyzes gradient alignment. It finds that the training recipe and task pairing matter, with stronger alignment associated with larger transfer gains.

  • Problem

    The benefits of visual generation supervision for visual understanding remain unclear, despite generation’s dense supervision of appearance, spatial relationships, and geometry.

    Source: §1 · p. 1

  • Method

    The study pairs I2I and I2T tasks, introduces OmniTaskonomy to map transfer across capabilities, and measures gradient alignment between generation and understanding objectives.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    I2I followed by I2T training yields gains that grow with I2I data, transfer varies by capability, and stronger gradient alignment is associated with larger transfer gains.

    Source: §1 · pp. 2–3, §7 · p. 10

  • Takeaways & Limitations

    Generation objectives should be selected for the visual capabilities they support and trained with a recipe that lets their supervision benefit the target understanding task.

    Source: §1 · p. 3, §7 · p. 10

    The reported alignment analysis is scoped to controlled paired tasks and transfer across the capabilities and task pairs studied in OmniTaskonomy.

    Source: §6 · p. 8

Abstract

from arXiv · show

Training a model to generate visual content can encourage it to learn rich perceptual capabilities related to geometry, spatial relationships, and objectness; yet, its benefits for visual understanding remain unclear. We ask: when and how does visual generation supervision improve visual understanding? We study controlled pairs of image-to-image (I2I) generation and image-to-text (I2T) understanding tasks that express the same underlying problem in different output modalities. We find that under the correct recipe, I2I training improves downstream I2T performance, with larger gains as the amount of I2I training data increases. We next ask which generation tasks benefit which understanding capabilities. To study transfer beyond paired tasks, we introduce OmniTaskonomy, a unified taxonomy spanning 19 I2I generation tasks and 25 I2T understanding capabilities. The resulting transfer map reveals selective, task-dependent benefits. Some follow intuitive correspondences, e.g., depth prediction improving metric 3D reasoning, object pointing improving counting, and jigsaw reconstruction improving 2D ordering. Interestingly, we also uncover surprising connections: 2.5D segmentation improving category recognition and Z-depth prediction improving localization. To probe these patterns, we analyze gradient alignment between generation and understanding tasks and find that stronger alignment is associated with larger downstream transfer gains. Together, our results highlight visual generation as a rich source of supervision for visual understanding and provide a roadmap for unlocking its benefits through the right training curriculum and task selection. Project page: https://omni-taskonomy.github.io/.

1 INTRODUCTION

The paper asks when visual generation supervision can improve visual understanding, since the reverse transfer is less clear despite generation’s dense perceptual supervision. It studies training curricula, task-specific transfer, and gradient alignment, finding that benefits depend on both training recipe and shared visual capabilities.

1 INTRODUCTION

  • The study frames its investigation around which curricula enable transfer, which generation tasks support which understanding capabilities, and whether gradient alignment signals successful transfer.The motivation is that generation supplies pixel-level supervision for appearance, spatial relationships, and geometry relevant to understanding tasks.Source: §1 · pp. 1–2
  • I2I training followed by I2T fine-tuning yields gains that grow with additional I2I data, while directly mixing objectives does not show the same scaling benefit.I2I training also reduces the I2T supervision needed for a given performance level, especially when I2T data is limited.Source: §1 · p. 2
  • OmniTaskonomy maps transfer between generation tasks and understanding capabilities, revealing intuitive links and less expected gains from 2.5D segmentation to category recognition and Z-depth prediction to localization.Examples of intuitive links include Z-depth improving metric 3D reasoning and object pointing improving counting.Source: §1 · p. 2
  • Higher mean gradient alignment across generation tasks is associated with larger mean transfer gains across understanding capabilities and individual source-target pairs.In controlled pairs, alignment is strongest in early pre-attention normalization parameters.Source: §1 · p. 3
  • The authors propose establishing a useful initialization through generation training, selecting tasks for the capabilities they support, and investigating gradient alignment as a signal for promising pairings.They present OmniTaskonomy as making these transfer relationships explicit.Source: §1 · p. 3

2 RELATED WORK

Prior work studies unified multimodal models and generation-understanding interactions, but existing evaluation frameworks do not necessarily identify shared visual capabilities across output modalities. OmniTaskonomy addresses this by organizing generation tasks and understanding capabilities in a common capability-based taxonomy.

2 RELATED WORK

  • Omni models unify multimodal understanding and generation, and this study evaluates transfer under a Mixture-of-Transformers architecture using BAGEL as its baseline.Prior architectures include fully autoregressive discrete-token models and hybrids combining autoregressive language modeling with diffusion.Source: §2 · p. 3
  • Prior training studies combine generation and understanding data or add generation tasks during post-training, while another line uses understanding signals to supervise generation.This study instead measures how individual I2I tasks affect individual visual understanding tasks.Source: §2 · p. 3
  • Shared evaluation suites or interleaved multimodal tasks do not by themselves establish which I2I and I2T tasks depend on the same visual capability.OmniTaskonomy places both task types in one capability-based taxonomy to analyze transfer at the level of individual visual capabilities.Source: §2 · p. 3

3 DOES VISUAL GENERATION HELP VISUAL UNDERSTANDING?

Controlled paired tasks test generation-to-understanding transfer while holding the input and visual problem fixed and varying output modality. The results favor initial shared-parameter I2I training followed by I2T learning, with transfer also reducing the I2T data needed and extending to paired examples.

3 DOES VISUAL GENERATION HELP VISUAL UNDERSTANDING?

  • The controlled Jigsaw and Zoom-In tasks require the same visual operation on the same input, with image output for I2I and permutation text for I2T.The paired evaluation tests transfer exclusively on the corresponding I2T task.Source: §3 · pp. 3–4, Figure 2 · p. 4
  • An initial I2I stage that updates parameters shared with I2T provides a useful initialization for subsequent I2T learning.Recipes beginning with shared-parameter I2I training scale consistently as I2I data increases; freezing those shared weights or mixing objectives from the outset yields weaker or less stable gains.Source: §3 · p. 4
  • 100k I2I examples followed by 1k I2T examples on Zoom-In, or 3k I2T examples on Jigsaw, achieve performance comparable to 10k I2T examples alone.I2I improves performance across the tested I2T budgets, with larger gains at smaller budgets.Source: §3 · pp. 4–5
  • Paired I2T predictions are more accurate on examples whose corresponding I2I prediction is correct than on examples whose I2I prediction is wrong.Both outputs are evaluated from the same checkpoint and input image and question.Source: §3 · p. 5

4 OMNITASKONOMY: A UNIFIED TAXONOMY OF VISUAL CAPABILITIES

OmniTaskonomy places image-to-image generation tasks and image-to-text understanding capabilities in one hierarchy organized by shared visual capabilities. It contains 19 generation tasks and 25 understanding capabilities, with benchmark samples assigned to the understanding leaves.

4 OMNITASKONOMY: A UNIFIED TAXONOMY OF VISUAL CAPABILITIES

  • OmniTaskonomy places I2I objectives and I2T capabilities in the same 3R hierarchy according to the visual capability each supervises or requires.The top-level families are Recognition, Reconstruction, and Reorganization; generation objectives and understanding capabilities remain separate leaves.Source: §4 · pp. 5–6
  • Fine-grained understanding capabilities were derived from seven vision-language benchmarks through attribute annotation, tree construction, and manual refinement.The resulting leaves were placed under the 3R families after synonymous leaves were combined and mixed or redundant categories were refined.Source: §4 · p. 6
  • The final taxonomy contains 19 I2I tasks and 25 understanding capabilities, with tasks assigned by the visual capability directly supervised by each objective.For example, Z-depth and Euclidean depth prediction sit under Reconstruction, while 2D keypoint prediction sits under Reorganization.Source: §4 · p. 6
  • Three VLM judges assigned eligible benchmark samples to taxonomy leaves by majority vote, yielding 9,444 samples; human review found 97.4% label agreement among definite judgments.Human-VLM agreement had mean Cohen’s κ = 0.972, and human inter-annotator agreement was Krippendorff’s α = 0.943 on examples reviewed by all four annotators.Source: §4 · p. 6
  • Figure 4 maps I2I source tasks by column against understanding capabilities by row, with cell colors encoding transfer direction and outlines marking significant gains.Cells show I2T accuracy change in percentage points versus the I2T-only baseline; results are averaged over three seeds.Source: Figure 4 · p. 7

5 WHICH GENERATION TASKS HELP WHICH CAPABILITIES?

The transfer map shows selective, task-dependent benefits from I2I training across understanding capabilities. Gains often track shared visual operations, but some generation tasks also benefit less directly related targets.

5.1 EXPERIMENTAL SETUP

  • The study evaluates all 19 I2I tasks against 25 understanding capabilities using roughly 50k I2I examples followed by 50k LLaVA-Instruct examples, with I2T-only training as the baseline.Transfer is measured as accuracy change in percentage points on the same 9,444 evaluation examples; significance is tested against the baseline.Source: §5.1 · p. 7, §5 · p. 7

5.2 TRANSFER IS SELECTIVE ACROSS SOURCE AND TARGET TASKS

  • Five of the 19 displayed capabilities improve significantly with at least one I2I source, while several others show little or no positive transfer; metric 3D relation and counting benefit from 12 sources each.2D ordering improves with eleven sources, whereas OCR, text recognition, and appearance understanding show no improvement from any tested source.Source: §5.2 · p. 7
  • Localization and object pointing yield the largest counting gains (+2.5 and +2.0 pp), while Z-depth, Euclidean depth, and surface normals improve metric 3D relation by 3.6, 3.8, and 3.4 pp.Jigsaw improves 2D ordering by 6.8 pp; these cases share spatial localization, geometry, or arrangement operations with their targets.Source: §5.2 · p. 8
  • Inpainting improves counting (+1.5 pp) and 2D ordering (+7.2 pp), while 2.5D segmentation improves category recognition (+1.2 pp).The authors suggest inpainting may encourage inference of object quantity and global spatial structure, and that object-boundary cues may support category recognition.Source: §5.2 · p. 8
  • The reported examples include significant gains both within and across task families, indicating that useful transfer is not limited to closely matched capabilities.The authors’ finding highlights localization and object pointing for counting, and depth and surface-normal prediction for metric 3D relation.Source: §5.2 · p. 8

6 WHAT EXPLAINS VISUAL GENERATION-TO-UNDERSTANDING TRANSFER?

The paper measures gradient alignment between generation and understanding objectives, first in controlled task pairs and then across OmniTaskonomy. Alignment is strongest in early pre-attention normalization layers and is positively associated with downstream transfer across capabilities and task pairs.

6 WHAT EXPLAINS VISUAL GENERATION-TO-UNDERSTANDING TRANSFER?

  • The authors hypothesize that transfer is stronger when generation and understanding objectives favor similar parameter updates, then test that relationship in controlled pairs and across tasks.The hypothesis is motivated by prior work on conflicting task gradients and gradient-based task affinity.Source: §6 · p. 8, §6.1 · p. 9

6.1 MEASURING GRADIENT ALIGNMENT

  • The alignment score compares normalized I2I and I2T gradient directions in a shared PCA subspace and scales their cosine similarity by estimated effective dimensionality.The shared PCA basis retains the fewest components whose eigenvalues account for at least 99% of total variance; positive scores indicate stronger directional alignment and negative scores opposing directions.Source: §6.1 · p. 9

6.2 GRADIENT STRUCTURE IN THE CONTROLLED SETTING

  • In controlled Jigsaw and Zoom-In pairs, alignment is strongest in the understanding branch’s pre-attention RMSNorm parameters, particularly in earlier transformer layers.The study samples 500 matched examples per task and computes gradients at the pretrained checkpoint.Source: §6.2 · p. 9

6.3 DOES GRADIENT ALIGNMENT TRACK TRANSFER ACROSS TASKS?

  • Across OmniTaskonomy, the analysis focuses on pre-attention RMSNorm parameters across 28 transformer layers and compares mean post-PCA gradient cosine with transfer gain.It covers all 19 I2I tasks and seven understanding capabilities with more than 500 evaluation examples, measuring gradients at the pretrained checkpoint and transfer after I2I training followed by I2T finetuning.Source: §6.3 · p. 9
  • Gradient alignment is positively associated with downstream transfer across understanding capabilities (r = 0.795) and across 133 source–target pairs (r = 0.529).The authors identify alignment as an optimization-level signal associated with generation-to-understanding transfer.Source: §6.3 · pp. 9–10

7 CONCLUSION

The conclusion finds that visual generation can improve understanding under an appropriate training recipe, with transfer varying by capability and relating positively to gradient alignment. It recommends choosing generation objectives for the capabilities they support and training them with a recipe that benefits the target task.

7 CONCLUSION

  • I2I training followed by I2T finetuning improves understanding, with gains growing with I2I data and especially pronounced when I2T data is limited.Transfer is capability-specific and occurs both within and across task families; gradient alignment is strongest in early pre-attention normalization layers in the controlled setting and is positively associated with transfer.Source: §7 · p. 10
  • The study recommends selecting generation tasks for the capabilities they support and using a training recipe that lets this supervision benefit the target understanding task.It also proposes investigating gradient alignment as a signal for promising task pairings.Source: §7 · p. 10

APPENDIX CONTENTS

  • The appendix contents list further materials on recognition editing-data routing.The conclusion also points to OmniTaskonomy as making generation–understanding relationships explicit.Source: §7 · p. 10, APPENDIX CONTENTS · p. 16

A CONTROLLED GENERATION-TO-UNDERSTANDING STUDY

  • The controlled study uses paired Jigsaw and Zoom-In I2I and I2T tasks that share each input and differ in whether the answer is an image or text.The appendix provides training details, model architecture, and instance-level evaluation for this study.Source: A CONTROLLED… · p. 17

A.1 TRAINING SETTING

The controlled experiments finetune BAGEL-7B-MoT and compare recipes that vary I2I/I2T staging, mixing, and whether shared parameters are frozen. The appendix also describes model gradient flow, instance-level scoring, and the three-stage OmniTaskonomy construction pipeline.

A.1 TRAINING SETTING

  • Three-seed mean scores are reported for BAGEL-7B-MoT Jigsaw and Zoom-In experiments, with Figure 2 shading indicating one standard error of the mean.Each multi-stage recipe loads the previous stage's weights and starts a fresh optimizer and learning-rate schedule; Table 2 lists shared optimization settings.Source: §A.1 · p. 17, Table 2 · p. 17
  • I2I conditioning dropout independently removes instruction text, source-image ViT tokens, or source-image VAE latents at probability 0.1, while retaining the target image; I2T uses none.Mixed batches preserve 16 I2T examples per GPU and add a fixed number of I2I examples; Table 3 lists the I2I pool size and batch ratio.Source: §A.1 · p. 17, Table 3 · p. 17

A.2 MODEL ARCHITECTURE AND TRAINABLE MODULES

  • BAGEL separates generation and understanding projections, normalization layers, and MLPs while connecting them through shared attention.I2I gradients update the generation pathway and can backpropagate through attention into understanding components; I2T gradients train the understanding pathway, and mixed training combines both losses.Source: §A.2 · p. 18

A.3 INSTANCE-LEVEL AGREEMENT

  • Instance-level evaluation pairs I2I and I2T predictions from the same checkpoint, input, and question, and reports I2T accuracy conditioned on I2I correctness.A VLM judge scores generated images; deterministic checks require all positions and source elements to match, with task-fulfillment and content-fidelity scores of at least 3/4.Source: §A.3 · pp. 18–19

B.1 OVERVIEW OF THE CONSTRUCTION PIPELINE

  • OmniTaskonomy is built through schema annotation, taxonomy refinement, and VLM verification.The pipeline extracts attributes, groups samples into local trees before merging and refining a global 3R taxonomy, then retains examples with at least two matching capability votes.Source: §B.1 · p. 19, Figure 7 · p. 19

B.2 FROM SAMPLE ATTRIBUTES TO A FIXED SCHEMA

OmniTaskonomy construction turns benchmark examples into schema-annotated samples, refines a 3R capability taxonomy, and verifies labels with VLM votes and human review. Its appendices describe capability and I2I inventories and the evaluation corpus coverage.

B.2 FROM SAMPLE ATTRIBUTES TO A FIXED SCHEMA

  • From 1,000 sampled examples, the schema process extracts 50–80 free-form attributes per example, consolidates them into 57 fields, and applies the fixed schema to all 10,422 samples.Attributes include image content, perceptual operations, difficulty, question–answer properties, and data quality.Source: §B.2 · p. 20, Figure 8 · p. 20

B.3 FROM CANDIDATE TREES TO THE REFINED 3R TAXONOMY

  • Taxonomy refinement merges ten local trees with 474 leaves into a 42-leaf candidate tree, then human review yields 35 leaves organized under Recognition, Reconstruction, and Reorganization.Review splits mixed capabilities, merges redundant leaves, and sharpens boundaries; for example, camera-relative depth is separated from metric inter-object distance and physical size.Source: §B.3 · p. 21, Table 4 · p. 21

B.4 VLM ANNOTATION AND EXCLUSIONS

  • Three VLMs retain examples when at least two capability labels agree: 9,444 are retained, including 8,811 unanimous labels, while 978 are excluded.The exclusions comprise 949 during taxonomy refinement and 29 without a two-of-three majority.Source: §B.4 · p. 21

B.5 HUMAN VALIDATION

  • Reviewers accept VLM labels in 378 of 400 judgments (94.50%); pairwise exact agreement ranges from 91.67% to 97.92%, and four-reviewer Fleiss’ κ is 0.96.The review covers 250 retained examples, with 50 shared across reviewers; agreement calculations exclude examples marked unsure.Source: §B.5 · pp. 21–22, Table 6 · p. 22

B.6 PROMPTS FOR CONSTRUCTION AND ANNOTATION

  • The appendix collects prompts for attribute extraction, schema induction, fixed-schema annotation, local tree induction, leaf assignment, global merging, 3R assignment, and VLM verification.The human taxonomy splitting, merging, and refinement steps do not use a model prompt.Source: §B.6 · p. 22

C.1 UNDERSTANDING CAPABILITIES

  • The inventory lists understanding capabilities, including multi-view reasoning, defined as transforming or integrating camera/observer coordinate frames such as camera motion/pose, requested projection, novel view, multi-view layout, or egocentric navigation.Table 7 describes the visual information required by each understanding capability and gives the number of retained evaluation examples (N).Source: §C.1 · pp. 23–24, Table 7 · p. 23

C.2 I2I TASK INVENTORY

  • The I2I inventory contains 19 tasks—two Recognition, nine Reconstruction, and eight Reorganization tasks—placed by the visual capability their output directly supervises.I2I tasks and I2T capabilities remain separate leaves even when they share a family.Source: §C.2 · p. 24, Table 8 · p. 24

C.3 EVALUATION SOURCES AND COVERAGE

  • The evaluation corpus contains 10,422 examples from seven benchmark collections, with 9,444 retained labels spanning Recognition (3,475), Reconstruction (3,038), and Reorganization (2,931).The authors count CV-Bench-2D and CV-Bench-3D as separate sets, yielding eight evaluation sets; 978 examples are excluded.Source: §C.3 · p. 25, Table 9 · p. 25, Figure 9 · p. 26

C.4 I2I TRAINING DATA

The I2I training inventory combines dense prediction, reconstruction, editing, and puzzle objectives, with each task trained separately on 50,000 examples. Representative examples illustrate the inventory, while the transfer training setup pairs each I2I task with subsequent I2T finetuning.

C.4 I2I TRAINING DATA

  • Each of the 19 I2I tasks is trained in a separate run with 50,000 examples, and tasks sharing source images use different targets.Twelve tasks use paired Taskonomy data, while Jigsaw and object pointing use VisGym data; editing tasks use ConceptEdit-12M pairs routed by capability.Source: §C.4 · p. 27, Table 10 · p. 27
  • The training sources include semantic segmentation targets from COCO, inpainting targets that restore complete images, and localization targets that mark referred-object bounding boxes.Inpainting combines COCO photographs with synthetic ConceptEdit source images; localization uses RefCOCOg records, reused to reach 50,000 examples.Source: §C.4 · pp. 27–28

C.5 REPRESENTATIVE EXAMPLES ACROSS OMNITASKONOMY

  • The appendix illustrates all 19 I2I objectives and 25 understanding capabilities with I2I input-to-target examples and I2T image-question-answer examples.The supplied examples include editing instructions, segmentation, counting, ordering, and visual correspondence tasks.Source: §C.5 · p. 29, Anomaly detection I2T · p. 31

D.1 TRAINING SETUP

  • For transfer training, each source checkpoint is finetuned on one I2I task and then on 50,000 LLaVA-Instruct examples; the I2T-only baseline uses the same starting checkpoint and settings.Each training uses three random seeds, and the VAE is frozen during I2I training and unused in the I2T stage.Source: §D.1 · p. 32, Table 11 · p. 32

D.2 FULL TASK-BY-CAPABILITY MATRIX

Appendix D.2 gives the full transfer matrix across 19 generation sources and 25 understanding capabilities, while Appendices D.3 and E.1–E.3 detail the significance tests and gradient-analysis procedures.

D.2 FULL TASK-BY-CAPABILITY MATRIX

  • The complete matrix reports accuracy changes over the I2T-only baseline for all 475 source–target pairs; six low-count capabilities are omitted from the main-text map.Outlined borders indicate paired-permutation significance at p < 0.05.Source: §D.2 · p. 32, Figure 10 · p. 33

D.3 PAIRED PERMUTATION TESTS OF TRANSFER GAINS

  • Each source–target pair is tested against the I2T-only baseline on the same questions across three seeds, using paired swaps and a p < 0.05 threshold.The reported gain is in percentage points.Source: §D.3 · p. 34

E.1 GRADIENT COLLECTION

  • Six paired tasks provide 500 source pairs each; gradients are collected at the pretrained EMA checkpoint, with five folds ensuring each scored pair is excluded from PCA fitting.The paired I2I and I2T examples need not have identical rendered inputs.Source: §E.1 · p. 34

E.2 LOSS DEFINITIONS AND GRADIENT DIMENSIONS

  • I2I uses flow-matching velocity loss, while I2T uses teacher-forced cross-entropy over supervised output tokens; the losses are computed separately without a mixing coefficient.Their absolute scales are not comparable, so losses are compared across checkpoints only within each objective.Source: §E.2 · pp. 34–35
  • Large gradient tensors are CountSketched, while small tensors and normalization weights retain original coordinates; blocks are concatenated by parameter type or kept layer-wise.Large ViT tensors use 8,192 coordinates and other large tensors use 16,384.Source: §E.2 · p. 35

E.3 PCA AND EFFECTIVE DIMENSION

  • PCA weights 12 task–objective groups equally and retains the leading eigenspace covering 99% of reference trace, placing both objectives in the same comparison subspace.For concatenated pre-attention RMSNorm, held-out gradients retain a mean 98.67–98.71% of energy across folds.Source: §E.3 · p. 35
  • Effective dimension is computed from projected, normalized reference directions and averaged across 500 source pairs; Figure 5(a) aggregates scaled scores by module.The ViT score is the equally weighted mean over its 11 parameter types.Source: §E.3 · p. 36

E.4 CROSS-TASK ALIGNMENT AND TRANSFER

Cross-task gradient alignment is positively correlated with transfer gains. Separately, I2I-only training lowers paired I2T losses at 30k examples, despite no I2T supervision at those checkpoints.

E.4 CROSS-TASK ALIGNMENT AND TRANSFER

  • For seven capabilities, 500 examples each and 500 per I2I source feed gradient alignment, averaged over 10,000 paired minibatch draws per source–target cell.Alignment averages minibatch cosines rather than taking the cosine between full-pool gradients.Source: §E.4 · p. 36
  • Alignment is associated with transfer gains: Pearson r = 0.795 across seven capability means and r = 0.529 across 133 source–target pairs.Transfer gain is the accuracy after I2I training followed by I2T finetuning, minus the I2T-only baseline, in percentage points.Source: §E.4 · p. 36

E.5 LOSSES AND GRADIENT NORMS AFTER I2I-ONLY TRAINING

  • At 30k, I2T loss falls from 0.43 to 0.28 on Jigsaw and from 0.29 to 0.22 on Zoom-In, despite no I2T supervision.The losses and gradient norms are reported after I2I-only training.Source: Table 15 · p. 37, §E.5 · p. 37
  • The 3k, 10k, and 30k I2I-only budgets are separate runs from the same pretrained model, and none receives I2T finetuning.The same 500 matched I2I–I2T pairs per task are evaluated at each checkpoint.Source: §E.5 · p. 37
  • At 30k, I2T gradient norm falls from 8.89 to 2.18 on Jigsaw and from 4.56 to 0.74 on Zoom-In.The checkpoints are separate runs from the same pretrained model, not intermediate checkpoints of one run.Source: §E.5 · p. 37

F PROMPTS

Appendix F documents the prompts and procedures used to build OmniTaskonomy, annotate benchmark capabilities, judge generation outputs, and route Recognition editing data through structured schemas, independent review, and explicit criteria. The routing prompts match edits to visual-understanding capabilities or select “none” when no semantic target applies; they are selection instructions, not BAGEL training prompts.

F.1 TAXONOMY CONSTRUCTION

  • Taxonomy prompts first elicit 50–80 exhaustive, orthogonal attributes per sample, then induce a fixed schema that conservatively merges only synonymous keys.The schema prompt preserves distinctions between attributes that capture different properties.Source: §F.1 · pp. 39–40
  • The local capability tree has four levels—domain, coarse ability, fine-grained skill, and difficulty/granularity/subtype leaf—and paths must be mutually exclusive, exhaustive, and supported by the data.Sparse skills may stop early, while richer skills receive more leaves.Source: §F.1 · p. 43

F.2 THREE-MODEL CAPABILITY ANNOTATION

  • Three independent raters—Gemini, GPT, and Claude—receive the same capability definitions and instructions, with catalogue order shuffled per rater and predictions hidden from one another.Assignments return a primary leaf, optional secondary leaf, fit rating, and rationale.Source: §F.2 · pp. 48, 50
  • An example is retained only when at least two raters choose the same primary leaf; malformed assignments are retried, and no fallback adjudicator or tie-breaker is used.The consensus fit is the least favorable fit among supporting raters and does not affect acceptance or evaluation weight.Source: §F.2 · p. 50

F.3 JUDGING I2I GENERATIONS

  • Each Jigsaw or Zoom-In generation receives one structured-JSON judgment from Gemini using the input image, reference image, and candidate output.The prompts are for instance-level evaluation.Source: §F.3 · p. 51

F.4 RECOGNITION EDITING-DATA ROUTING

  • The Recognition-routing prompt predates Recognition tasks and refers to 15 I2I objectives, whereas the paper’s expanded inventory contains 19; its routing criteria remain unchanged.Three Chinese removal aliases in the original prompt are rendered in English in the appendix.Source: §F.4 · p. 53
  • Recognition-data routing uses edit labels, instructions, and source captions to select the single Recognition capability required by the edited image, returning “none” when its primary requirement is non-recognition.The prompt directs the classifier to judge the supervised output rather than the dataset name or incidental semantic content; geometric or photometric requirements are Reconstruction.Source: §F.4 · p. 53
  • Reorganization covers grouping and segmentation, correspondence, ordering, counting, localization, and connection/topology without necessarily naming visual elements.It addresses what goes with what and how elements are organized.Source: §F.4 · p. 53
  • Use “none” when an edit has no particular visual-understanding target, such as a global filter, resolution or quality change, or semantically untargeted re-render.Source: §F.4 · p. 53
  • “None” is a normal, frequent routing answer and is preferred to a weak or merely topical match.Source: §F.4 · p. 53
  • Figures 22 and 23 identify the recognition-data routing system instruction and per-item input, while Figure 24 presents example recognition training instructions.Source: Figure 22 · p. 54, Figure 23 · p. 54, Figure 24 · p. 55
  • The example instructions come from training records paired with original source images and supervised by the corresponding edited images.Source: §F.4 · p. 55
  • Routing prompts are used only to select data and are not used as BAGEL training prompts.Source: §F.4 · p. 55
Loading 2609.38079v1…