PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs

Xin Gao, Cheng Yang, Chufan Shi, Taylor Berg-Kirkpatrick

arXiv:2606.00477v1cs.CLcs.CV

TL;DR

The paper asks whether text-side knowledge edits reliably transfer to visually verifiable image generation in unified multimodal models. It introduces UniKE with VQA-based verification and proposes reasoning-augmented parameter editing; results reveal a modality gap that reasoning partially mitigates. The paper attributes this gap to incomplete preservation or alignment of edited textual representations in visual-conditioning pathways.

  • Problem

    It remains unclear whether knowledge edits that successfully modify textual outputs also transfer to image generation in unified multimodal models.

  • Method

    The paper introduces UniKE, a benchmark coupling text-side knowledge edits with VQA-based visual verification, and adds an intermediate reasoning step before image generation.

  • Results

    Text-side editing success does not reliably produce corresponding visual evidence, while reasoning-augmented generation partially mitigates the cross-modality gap.

  • Takeaways & Limitations

    Textual knowledge edits do not guarantee reliable cross-modality transfer, motivating methods that account for visual-conditioning pathways.

  • Takeaways & Limitations

    Current editing methods exhibit inconsistent cross-modality transfer, allowing text-based edits to fail to reliably propagate to visual generation.

Abstract

from arXiv · show

Unified multimodal models (UMMs) have emerged as a promising paradigm for general-purpose multimodal intelligence. As they are deployed in real-world applications, effectively updating internal knowledge becomes critical. While knowledge editing has matured for text-only models, it remains unclear whether edits that successfully modify textual outputs also transfer to image generation in UMMs. To study this question, we introduce UniKE, the first benchmark for cross-modality knowledge editing in UMMs, comprising 2,971 edit subjects spanning attribute and relation edits. Using VQA-based visual verification, we reveal a striking modality gap: text-side efficacy can reach approximately 92%, whereas the best overall VQA accuracy under direct image generation is only 18.5%. We further propose Reasoning-augmented Parameter Editing, which explicitly activates edited knowledge before generation and improves overall VQA accuracy for all evaluated model-editor pairs, with gains up to 18.6 percentage points. Mechanistic analysis shows that this gap is associated with partial alignment between edited textual representations and the conditioning pathways for visual generation, where edits sufficient for text outputs may remain too weak or misaligned to steer image synthesis. These findings show that textual knowledge edits do not guarantee reliable cross-modality transfer and motivate modality-aware editing methods. Our code and data are available at https://github.com/gxx27/UniKE.

1. Introduction

The paper studies whether text-side knowledge edits in unified multimodal models transfer to visually verifiable image generation. It introduces UniKE and analyzes a reasoning-based intervention and the conditioning bottleneck behind the modality gap.

  • The benchmark contains 2,971 edit subjects spanning attribute and relation edits, expanded into 5,535 evaluation instances.
  • VQA-based visual verification tests whether generated images are consistent with post-edit knowledge.
  • Text-side editing can substantially improve textual outputs while producing weaker and less reliable changes in image generation.
  • Reasoning-augmented Parameter Editing elicits an intermediate textual reasoning step to activate edited knowledge before image generation.The intervention is intended to provide a stronger semantic constraint for visual synthesis.
  • Mechanistic analyses associate limited transfer with a conditioning-pathway bottleneck that attenuates or weakly expresses edit-induced perturbations before image generation.Reasoning augmentation introduces a stronger textual conditioning shift that partially improves transfer.
  • UniKE is introduced as a benchmark for cross-modality knowledge editing in unified multimodal models.

2. Related Work

The related work covers knowledge editing methods and benchmarks for text and multimodal models, alongside visual editing methods designed primarily for modular text-to-image systems. It situates UniKE among evaluations of factual updates across modalities.

  • Knowledge Editing: Knowledge editing methods update factual beliefs without expensive retraining, using either parameter-preserving demonstrations or parameter-modifying weight edits.
  • Visual Knowledge Editing: Text-to-image editing methods such as TIME, ReFACT, and DiffQuick-Fix primarily target distinct text encoders or cross-attention layers in modular architectures.
  • Knowledge Editing Benchmarks: Existing knowledge-editing benchmarks evaluate text-based edits, counterfactual rewrites, robustness, locality, or multi-hop textual reasoning.
  • Unified Multimodal Models: Unified multimodal models integrate textual and visual modalities within a single framework, typically using a shared transformer backbone.

3. The UNIKE Benchmark

UNIKE benchmarks whether text-side knowledge edits in unified multimodal models transfer to visually verifiable image generation. It covers attribute and relation edits using answer-neutral prompts, staged difficulty, and VQA-based validation.

  • Benchmark Design: UNIKE evaluates text-side edits by pairing each target fact with an image prompt, visual target description, and VQA verification query.Each instance specifies q, y, y′, pimg, tvis, and qvqa.
  • Benchmark Design: The benchmark covers attribute edits to visual properties and relation edits restricted to relationships with depictable visual implications.Attribute domains include color, material, shape, size, and pattern; relation categories include affiliation, creator, location, and occupation.
  • Data Construction: Answer-neutral prompts and visualizability filtering ensure that target outcomes are visually distinguishable and cannot be revealed directly by the prompt.Retained instances undergo validation for answer-neutrality, non-tautological targets, and unambiguous VQA queries.
  • Attribute Edits: Attribute edits are instantiated across stages that progressively increase grounding and compositional difficulty, from atomic objects to complex scenes and derived references.Stages vary lighting, viewpoint, background, multi-entity interactions, and whether the prompt directly names the edited entity.
  • Benchmark Composition: 2,971 edit subjects and 5,535 evaluation instances comprise the final UNIKE benchmark after visualizability filtering and automated validation.The benchmark includes 964 attribute edits and 2,007 relation edits.

4. Evaluation of Cross-Modality Knowledge Editing

Evaluation reveals a substantial gap between textual edit success and visually verifiable generation, while reasoning augmentation improves but does not eliminate this gap. The remaining failures vary by prompt stage, edit category, and model architecture, reflecting distinct retrieval and visual-conditioning bottlenecks.

  • Direct-generation results: Text-side efficacy can remain strong while only a small fraction of edits appear in visually verifiable images.Across the best direct settings, VQA accuracy retains only about one eighth to one quarter of corresponding text-side efficacy.
  • Reasoning-augmented results: Reasoning augmentation improves overall VQA accuracy for every model-editor pair, but gains depend on architecture and category.The method is most useful when edited knowledge is present but not naturally exposed to the visual-conditioning pathway; textual recovery and visual realization remain distinct.
  • Stage-wise transfer: Performance follows a cascade from text-side efficacy to reasoning accuracy and then VQA accuracy, with drops reflecting prompt generalization and cross-modal visual grounding.Image generation must bind the recovered target to the correct referent, overcome pretrained visual priors, and render evidence clearly enough for VQA verification.
  • Edit categories: Attribute and relation edits fail differently: attributes are harder to edit textually, whereas relations can be harder to verify visually despite successful textual recall.Relation verification often requires contextual evidence such as landmarks, artifacts, logos, or identity cues.
  • Stage-wise analysis: Across attribute stages, text-side efficacy drops roughly 70% from Stage 1 to Stage 2, while reasoning accuracy loses about one tenth at Stage 2 and about one quarter by Stage 4.VQA accuracy decreases roughly 35–40% from Stage 1 to later stages and remains below reasoning accuracy throughout.
  • Category-wise analysis: Category-wise results favor visually concrete edits: size and occupation are easiest, while shape, pattern, location, affiliation, and creator are more difficult.The harder categories depend on precise geometry, fine-grained textures, contextual cues, or distinctive identity evidence.
  • Mechanistic analysis: Mechanistic analysis links the modality gap to conditioning-pathway alignment rather than edit magnitude alone.A projection can attenuate off-distribution edit perturbations, and reasoning amplifies the effective DiT conditioning shift most strongly for models with weak direct signals.

5. Conclusion

The paper finds that text-side knowledge edits do not reliably transfer to visually consistent image generation in UMMs. Reasoning-augmented generation partially mitigates this gap, which the analysis associates with a conditioning-pathway bottleneck.

  • Text-based editing success does not necessarily induce corresponding visual evidence in generated images.
  • Reasoning-augmented generation partially mitigates the cross-modality gap by explicitly activating edited knowledge before visual synthesis.
  • The limitation is associated with only partial preservation of edit-induced textual changes in image-generation conditioning signals.
  • The findings motivate editing methods that directly target modality-relevant generation pathways.

Impact Statement

The paper studies efficient factual updates in UMMs, while highlighting that edits may transfer inconsistently from text to visual generation. It also notes possible misuse and calls for safeguards and transparency.

  • Current editing methods show inconsistent cross-modality transfer, with text-based edits not reliably propagating to visual generation.
  • Knowledge editing could be misused to alter historical facts or create biased representations, motivating safeguards and transparency about modifications.
  • Knowledge editing aims to update specific model facts without retraining the entire model.
  • A good edit changes behavior on queries about the target knowledge while preserving unrelated behaviors as much as possible.
  • Single-edit evaluation resets the model between knowledge items, whereas sequential editing applies a stream of updates to one model instance.Sequential editing can accumulate off-target changes and cause later edits to interfere with earlier ones.

B. Dataset Introduction

The section introduces established knowledge-editing datasets and editing mechanisms, then describes projector-based parameter updates for shared visual backbones. It motivates softened projection because shared representations can leave too little effective null space.

  • ZsRE evaluates whether edited answers persist under rephrased questions and typically includes unrelated questions for locality testing.
  • CounterFact tests counterfactual factual rewriting while probing locality with semantically similar subjects under the same predicate.
  • MQuAKE measures whether edits support multi-hop reasoning consequences rather than only direct factual recall.
  • TMKE evaluates whether knowledge updates applied in one modality remain consistent across multimodal inputs, prompt formulations, and visual contexts.
  • AlphaEdit restricts updates toward directions weakly used by pre-edit activations through a null-space projector.
  • Shared visual backbones can occupy more representation space, shrinking the effective null space and making the original projector overly restrictive.
  • A softened projector interpolates between null-space enforcement and unconstrained updates, allowing higher-energy directions when the null space is insufficient.
  • The implementation applies model-specific projector settings and clips update norms for selected models to balance text-side efficacy with stable image generation under sequential editing.

D. Conditioning Pathway Analysis

The section presents experimental analyses of the conditioning pathway across models. It examines bottlenecks, conditioning drift, diffusion-transformer propagation, and reasoning decomposition.

  • The analysis first summarizes each model’s conditioning architecture.
  • It then presents an SVD bottleneck analysis for Ovis-U1 and full conditioning drift breakdowns.
  • The remaining analyses verify DiT propagation and decompose the effects of reasoning.

D.1. Conditional architecture Difference

The evaluated UMMs use distinct interfaces to pass edited language representations into their DiT image generators. These conditioning architectures differ in which language states are transformed, queried, or passed directly to the DiT.

  • Ovis-U1 concatenates its last two language hidden layers and maps them through a frozen linear projection before DiT conditioning.
  • BLIP3o-4B appends 64 learnable query tokens whose hidden states pass through edited LLM layers and directly condition the DiT.
  • OmniGen2 passes final-layer text hidden states directly to the DiT.

D.2. SVD Bottleneck Analysis

The SVD analysis examines how Ovis-U1’s frozen projection preserves edit perturbations. The observed perturbations are broadly distributed, consistent with the expected retention of a 1536-dimensional projection from a 4096-dimensional space.

  • Ovis-U1 maps concatenated last-two-layer hidden states into DiT input space through a frozen projection W ∈R1536×4096.
  • The analysis measures the cumulative edit-perturbation fraction captured by the top-k right singular vectors of W.These vectors represent dominant input directions preserved by the projection, while δ is the pre-projection edit perturbation.
  • Near-linear ρ(k) growth indicates that edit perturbations are broadly distributed across the 4096-dimensional space rather than strongly aligned with principal projection directions.
  • 37.5% of squared perturbation norm is expected to be preserved for a uniformly distributed vector under this 1536-to-4096 dimensional projection.The passage derives this expectation as 1536/4096 = 37.5%.
  • 32–35% signal retention remains stable across all three editing methods, indicating an architectural property independent of the editing algorithm.

D.3. DiT Block Sensitivity

The analysis attributes the visual-editing gap to the conditioning interface rather than downstream DiT processing. A reasoning-augmented example shows that correctly verbalized edits can nevertheless produce consistent visual changes across generation stages.

  • D.3. DiT Block Sensitivity: The perturbation is traced through Ovis-U1’s Yak DiT at timestep t=0.5 using identical random image latents for fresh and edited models.
  • D.3. DiT Block Sensitivity: The DiT’s near-constant per-block drift confirms linear propagation without amplification or attenuation of the perturbation.
  • D.3. DiT Block Sensitivity: 0.077 relative drift at Ovis-U1’s DiT input contrasts with 0.458 for BLIP3o-4B and 0.435 for OmniGen2.
  • D.3. DiT Block Sensitivity: The 5–6× conditioning-interface gap constrains achievable image-generation differences and places the bottleneck upstream at the projection interface.
  • D.3. DiT Block Sensitivity: In the peacock example, reasoning repeatedly recalls the edited “floral” attribute and describes flowers, leaves, vines, and scene interactions before image generation.
  • D.3. DiT Block Sensitivity: Across stages, the reasoning-augmented protocol changes peacock plumage and a peacock-inspired scarf from standard eye-spot patterns to visible floral motifs.

F. Prompt

The prompt design constructs multi-stage attribute-editing examples with neutral image prompts and explicit VQA targets. It covers diverse attribute categories while requiring concise, unambiguous textual supervision.

  • F. Prompt: The data-generation pipeline uses Gemini-3.0-Flash for diverse coverage and Qwen3-VL-235B-A22B-Instruct for deterministic visual verification.
  • F.1. Prompt for generating attribute data: Attribute data spans size, color, material, and shape categories, each defined by the property being changed.
  • F.1. Prompt for generating attribute data: Prompts use incomplete cloze statements rather than questions, and ground-truth values are concise one- or two-word attribute names.
  • F.1. Prompt for generating attribute data: Size edits include absolute scale changes and contextual comparisons against familiar reference objects, with answers naming only the relevant value or object.
  • F.1. Prompt for generating attribute data: Pattern prompts focus on surface designs, while image prompts must explicitly contain the tested object or derivative and include reference objects when required.
  • F.1. Prompt for generating attribute data: Each example specifies an entity, an original and counterfactual attribute, four increasingly complex stages, and repeated full entity names to eliminate ambiguity.
  • F.1. Prompt for generating attribute data: Image prompts remain answer-neutral by avoiding original and target values or their implied appearance, while VQA questions name the subject and visual targets assert the required property.
  • F.1. Prompt for generating attribute data: Stages progress from direct attribute references to realistic contexts, complex interactions, and derived products or uses.
Loading 2606.00477v1…