PaperlayerView exact revision on arXiv

Source-linked AI summary

WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation

Wei Chow, Jiachun Pan, Yongyuan Liang, Mingze Zhou, Xue Song, Liyu Jia, Saining Zhang, Siliang Tang, Juncheng Li, Fengda Zhang, Weijia Wu, Hanwang Zhang, Tat-Seng Chua

arXiv:2511.11434v1cs.CV

TL;DR

Existing datasets and benchmarks largely miss the historical context and iterative dependencies of real-world image creation and editing. WEAVE combines a large interleaved dataset with a human-annotated benchmark; training improves performance across tasks, while evaluations reveal persistent multi-turn generation challenges.

  • Problem

    Existing datasets and benchmarks focus primarily on single-turn interactions rather than the historical context and iterative workflows of real-world image creation and editing.

    Source: §1 · p. 1

  • Method

    WEAVE combines 100K interleaved samples spanning over 370K dialogue turns and 500K images with a human-annotated 100-task, 480-image benchmark using hybrid VLM evaluation.

    Source: §1 · pp. 1–2

  • Results

    Training on WEAVE-100k improves established benchmarks across tasks, including 9.8% on MMMU, and facilitates visual memory, while WEAVEBench shows persistent multi-turn generation struggles.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    WEAVE provides a foundation for studying in-context interleaved multimodal comprehension and generation.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent advances in unified multimodal models (UMMs) have enabled impressive progress in visual comprehension and generation. However, existing datasets and benchmarks focus primarily on single-turn interactions, failing to capture the multi-turn, context-dependent nature of real-world image creation and editing. To address this gap, we present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation. Our suite consists of two complementary parts. WEAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images, covering comprehension, editing, and generation tasks that require reasoning over historical context. WEAVEBench is a human-annotated benchmark with 100 tasks based on 480 images, featuring a hybrid VLM judger evaluation framework based on both the reference image and the combination of the original image with editing instructions that assesses models' abilities in multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments demonstrate that training on WEAVE-100k enables vision comprehension, image editing, and comprehension-generation collaboration capabilities. Furthermore, it facilitates UMMs to develop emergent visual-memory capabilities, while extensive evaluations on WEAVEBench expose the persistent limitations and challenges of current approaches in multi-turn, context-aware image generation and editing. We believe WEAVE provides a view and foundation for studying in-context interleaved comprehension and generation for multi-modal community.

1. Introduction

Real-world image creation involves iterative refinement and reuse of prior results, but existing datasets largely treat edits as isolated single-turn instructions. WEAVE pairs a context-dependent dataset with a benchmark, and experiments report cross-task gains alongside persistent challenges in multi-turn generation.

  • Real-world image creation involves reversible refinement and reuse of earlier results, while existing datasets and most open-source models remain focused on single-turn editing.Source: §1 · p. 1
  • WEAVE-100k covers comprehension, editing, and generation requiring historical-context reasoning, with 100K interleaved samples and over 370K dialogue turns.Source: §1 · pp. 1–2
  • WEAVEBench finds that current models struggle with in-context interleaved generation and degrade as content length increases.Source: §1 · p. 2
  • WEAVEBench is a human-annotated benchmark of 100 tasks and 480 images that evaluates multi-turn generation, visual memory, and world-knowledge reasoning.Source: §1 · p. 2
  • Training on WEAVE-100k improves vision comprehension (9.8% on MMMU), image editing (4.8% on GEditBench), and comprehension-generation collaboration (approximately 50% on RISE).Source: §1 · p. 2

2. Related Works

Prior work integrates comprehension and generation in unified multimodal models and advances text-guided editing, but much editing data lacks historical context and iterative dependencies. WEAVE is presented as a dataset that explicitly captures these dependencies.

  • Early unified multimodal models generate visual tokens through image tokenization and autoregressive prediction [16] [55], while later systems add diffusion or flow-matching heads [47].Source: §2 · p. 2
  • Editing datasets such as AnyEdit, GPT-Image-Edit-1.5M [72], Echo-4o [84], and MagicBrush remain limited in historical context or multi-turn dependencies.MagicBrush treats each instruction as an independent request, while the other described approaches focus on one-shot edits.Source: §2 · pp. 2–3
  • WEAVE-100k is presented as a cross-modal dataset capturing multi-turn editing and context-dependent generation, with benchmark comparisons spanning interleaving, memory, evaluation, and annotation dimensions.Source: §2 · p. 3, Table 1 · p. 3

3. WEAVE

WEAVE combines a multi-turn dataset with a human-annotated benchmark to study interleaved multimodal comprehension and generation. Its collection and evaluation setup covers visual-memory editing, world-knowledge tasks, and comprehension.

3.1. Data Collection

  • WEAVE-100k uses four pathways for multi-turn editing, while WEAVEBench covers 100 items across 16 categories, including visual-memory editing and world-knowledge challenges.The dataset pathways are multi-image fusion, remove-then-back, derivative imagination and comparison, and sequential procedures.Source: §3.1 · p. 3

3.2. Evaluation Settings and Metrics

  • The hybrid VLM judge evaluates editing with Key Point Correctness, Visual Consistency, and Image Quality, and comprehension with Accuracy.It assesses against both a reference image and the original image paired with editing instructions.Source: §3.2 · p. 3

3.3. Data Statics

  • Training instances average 3.8 conversational turns, most contain more than five images, and category distributions are relatively balanced across training and test sets.Source: §3.3 · p. 5

4. Experiment

Experiments evaluate models on WEAVEBench, test Bagel fine-tuned on WEAVE-100k, analyze output quality, and check VLM-judge reliability. Fine-tuning improves performance across several benchmarks, while context length and task complexity remain challenging.

4.1. WEAVEBench

  • The experiments evaluate 22 models across no, partial, and complete in-context conditions, with image-placement and input-format variations.Source: §4 · p. 5, §4.1 · p. 5
  • The best Edit and UMM models scored at most 0.68 and 0.767, respectively, and creative imagery outperformed scientific and logical domains.Performance varied with context: QwenVL improved 163% on comprehension, while open-source Qwen-Edit declined 5.3% and 8.6% with additional context.Source: §4.1 · pp. 5–6, Table 2 · p. 6
  • Ground-truth in-context images improved performance across models, and sequential image input avoided Bagel's 10.3% degradation from concatenation.Qwen-Image-Edit improved 7.1% with ground-truth examples.Source: §4.1 · p. 6, Figure 6 · p. 8

4.2. Train on WEAVE-100k

  • 42.5% improvement over Bagel on WEAVEBench included a 34.6% gain on science questions; fine-tuning also improved MMMU 9.8%, GEditBench 4.8%, and RISE spatial and logical reasoning 100%.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.3. Quality Analysis.

  • Qualitative examples show fine-tuned models distinguishing protagonists by clothing and removing then reinserting people, while other models still miss instructions.Source: §4.3 · p. 7, Figure 7 · p. 8

4.4. Reliability of Judge Usage

  • GPT-4.1 score correlations with expert ratings consistently exceeded 0.8 across 100 instances per model, and Claude evaluations showed strong cross-VLM consistency.The expert evaluation covered Nano-banana, Qwen-Image-Edit, and SeeDream.Source: §4.4 · pp. 7–8, Figure 6 · p. 8

5. Conclusion

The conclusion presents WEAVE as a suite for interleaved multimodal comprehension and generation, comprising WEAVE-100k and WEAVEBench. Training results are positive across benchmarks, but evaluations still expose difficulty with longer multi-turn generation.

  • WEAVE-100k contains 100K samples, 370K dialogue turns, and 500K images; WEAVEBench contains 100 tasks with 480 images and a hybrid VLM judge.Source: §5 · p. 8
  • Training on WEAVE-100k improved MMMU by 9.8% and GEditBench by 4.8%, while facilitating visual-memory capabilities in UMMs.Source: §5 · p. 8
  • WEAVEBench evaluations found that models struggle with multi-turn, context-aware generation, particularly as content length increases.Source: §5 · p. 8

A. Weave Analysis

The appendix describes WEAVE-100k's collection procedures, which combine multiple image-editing pathways with verification and refinement. It also extends generated data with comprehension tasks.

A.1. Collection Process

  • Manual sampling verification checks each pipeline after filtering, while a post-verification stage uses CLIP similarity and joint CLIP-Qwen3-VL-4B supervision to re-edit or exclude problematic samples.Source: §A.1 · pp. 14–15
  • Multi-image fusion sequences combine generated or edited images with instructions, then build multi-round edits through add, remove, replace, color-alter, or background-change operations.Source: §A.1 · p. 14
  • Remove-then-back examples instruct a later turn to restore an object removed or replaced earlier; about 25% of generated instructions passed filtering.Source: §A.1 · p. 14
  • Derivative-imagination data adapts chess and visual-jigsaw tasks, with GPT-4.1 recombining and self-verifying examples and translating abbreviated chess notation into explicit instructions.Source: §A.1 · pp. 14–15
  • Sequential procedures create narrative edits requiring explicit inter-step relations and identifiable characters; story-based test content is human-screened because generation quality required supervision.Source: §A.1 · p. 15
  • Comprehension extensions sample filtered generated data and add at most one turn of captioning, image questions, or a smaller set of knowledge-based inquiries.Source: §A.1 · p. 15

A.2. Data Source for WEAVEBench

WEAVEBench combines web-collected, refined, and synthetic images, with dataset statistics and categories described in accompanying tables. Its evaluation prompts assess four dimensions using GPT-4o as judge.

A.2. Data Source for WEAVEBench

  • WEAVEBench uses primarily web-collected data, with selected images refined by SeedEdit 3.0 and additional images generated by Seedream 4.0, Nano Banana, and SeedEdit 3.0.Jigsaw and chess images come from Zebra-CoT, while optical and physical phenomena images come from PhysBench.Source: §A.2 · p. 15

A.3. Statistics

  • Table 5 provides a granular breakdown of sub-domains and domains in WEAVE-100k, while the dataset statistics and category tables are identified as Tables 5 and 6.Table 6 covers categories, main content, scenarios, and editable dimensions; the passages do not provide the tables’ values.Source: Table 5 · p. 15, §A.3 · p. 16, Table 6 · p. 17

B.1. Evaluation Prompts

  • GPT-4o judges four evaluation dimensions: Key Point Correctness, Visual Consistency, Image Quality, and Accuracy.Figures 9–12 show prompts for Key Point Correctness, Visual Consistency, Image Quality, and Comprehension Accuracy, respectively.Source: §B.1 · p. 16, Figure 9 · p. 18

B.2. Training Details

Training used 30,000 steps on eight H100 GPUs, with random turn sampling because complete image-heavy dialogues could exceed the available context length. The training configuration also specifies image-reference post-processing and hyperparameters.

B.2. Training Details

  • The model was trained for 30,000 steps on 8 NVIDIA H100 GPUs, with batch size 1 per GPU, requiring approximately 60 hours of compute time.Source: §B.2 · p. 18
  • To avoid context-length overflow, training randomly sampled individual conversation turns instead of complete dialogue sequences.Post-processing adjusted numerical image references to match the images’ sequential positions after turn sampling.Source: §B.2 · p. 18
  • Training used a maximum latent size of 64, learning rate 2 × 10^-5, maximum sequence length 11,520 tokens, and maximum sample length 10,240 tokens.Vision-transformer conditional dropout was 0 and EMA decay was 0.9999.Source: §B.2 · p. 21

B.3. Details on Benchmarks and Metrics

The section specifies weighted evaluation scores and conditions for using dialogue history, alongside an image-concatenation procedure for models that cannot process sequential image inputs. It also points readers to WEAVEBench leaderboard and category results tables.

B.3. Details on Benchmarks and Metrics

  • WEAVEBench leaderboard scores are in Table 2, with model metrics across Science, Creation, Logic, and Game reported in Tables 7–10.The supplied passages identify these tables’ coverage but do not provide their score values.Source: §B.3 · p. 22
  • History conditions are no history, partial history with self-generated images and explicit visual context, or complete history with all prior interactions.For complete history, VLMs receive historical dialogue, while generative models generally receive only historical images.Source: §B.3 · p. 22
  • Models unable to process image sequences receive images arranged horizontally in one row with sequential number labels.Some models, including Step1X, could not identify which image to edit and retained original dimensions, so displayed examples were cropped and rescaled for comparison.Source: §B.3 · p. 22

B.4. Baselines Details

The paper evaluates language, image-editing, and unified multimodal baselines, then situates WEAVE among work on interleaved reasoning and multimodal benchmarks. It presents WEAVE as the first benchmark to comprehensively assess interleaved multi-turn generation and understanding.

B.4. Baselines Details

  • The WEAVEBench evaluation includes 4 LLMs, 7 editing models, and 11 unified multimodal models.The section says model parameter configurations are detailed for these baselines.Source: §B.4 · p. 22
  • The unified-model baselines include Bagel, OmniGen2, OmniGen, Ovis-U1, UniPic variants, NextStep-1-Large-Edit, Seedream 4.0, and Nano Banana.Ovis-U1 generation supports only single-image input, so the authors concatenated inputs when examples contained multiple images.Source: §B.4 · pp. 22–23
  • The editing baselines include AnyEdit, UltraEdit, FLUX.1 Kontext, VAREdit-8B, Qwen-Image-Edit, and Step1X-Edit versions 1.1 and 1.2.The authors state that they follow baseline repositories’ or Hugging Face implementations’ default hyperparameters.Source: §B.4 · pp. 23–24

C. More Related Works

  • Related interleaved-reasoning work includes visual-symbol reasoning, Zebra-CoT trajectories, iterative image refinement, and research on reciprocal generation–comprehension capabilities.The paper distinguishes WEAVE by its focus on in-context interleaved multimodal comprehension and generation.Source: C. More Related Works · p. 26
  • Existing UMM benchmarks assess text-to-image generation, vision comprehension, and image editing, while ROVER evaluates reciprocal cross-modal reasoning.The paper describes WEAVE as the first benchmark to comprehensively evaluate interleaved multi-turn generation and understanding.Source: C. More Related Works · pp. 26–27

D.1. Additional Examples for WEAVE-100k

The appendix illustrates WEAVE-100k’s editing, fusion, and recall examples, then samples WEAVEBench’s domain coverage and discusses release risks.

D.1. Additional Examples for WEAVE-100k

  • Editing examples include intricate spatially and semantically coherent manipulations and adding human subjects to a composition.Figures 17 and 18 illustrate these operations.Source: §D.1 · p. 27, Figure 17 · p. 31, Figure 18 · p. 32
  • Multi-image fusion examples preserve footwear, process faces in two tasks, and follow hairstyle and headphone-interaction requirements.The examples include headphones held by one subject and worn by another.Source: §D.1 · p. 27, Figure 13 · p. 28, Figure 14 · p. 28
  • Recall examples restore removed trousers, reconstruct a body and scene from an earlier image while mirroring the background, and reinsert a person.These examples illustrate retrieval and reuse of visual information from prior turns.Source: §D.1 · p. 27, Figure 19 · p. 33, Figure 20 · p. 34
  • Chess-game manipulation and visual jigsaw processing provide examples of structured visual reasoning tasks.The examples are presented as evidence of the framework’s adaptability to these tasks.Source: §D.1 · p. 27, Figure 22 · p. 36, Figure 23 · p. 37

D.2. More example for WEAVEBench

  • WEAVEBench examples cover science, games, mathematics, geography, optics, memory, spatial reasoning, narratives, and image editing.The examples test capabilities such as understanding game environments, recalling information, and solving spatial tasks.Source: §D.2 · p. 27

E. Broader Impact

  • The authors identify potential hallucinations and inherited foundation-model biases as risks of WEAVE-100k’s release.They note that bias may affect retrieval and equitable representation across cultural contexts.Source: E. Broader Impact · p. 27

Related papers

Loading 2511.11434v1…