PaperlayerView exact revision on arXiv

Source-linked AI summary

YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li, Ruiqi Lei, Fushuo Huo, Yuhao Pan, Yunfeng Fan, Ding Wang, Nan Cheng, Haozhao Wang, Wenchao Xu

arXiv:2611.05280v1cs.CV

TL;DR

Existing multimodal datasets often lack supervision for dialogues that require tracking evolving state across turns. YANAMI provides a large-scale dataset and a benchmark that evaluates dialogue-state understanding and visual realization; training improves several capabilities, while current UMMs still struggle with stateful interaction.

  • Problem

    Many multimodal datasets focus on single-turn tasks, and multi-turn examples often do not require tracking information across turns, limiting supervision and evaluation for stateful interaction.

    Source: §1 · p. 2

  • Method

    YANAMI pairs 400K stateful multimodal dialogues with a 400-dialogue benchmark that jointly evaluates dialogue-state understanding and visual realization.

    Source: §1 · pp. 2–3

  • Results

    Training on YANAMI-400K improves image generation, editing, and stateful interaction while preserving general visual understanding, but existing UMMs still struggle with evolving dialogue states.

    Source: §1 · p. 3

  • Takeaways & Limitations

    YANAMI provides a unified foundation for training and evaluating UMMs toward robust multimodal stateful interaction.

    Source: §1 · p. 3

    YANAMI is intended for research on multimodal stateful interaction and is constructed from existing public datasets and model-generated content.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.

1 introduction

YANAMI addresses the limited support for dialogues that require models to track and revise multimodal state across turns. It pairs a large-scale dataset with a benchmark for understanding states and realizing them visually.

1 introduction

  • UMMs integrate image understanding, generation, and editing in one framework, but their performance in dialogues with evolving contextual dependencies remains underexplored.The unified framework interleaves textual and visual processing.Source: §1 · p. 2
  • Existing datasets are dominated by single-turn tasks, and many multi-turn examples can be solved independently without tracking cross-turn information.Real-world image creation and editing often involve continuous interaction and gradual refinement.Source: §1 · p. 2
  • Training on YANAMI-400K improves image generation, editing, and stateful interaction while preserving general visual understanding; existing UMMs still struggle with evolving states.The benchmark highlights challenges in maintaining, updating, and visually realizing dialogue states.Source: §1 · p. 3
  • YANAMI-400K contains 400K multi-turn dialogues, more than 2M images, and 2.6M turns across four scenarios and six state-management capabilities.The capabilities make tracking, preservation, and revision of multimodal states explicit across turns.Source: §1 · pp. 2–3
  • YANAMI-Bench comprises 400 multi-turn dialogues and jointly evaluates dialogue-state understanding and faithful visual realization.Its two dimensions characterize the gap between understanding dialogue states and realizing them in visual outputs.Source: §1 · pp. 2–3

2 related work

Prior multimodal datasets and benchmarks cover individual capabilities, multi-image tasks, and increasingly multi-turn interaction. However, they often do not require tracking evolving information across dialogue turns.

2 related work

  • UMMs combine visual understanding and generation through autoregressive or hybrid architectures, but their capabilities in dialogues with evolving contextual dependencies remain underexplored.Hybrid frameworks combine autoregressive language modeling with diffusion- or flow-based visual generation.Source: §2 · p. 3
  • Existing multimodal datasets mainly provide single-turn supervision for understanding, generation, or editing, with some covering multi-image settings.Recent work also explores sequential editing and interleaved dialogues.Source: §2 · p. 3
  • Benchmarks have expanded from isolated capabilities toward unified, interleaved, and multi-turn settings spanning understanding, generation, and editing.These developments include benchmarks for unified multimodal interaction.Source: §2 · p. 3
  • Many existing datasets and benchmarks can still be approached turn by turn, limiting support for tracking, updating, and faithfully realizing dialogue states across turns.YANAMI jointly provides large-scale supervision and systematic evaluation for these stateful interactions.Source: §2 · pp. 3–4

3 YANAMI

YANAMI combines large-scale stateful dialogue data with a benchmark for evaluating models across understanding, generation, and editing. Its pipeline constructs and verifies dialogues, while its evaluation and statistics describe benchmark coverage and dataset scale.

3 YANAMI

  • YANAMI-400K constructs stateful dialogues from visual materials, while YANAMI-Bench evaluates understanding, generation, and editing.The section introduces data construction, evaluation, and dataset statistics as its three main components.Source: §3 · p. 4, §3.1 · p. 4

3.1 Data Pipeline

  • The data pipeline filters images into four scenario-specific pools, constructs dialogues with scenario- and capability-oriented prompts, and iteratively verifies and regenerates failed samples.The benchmark is also human-verified and corrected, and SHA-256 hash comparisons found no exact image duplicates with the training set.Source: §3.1 · p. 4, Figure 3 · p. 6

3.2 Evaluation Setting and Metrics

  • Four scenarios exercise six state-management capabilities, and the benchmark evaluates dialogue-state understanding alongside visual quality and state consistency.Dialogue-state understanding covers reference resolution, context retrieval, constraint tracking, and correction updating; visual realization assesses visual quality and state consistency.Source: §3.1 · p. 4, §3.2 · pp. 4, 6

3.3 Data Statistics

  • The training set varies in scenario proportions, whereas the test set is balanced across the four scenarios for controlled cross-scenario evaluation.Most dialogues contain at least four images and average 6.56 conversational turns.Source: §3.3 · p. 6, Figure 4 · p. 6, Table 1 · p. 6

4 Experiment

Experiments evaluate models on YANAMI-Bench, test BAGEL fine-tuned on YANAMI-400K, and assess the VLM-as-a-Judge framework. The results expose challenges in stateful interaction and show gains from fine-tuning.

4.2 Train on YANAMI-400K

  • 13.23%: fine-tuned BAGEL improves over original BAGEL on YANAMI-Bench, alongside gains of 10.28% on GEdit-Bench-EN and 2% on GenEval.Fine-tuning largely preserves general visual understanding; MMBench improves 4.1% and POPE 0.9%, while MME-P decreases slightly.Source: Table 2 · p. 7, §4.2 · p. 9

4 Experiment

  • The experiment evaluates 15 models across closed-source, open-source image-editing, and unified multimodal systems, using three image-input organizations and GT-History.GT-History separates models’ use of dialogue context from error propagation caused by previous generations.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1 YANAMI-Bench

  • The best average scores for existing UMMs and open-source image-editing models are 67.41 and 67.30, while GPT-Image-2 and Nano Banana 2 reach 83.05 and 80.93.Visual Reasoning yields the lowest scores for most evaluated models; the evaluation covers 15 models under GT-History.Source: §4.1 · p. 7, Table 2 · p. 7
  • Most UMMs score below 82 on average DSU, and correct state understanding does not guarantee faithful visual realization.The understanding–realization gap is measured on items where models receive full state-understanding credit, using aligned generation-side checks.Source: §4.1 · pp. 7–8, Table 3 · p. 8, Figure 6 · p. 8
  • First-to-last-turn performance drops 2% for BAGEL and 6% each for FLUX.2 and InternVL-U under GT-History.Under Self-History, scores decrease 5.16% for BAGEL and 3.77% for Qwen-Edit compared with GT-History, consistent with error accumulation from self-generated visual histories.Source: §4.1 · p. 8, Figure 6 · p. 8

4.3 Reliability of Judge Usage

  • Qwen-3.8-Flash-Next judge scores correlate above 0.80 with human ratings and show high cross-VLM correlation with GPT-5.6-sol.The human comparison used four expert annotators and 100 instances each from BAGEL+YANAMI, Emu3.5, and Nano Banana 2.Source: §4.3 · p. 10

5 Conclusion

YANAMI combines a large-scale dataset and benchmark for multimodal stateful interaction across understanding, generation, and editing. The conclusion presents it as a foundation for this research, with construction, ethics, and reproducibility information described in accompanying statements.

5 Conclusion

  • YANAMI-400K contains 400K dialogues, more than 2M images, and 2.6M turns across four scenarios and six types of contextual dependency; YANAMI-Bench contains 400 dialogues evaluating state understanding and visual realization.The authors report training gains in contextual understanding, image generation, and image editing while preserving general visual understanding; existing UMMs still struggle to maintain, update, and realize dialogue states.Source: §5 · p. 10

AI Use Statement

  • The authors report using language models and image-generation tools for polishing, code, data filtering, dialogue construction, quality control, image generation, repair, and automated evaluation.The authors state that benchmark samples and annotations were manually verified and that they reviewed AI-assisted writing and code.Source: AI Use Statement · p. 10

Ethics Statement

  • YANAMI is constructed from existing public datasets and model-generated content, with source-dataset and model licenses and usage conditions followed.Automated filtering and human verification are applied to improve data quality.Source: Ethics Statement · p. 10

Reproducibility Statement

  • The authors document construction, quality control, benchmark annotation, evaluation protocols, and fine-tuning configurations, and state plans to release the dataset, benchmark, and prompts.The statement points to appendices for detailed procedures and results.Source: Reproducibility Statement · p. 10

A.1 Comparison with Existing Multimodal Datasets

The appendix distinguishes datasets with explicit dialogue-state management from those that merely use multi-turn formatting, then describes YANAMI’s data preparation, stateful dialogue construction, and verification pipeline.

A.1 Comparison with Existing Multimodal Datasets

  • Table 6 distinguishes multi-turn formatting from explicit state management and asks whether target turns remain irreducible to single-turn tasks when historical inputs are supplied.Scale follows each dataset’s reported unit and is not directly comparable across all rows.Source: Table 6 · p. 15, §A.1 · p. 15

A.2 Data Preparation and Composition

  • Image materials are quality-screened into four scenario-specific pools, with rejected images repaired when possible and groups discarded if more than half their images are rejected.The screening is permissive, rejecting only clearly unusable images; the procedure does not map source datasets one-to-one to interaction scenarios.Source: §A.2 · p. 15

Image Quality Filtering Prompt

  • The filtering prompt treats screening as coarse: it rejects severely corrupted, invalid, or barely visible images, while keeping images with minor or moderate imperfections.When uncertain, the prompt directs the filter to keep the image.Source: Image Quality Filtering Prompt · p. 16
  • 402,639 dialogues comprise Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning shares of 36%, 25%, 28%, and 11%, respectively.The training distribution is retained while the benchmark is balanced at 100 dialogues per scenario for controlled comparison.Source: Image Quality Filtering Prompt · p. 16

A.3 Data Construction

  • Four scenarios instantiate different visual interactions, while six state-management capabilities cut across them; every dialogue uses at least four capabilities and multiple cross-turn dependencies.The capabilities include resolving references, retrieving dialogue history, maintaining cross-turn and spatial consistency, adhering to constraints, and updating state after corrections.Source: §A.3 · pp. 16–17

A.4 Automatic Verification

  • After construction, dialogues are checked for cross-turn logical coherence and image–text consistency, with failed samples corrected or regenerated and re-verified.Passing dialogues form YANAMI-400K or enter the benchmark-candidate pool.Source: §A.4 · p. 17

A.5 Human Verification and Benchmark Construction

YANAMI-Bench is built from automatically verified candidates and then exhaustively checked and corrected by human annotators. Its evaluation separates dialogue-state understanding questions from atomic checks of visual state consistency.

A.5 Human Verification and Benchmark Construction

  • The benchmark balances four scenarios at 100 dialogues each and includes four understanding questions plus atomic visual criteria per dialogue.Atomic criteria assess state consistency separately from holistic visual quality.Source: §A.5 · p. 17
  • Three graduate-level STEM annotators inspect dialogue logic, visual evidence, questions, and criteria, correcting or regenerating samples before review until retained samples are manually verified.The final benchmark has 1,600 understanding questions and 8,467 atomic visual checks.Source: §A.5 · p. 17
  • The verification interface lets annotators inspect full dialogues and visual outputs, revise atomic criteria, and mark samples for approval or image refinement.It supports the benchmark’s exhaustive human verification and correction process.Source: §A.5 · p. 17, Figure 7 · p. 18

A.6 Detailed Data Statistics

YANAMI-400K contains 402,639 dialogues, while YANAMI-Bench contains 400 dialogues distributed evenly across the four scenarios.

A.6 Detailed Data Statistics

  • YANAMI-400K contains 402,639 dialogues, 2,047,236 images, and 2,643,291 user–assistant turns.Dialogues average 5.08 images and 6.56 turns, with a maximum of 43 turns.Source: §A.6 · p. 18
  • 321,868 training dialogues contain at least four images, 215,027 at least five, and 138,669 at least six.Table 7 summarizes corpus statistics; Fig. 4 presents the scenario distribution.Source: §A.6 · p. 18
  • YANAMI-Bench contains 400 dialogues, 1,883 images, 3,173 turns, 1,600 understanding questions, and 8,467 atomic visual checks.Each dialogue has one question for each of the four understanding capabilities.Source: §A.6 · p. 18

B.1 Dialogue-State Understanding Evaluation

The evaluation scores dialogue-state understanding and visual outputs separately, then combines them under defined weighting schemes. It also compares models’ realization of properties they all correctly understood using shared dialogue subsets.

B.1 Dialogue-State Understanding Evaluation

  • Each dialogue has four diagnostic questions covering Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating, scored by semantic correctness rather than lexical match.An answer must include all required information without material error or contradiction; overall understanding is the arithmetic mean of the four capability scores.Source: §B.1 · p. 18

B.2 Visual Quality and State Consistency Evaluation

  • State Consistency is the fraction of atomic visual checks whose required conditions are visibly satisfied; Visual Quality is separately rated from 0–10 for coherence, fidelity, artifacts, and naturalness.Semantically equivalent visual realizations count when they preserve the evaluated state property.Source: §B.2 · p. 20

B.3 Score Calculation

  • For models with understanding and generation, scenario scores use a 4:4:2 weighting over dialogue-state understanding, State Consistency, and Visual Quality.For models without dialogue-state understanding, State Consistency and Visual Quality use a 6:4 weighting, keeping state faithfulness primary while retaining perceptual quality.Source: §B.3 · p. 20

B.4 History and Input Formatting

  • GT-History replaces earlier assistant images with ground-truth images, whereas Self-History feeds each model’s earlier outputs back into later turns, allowing prior visual errors to affect subsequent context.Both settings retain the preceding textual dialogue and unchanged user-provided reference images.Source: §B.4 · p. 20

B.5 Baseline Inference Configuration

  • Models are evaluated on the same dialogues and target turns, with resolution, deterministic sample seeds, and decoding settings specified where supported; closed-source decoding parameters are unavailable.Qwen-3.8-Flash-Next evaluates understanding, State Consistency, and Visual Quality with deterministic decoding and explicit reasoning disabled.Source: §B.5 · p. 21

B.6 Understanding–Realization Gap Evaluation

  • The understanding–realization gap uses shared dialogue–capability pairs where every evaluated model fully answers the understanding question and has aligned generation checks.Scores are averaged within each pair and then across eligible dialogues so dialogues with more checks do not receive disproportionate weight; the analysis covers Constraint Tracking and Correction Updating.Source: §B.6 · pp. 21–22

C Training Details

BAGEL-7B-MoT is fine-tuned on YANAMI-400K with joint visual-generation and visual-understanding objectives. The setup keeps the VAE frozen while training the language model, vision encoder, and understanding module.

C Training Details

  • Fine-tuning uses BAGEL-7B-MoT on NVIDIA A800 and A100 GPUs, with six-GPU A800 and 16-GPU A100 configurations.The provided launch configuration uses six processes on a single node and FSDP under FULL_SHARD.Source: C Training Details · p. 22
  • Training jointly optimizes visual generation and understanding, with the VAE frozen and the language model, vision encoder, and understanding module trainable.The setup uses cross-entropy loss weight 0.25 and MSE loss weight 1.0.Source: C Training Details · p. 22
  • The setup uses a constant learning rate of 1 × 10−5 with 100 warm-up steps and a maximum gradient norm of 1.0.Expected token count is 12,288; overall and per-sample maximum token counts are 16,384.Source: C Training Details · p. 22

D.1 Aggregate Results on YANAMI-Bench

This appendix section describes the reporting and evaluation of YANAMI-Bench results, covering scenario-level and dialogue-progression analyses, shared-subset gaps, qualitative examples, and VLM-as-a-judge prompts. It also specifies a visual-quality rubric and structured scoring format with a brief rationale.

D.1 Aggregate Results on YANAMI-Bench

  • Aggregate scores use valid evaluation records only: GPT-Image-2 has 948 valid outputs, excluding four policy-rejected requests, and Nano Banana 2 has 939.Missing records are not scored as zero.Source: §D.1 · p. 22

D.2 Dialogue-State Understanding Results

  • Dialogue-state understanding reports four capability scores and their mean, while generation results pair understanding with State Consistency and Visual Quality before a weighted overall average.Models limited to image generation or editing have dashes for unsupported understanding scores; scenario scores average RR, CR, CT, and CU, and Avg. macro-averages the scenarios.Source: §D.2 · p. 23, Table 11 · p. 23

D.3 Results across Dialogue Progression

  • Progression results compare first and last generation turns in dialogues with at least two such turns, combining State Consistency and Visual Quality with a 6:4 weighting.The reported difference is Last minus First.Source: §D.3 · p. 23, Table 20 · p. 28

D.4 Understanding–Realization Gap

  • Table 21 compares Constraint Keeping and Correction-Based State Updating gaps on shared subsets understood correctly by all evaluated models; lower gaps indicate smaller discrepancies.Overall is the macro-average of CK and CU.Source: §D.4 · p. 23, Table 21 · p. 28

E Additional Qualitative Examples

F.1 Dialogue-State Understanding Prompt

  • The dialogue-state understanding prompt judges semantic correctness against the dialogue context, question, and reference answer, requiring essential information and no material factual errors.It accepts equivalent wording and relevant correct detail, but rejects materially incomplete or unsupported answers.Source: §F.1 · p. 33

F.2 State Consistency Prompt

  • The State Consistency prompt evaluates atomic and optional capability checks independently against a ground-truth target image, scoring each as the fraction of satisfied checks.When capability checks are absent, their result is an empty array and their score is null.Source: §F.2 · pp. 33–34

F.3 Visual Quality Prompt

  • The Visual Quality prompt assesses generated-image quality on a 0–10 scale across structural coherence, fidelity, artifacts, and naturalness.A composite of multiple images is capped at 4.Source: §F.3 · p. 35
  • The rubric assesses image type, structural correctness, fidelity, artifacts, and overall naturalness and professional quality.It distinguishes single images from composites and checks geometric and anatomical correctness, texture, detail, color, and distortions.Source: §F.3 · p. 35
  • The evaluation must return a JSON object with an integer score from 0 to 10 and a simple explanation.Source: §F.3 · p. 35

G Limitations

YANAMI currently studies multimodal stateful interaction through four representative scenarios and six core state-management capabilities. The authors identify broader domains, languages, modalities, longer interactions, interactive studies, and additional model architectures as directions for extension.

G Limitations

  • The current YANAMI version is bounded to four representative interaction scenarios and six core state-management capabilities as a controlled study foundation.The settings cover forms of understanding, generation, and editing.Source: G Limitations · p. 35
  • Future extensions may add application domains, languages, modalities, and longer interaction horizons.The authors also identify broader interactive studies and evaluation across additional unified multimodal model architectures as valuable extensions.Source: G Limitations · p. 35
Loading 2611.05280v1…