PaperlayerView exact revision on arXiv

Source-linked AI summary

IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

Shaohan Yu, Jie Qin, Wende Tan, Haiwen Diao, Limeng Qiao, Yueming Lyu, Chenyang Si

arXiv:2610.38425v1cs.CV

TL;DR

Interleaved text and images are common, but generation remains underexplored amid gaps in task-specific training data and fine-grained evaluation. IntReal pairs a real-world dataset with a seven-dimension expert evaluation suite. Its experiments report reliable evaluation and show that modality-specific post-training can improve interleaved generation.

  • Problem

    Multimodal interleaved generation lacks well-defined tasks, generation-oriented training data, and fine-grained evaluation across capability dimensions.

    Source: §1 · p. 2

  • Method

    IntReal constructs a taxonomy-guided real-world dataset and an evaluation suite using coarse-to-fine settings and specialized experts across seven dimensions.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The suite's metrics show low redundancy, 83.3% agreement with human judgments, and Spearman correlation above 0.9 across expert evaluators; Flow-GRPO recovers SFT-degraded image generation and leads evaluated open-source UMMs overall.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Training a single generation modality under interleaved contexts can improve interleaved generation, extending optimization beyond joint SFT on interleaved sequences.

    Source: §1 · p. 3

Abstract

from arXiv · show

Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. Although Unified Multimodal Models (UMMs) can jointly generate text and images, multimodal interleaved generation remains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-defined interleaved generation tasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components: IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and the IntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specialized expert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman's $ρ\leq 0.42$ ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman's $ρ> 0.9$ ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show that IntReal Evaluation Suite reveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility of IntReal-100K for improving interleaved generation and further show that training unimodal generation can also benefit interleaved generation. These findings suggest that interleaved generation can benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.

1 INTRODUCTION

Multimodal interleaved generation is underexplored, with gaps in generation-oriented datasets and fine-grained evaluation. IntReal introduces a real-world dataset and expert-based evaluation suite, and experiments show that modality-specific post-training can improve interleaved generation.

1 INTRODUCTION

  • The paper identifies two gaps: existing interleaved datasets are not designed and quality-controlled for generation, and holistic VLM evaluation does not diagnose distinct capability dimensions.Prior fine-grained evaluation is described as lacking capability-aligned decomposition and, in one case, relying on a single VLM evaluator.Source: §1 · p. 2
  • 83.3% human-judgment agreement, maximum pairwise absolute Spearman correlation below 0.42, and inter-evaluator Spearman correlation above 0.9 support the suite's validation.The excerpt also reports that the metrics capture distinct capability dimensions and evaluations remain consistent across expert evaluators.Source: §1 · pp. 2–3
  • Flow-GRPO recovers image-generation capability degraded by SFT and achieves the highest overall IntReal score among the evaluated open-source UMMs.The results suggest that training a single generation modality can improve interleaved generation, beyond conventional SFT on multimodal interleaved data.Source: §1 · p. 3
  • IntReal-100K provides 100K high-quality real-world interleaved sequences, while the Evaluation Suite assesses generation across seven capability dimensions using coarse-to-fine evaluation and specialized experts.The dataset is organized around four real-world generation scenarios.Source: §1 · pp. 2–3

2 REAL-WORLD INTERLEAVED DATASET CONSTRUCTION

IntReal characterizes real-world interleaved content with four scenario categories and uses a staged curation pipeline to produce a high-quality training dataset. The resulting dataset retains a small fraction of collected samples, while the accompanying figure describes two evaluation granularities.

2.1 CHARACTERIZING REAL-WORLD INTERLEAVED CONTENT

  • The taxonomy identifies Planning, Knowledge, Tutorial, and Story, distinguishing content by how text and images coordinate goals, explanations, procedures, or narratives.The categories are drawn from real-world interleaved content collected from web pages and videos.Source: §2.1 · p. 3

2.2 DATA COLLECTION AND CURATION

  • Curation proceeds from low-cost filtering to semantic polishing, combining image-quality filtering, text polishing, and structured annotation for UMM fine-tuning.The stages include metadata and representation-based filters, VLM-based refinement that prioritizes original text, and LLM-based prompt and reasoning annotation.Source: §2.2 · pp. 3–4
  • Figure 2 distinguishes holistic evaluation of a complete interleaved sequence from fine-grained generation of the next image conditioned on preceding context.The figure caption describes selecting 600 high-quality samples from IntReal-100K for the evaluation suite.Source: Figure 2 · p. 4
  • Only about 10% of collected samples are retained after category-balanced sampling, yielding IntReal-100K.A defective segment anywhere in a long interleaved sample leads to removal.Source: §2.2 · p. 4

3 REAL-WORLD DATA-DRIVEN EVALUATION

The IntReal Evaluation Suite uses real-world samples and a coarse-to-fine protocol to assess interleaved generation. It pairs a curated, category-balanced set with holistic and next-image evaluation metrics, including specialized expert assessments.

3.1 EVALUATION SET CONSTRUCTION

  • The candidate pool is screened for text quality, image quality, and context-aware text-image alignment before manual curation.For alignment, the evaluator derives visual rubrics from preceding context without seeing the target image, then checks the image against those rubrics.Source: §3.1 · p. 5
  • The evaluation set contains 600 manually curated, stratified samples, with 150 per category, selected from an automatically filtered candidate pool.The set is kept compact because the protocol requires generating over 6,000 images interleaved with text per model, increasing evaluation cost if scaled further.Source: §3.1 · p. 5

3.2 EXPERT MODEL-BASED COARSE-TO-FINE BENCHMARKING

  • Holistic evaluation generates complete interleaved responses from scratch, while fine-grained evaluation tests next-image generation under fixed real-world contexts for controlled comparison.The suite combines these settings with specialized expert models to measure distinct capabilities.Source: §3 · p. 4, §3.2 · p. 5
  • Holistic metrics assess text richness, text logicality, image aesthetics, image diversity, and compliance with the sample's image count.Image diversity uses adjacent-image DINO feature similarity, with extremely high similarity indicating potential repetition.Source: §3.2 · p. 5
  • Fine-grained metrics measure context fidelity and subject consistency, with subject consistency computed only for Tutorial and Story.Knowledge and Planning do not necessarily require consistent subjects across images; Table 3 reports scores scaled from 0 to 100, with higher values better.Source: §3.2 · p. 5, Table 3 · p. 6

4 EXPERIMENT

Across the evaluated models, capabilities vary by dimension and category; post-training shows both trade-offs and image-generation gains that transfer to interleaved generation. The evaluation suite’s metrics show limited redundancy, human agreement, and consistency across expert evaluators.

4.1 EXPERIMENTAL SETUP

  • The experiments evaluate four UMM families under differing support for interleaved inference and fine-tuning, using post-training samples disjoint from the 600 evaluation-suite samples.Fine-tuned results are reported only for SenseNova-U1-8B-MoT; Bagel is evaluated after fine-tuning, while Emu3.5 is evaluated at its original checkpoint.Source: §4.1 · p. 6

4.2 BENCHMARKING UMMS ON INTREAL

  • No model performs consistently well across all dimensions: Emu3.5 leads open-source models in Text Logicality, while U1-8B-MoT is stronger in Quantity Compliance and Context Fidelity.NanoBanana2 leads on some unimodal dimensions but trails the best open-source results on Image Diversity and Context Fidelity.Source: §4.2 · p. 7
  • Planning and Knowledge pose greater challenges for text metrics, Tutorial for Context Fidelity and Subject Consistency, and Story for Quantity Compliance and Image Diversity.Figure 3 reports each metric by category, averaging scores over UMMs.Source: §4.2 · p. 7, Figure 3 · p. 7
  • Open-source UMMs show a larger gap from the closed-source API in text than in image capabilities.The authors suggest scaling model capacity may be important for stronger interleaved generation.Source: §4.2 · p. 7

4.3 POST-TRAINING ON INTREAL-100K

  • 29.04%, 6.68%, and 6.53% gains in Text Richness, Text Logicality, and Subject Consistency accompany SFT, while Image Aesthetic and Context Fidelity decrease by 5.65% and 4.89%.The authors attribute the degradation potentially to an optimization conflict between understanding and generation.Source: §4.3 · p. 7
  • U1-8B-MoT with SFT and RL achieves the highest mean score among OSS models, while RL mitigates SFT’s image-quality degradation and further improves Image Diversity.A similar Image Aesthetic and Context Fidelity trend holds for Bagel, whose already-high Image Diversity remains stable.Source: §4.3 · p. 7

4.4 RELIABILITY OF THE INTREAL EVALUATION

  • Below 0.42 absolute pairwise Spearman correlation, 83.3% agreement with expert judgments, and above 0.9 ranking correlations across evaluators support metric complementarity and evaluation consistency.The human-agreement analysis covers five subjective metrics; evaluator consistency is reported for Text Richness, Text Logicality, and Context Fidelity.Source: Figure 4 · p. 7, §4.4 · p. 8, Figure 5 · p. 8

5 RELATED WORK

Prior work has explored unified text-image generation and native interleaved-sequence modeling, while interleaved datasets and benchmarks have expanded from understanding and holistic evaluation toward generation-oriented data and finer-grained assessment.

5 RELATED WORK

  • Prior UMM work unifies text and image generation [Ramesh et al., 2021; Sun et al., 2024] or treats interleaved sequences as native pretraining [Cui et al., 2025; Tong et al., 2026].Some models extend multimodal conditioning to image editing and other any-to-image tasks.Source: §5 · p. 8
  • Chameleon exemplifies mixed-modal sequence modeling, while Bagel provides an open-source realization based on interleaved multimodal data [Team, 2024; Deng et al., 2025].Source: §5 · pp. 8–9
  • Earlier interleaved datasets mainly supported understanding and pretraining [Zhu et al., 2023; Laurenc¸on et al., 2023], while CoMM added generation-oriented filtering [Chen et al., 2025c].More recent studies address multimodal reasoning traces [Gu et al., 2026; Li et al., 2026a].Source: §5 · p. 9
  • Early benchmarks used downstream tasks and holistic VLM assessment; UEval and ISG-Bench instead decouple responses into structured evaluation targets [Li et al., 2026b; Chen et al., 2025a].Other benchmarks examine bidirectional understanding-generation synergy, multi-turn contexts, or broader omni-modal input-output combinations.Source: §5 · p. 9

6 CONCLUSION

IntReal combines a taxonomy-guided real-world dataset with a seven-dimension evaluation suite and studies their use in UMM post-training. The experiments find modality-specific improvements can transfer to interleaved generation, alongside data-retention and training-setting limitations.

6 CONCLUSION

  • IntReal constructs IntReal-100K from real-world content using a four-scenario taxonomy and introduces a coarse-to-fine evaluation suite with specialized expert models across seven dimensions.Source: §6 · p. 9
  • SFT on IntReal-100K brings gains and capability trade-offs, while Flow-GRPO uses the proposed metrics as learning signals for next-image generation under interleaved contexts.Source: §6 · p. 9
  • Improving unimodal generation can generalize to interleaved generation, suggesting modality-specific optimization as an option beyond joint text-image optimization.Source: §6 · p. 9
  • High-quality real-world data construction has a relatively low retention rate after aggressive filtering, and mixed post-training with unimodal data was not explored.Synthetic data augmentation and mixed-data post-training are left for future work.Source: §6 · p. 9

A.1 DATA SOURCES AND LICENSING

IntReal draws on travel, encyclopedia, cooking, and animated-story sources, with source-specific licensing and provenance practices. It applies a multi-stage filtering and annotation pipeline, producing a 100K-sample dataset with 538K images and 146.6M text tokens.

A.1 DATA SOURCES AND LICENSING

  • Planning data come from Wikivoyage and Wikitravel, Wikipedia supplies knowledge content, Xiachufang supplies cooking sequences, and three animated series provide story samples.Source: §A.1 · pp. 14–15
  • Source-specific licensing checks preserve provenance and exclude content whose terms do not permit the intended redistribution or research use.The dataset is intended solely for non-commercial academic research, and users remain responsible for complying with original sources’ terms.Source: §A.1 · pp. 14–15

A.2 DETAILED FILTERING PIPELINE

  • The unified filtering pipeline adapts criteria by source, then uses VLM captioning and LLM refinement grounded in original samples to prepare training supervision.The process is described as improving text-image consistency while preserving factual authenticity.Source: §A.2 · pp. 15–16
  • Filtering combines source-specific image-resolution and content checks with quality scoring and sequence deduplication, while avoiding face removal where faces or recurring characters are semantically important.Xiachufang uses face and watermark filtering; the Wiki and story sources do not use face filtering, and all sources receive RALI quality filtering and DINO-based deduplication.Source: §A.2 · p. 16
  • A four-stage refinement pipeline adds image captions, refines source text, creates sequence- and image-level reasoning annotations, and jointly improves prompt-content coherence.The stated aim is improved multimodal alignment and structured supervision while preserving factual fidelity.Source: §A.2 · p. 16

A.3 DATASET STATISTICS

  • 100K samples contain 538K images and 146.6M text tokens, with relatively balanced category coverage and story samples averaging around eight images per sample.Planning samples are relatively concise, while Knowledge and Tutorial samples contain richer multimodal content.Source: §A.3 · p. 16

B.1 EVALUATION SET CONSTRUCTION

The evaluation set is constructed through quality and context checks, followed by human review and stratified sampling. This yields 600 samples balanced across four categories and different output-image counts.

B.1 EVALUATION SET CONSTRUCTION

  • Evaluation candidates must average at least 3.8 in image aesthetic score and contain 3–10 output images; content screening requires prompt specificity ≥3, response richness ≥4, and topic informativeness ≥4.Source: §B.1 · pp. 16–17
  • Context-aware verification retains a sample only if every image satisfies at least one context-derived rubric and the sample’s rubric pass rate is at least 50%.The rubric criterion is relaxed because naturally occurring interleaved content may have imperfect image-text correspondence.Source: §B.1 · p. 17
  • After manual review of approximately 1.5K automatically filtered candidates, stratified sampling produces a balanced evaluation set of 600 samples.Sampling is stratified across the four categories and different output image counts.Source: §B.1 · p. 17

B.2 METRIC DEFINITIONS

IntReal defines seven metrics for evaluating interleaved generation, covering text quality, image quality and compliance, contextual fidelity, and subject consistency. It also provides prompts for expert evaluation of text richness, text logicality, and context fidelity, while cautioning that the overall score is only a coarse comparison aid.

B.2 METRIC DEFINITIONS

  • Text Richness compares generated and real-world responses for informativeness, completeness, and redundancy, counting a model win or tie as successful.The evaluation prompt asks an LLM to compare the responses directly, alternating their order to reduce positional bias.Source: §B.2 · pp. 17–18, §B.3 · p. 19
  • Text Logicality checks whether generated text follows prompt constraints, avoids contradictions and abrupt transitions, and remains plausible for non-creative tasks.Its prompt allows imaginative content in creative tasks while judging internal consistency.Source: §B.2 · p. 18, §B.3 · p. 20
  • Image Aesthetic normalizes per-image RALI scores before averaging, while Image Diversity penalizes excessive similarity between adjacent generated images.Image Diversity uses a similarity threshold of 0.85 and is determined by the most repetitive adjacent pair.Source: §B.2 · p. 18
  • Quantity Compliance requires an exact match between generated and reference image counts, while Context Fidelity scores satisfaction of visual rubrics implied by preceding context.Context Fidelity assigns zero when no expected visual properties are satisfied and increases as more rubrics are met.Source: §B.2 · pp. 18–19
  • Subject Consistency compares generated images with position-aligned real images using DINOv3 feature cosine similarity in Story and Tutorial settings.The overall score averages the seven metric scores with equal weights.Source: §B.2 · p. 19
  • The overall score equally weights the seven metric scores after averaging valid samples, but is intended for coarse comparison rather than definitive quality judgments.The authors recommend examining individual dimensions to understand model strengths and weaknesses.Source: §B.2 · p. 19

B.4 METRIC CALIBRATION CASES

Representative low-scoring examples illustrate the failure patterns captured by IntReal’s metrics, from weak text detail and logic to image-quality, quantity, context, and consistency errors.

B.4 METRIC CALIBRATION CASES

  • Low Text Richness reflects insufficiently specific insect characteristics and limited detail for distinguishing the target from similar species.The example is from Bagel+SFT.Source: Figure 8 · p. 20
  • Low Text Logicality reflects spatial and factual inconsistencies in the description.The example is from U1-8B-MoT.Source: Figure 9 · p. 21
  • Low Image Aesthetic reflects excessive saturation and contrast across later images, which appear less photorealistic.The example is from U1-8B-MoT-SFT.Source: Figure 10 · p. 21
  • Low Image Diversity reflects repeated similarity across image pairs 1–2, 3–4, and 5–6.The example is from Emu3.5.Source: Figure 11 · p. 21
  • Low Quantity Compliance reflects generating six images when the prompt requests eight.The example is from Bagel+SFT+RL.Source: Figure 12 · p. 21
  • Low Context Fidelity reflects insufficiently toasted bread and omission of the caramelized-onion preparation and spreading steps.The example is from U1-8B-MoT+SFT+RL.Source: Figure 13 · p. 22
  • Low Subject Consistency reflects changes in the dinosaur across images and a teddy bear changing color, unlike the stable reference subjects.The example is from U1-A3B-MoT.Source: Figure 14 · p. 22
  • The appendix also reports evaluation results across seven dimensions with Gemma4-31B and with Qwen3.6-35B-A3B as the alternative evaluator.Figures 15 and 16 show the two evaluator configurations.Source: Figure 15 · p. 22, Figure 16 · p. 22

C.1 MODEL TRAINING SETTINGS

Training and inference settings accommodate differences in model representations and visual-token budgets. Evaluation exclusions leave Emu3.5 and NanoBanana2 with fewer than 600 samples.

C.1 MODEL TRAINING SETTINGS

  • Bagel and SenseNova-U1 use model-specific resizing to accommodate different visual representations and token budgets while preserving aspect ratios and aligning dimensions to required grids.Bagel uses separate VAE and ViT resizing settings; SenseNova-U1 uses native-resolution resizing constrained by image-area ranges.Source: §C.1 · p. 23
  • RL post-training starts from SFT checkpoints and uses different samplers and guidance configurations for Bagel and SenseNova-U1.Bagel uses Flow-CPS without CFG; SenseNova-U1 retains Flow-SDE with Hybrid CFG, and both use aesthetic rewards.Source: §C.1 · pp. 23–24
  • SFT runs use one node of 8×H800-80G GPUs, while RL runs use two such nodes.During inference, models switch from autoregressive text generation to image generation at the <image> token; 1024 × 1024 is the recommended default resolution.Source: §C.1 · p. 24, §C.2 · p. 24

C.2 MODEL INFERENCE SETTINGS

  • Emu3.5 results use 592 samples because eight exceed context or memory limits, while NanoBanana2 results use 598 after two samples trigger API safety guardrails.Manual inspection found no apparent safety concerns in the two NanoBanana2 samples, suggesting the guardrails may have been falsely triggered.Source: §C.2 · p. 24

C.3 DETAILED EVALUATION RESULTS

Alternative-evaluator results largely preserve scores and model rankings, with a notable absolute-score discrepancy for Text Logicality. Category-wise results reveal different challenges for open-source models, while the appendix illustrates interleaved outputs across four task types.

C.3 DETAILED EVALUATION RESULTS

  • Across most dimensions, Qwen3.6-35B-A3B and Gemma4 produce nearly identical scores and highly consistent model rankings, but NanoBanana2’s Text Logicality scores differ by 12.71 points.The authors attribute the larger absolute discrepancy to independent binary coherence judgments without an explicit comparative anchor; relative rankings remain largely preserved.Source: §C.3 · pp. 24–25
  • Open-source models face greater Text Richness and Text Logicality challenges in Planning and Knowledge, Context Fidelity and Subject Consistency challenges in Tutorial, and Quantity Compliance and Image Diversity challenges in Story.NanoBanana2 shows consistently strong Text Richness and Text Logicality across categories.Source: §C.3 · p. 25, Figure 17 · p. 25

D QUALITATIVE EXAMPLES

  • The qualitative examples show interleaved outputs for Planning, Knowledge, Tutorial, and Story, respectively covering London-area geography, damselfly biology, roasting tomatoes, and dinosaurs exploring a jungle.The examples were generated by Nanobanana2, Emu3.5, U1-8B-MoT+SFT+RL, and Bagel+SFT+RL, respectively.Source: Figure 18 · p. 26, Figure 19 · p. 27, Figure 20 · p. 28

Related papers

Loading 2610.38425v1…