PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

Haoran Wang, Chaofan Ma, Ran Yi, Lizhuang Ma

arXiv:2608.16765v1cs.CVcs.AI

TL;DR

Existing benchmarks organize multi-reference image generation around predefined task types, limiting coverage and diagnostic structure for combinatorial settings. TRACE-Bench instead decomposes tasks into compositional operators and evaluates nine models, finding that disentanglement and attribute binding are the primary bottlenecks rather than scene-level composition.

  • Problem

    Existing multi-reference generation benchmarks organize cases by predefined task types, leaving evaluation coverage and diagnostic structure limited for combinatorial settings.

  • Method

    TRACE-Bench represents multi-reference tasks as formulas over four atomic operators, controlling complexity by operator slots and enabling operator-aligned scoring with diagnostic failure localization.

  • Results

    Nine-model evaluation identifies attribute disentanglement and binding as the primary bottlenecks rather than scene-level composition, with the best model scoring 0.7384 on disentangle.

  • Takeaways & Limitations

    Operator-level diagnosis distinguishes cross-reference interference from standalone capability deficits, exposing failures that holistic scoring does not capture.

  • Takeaways & Limitations

    Shared-layout conditions can cause identity loss or subject duplication, indicating a limitation in group-photo composition.

Abstract

from arXiv · show

Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task types (e.g., "subject composition"), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators: Anchor ($f$), Disentangle ($g$), Apply ($\oplus$), and Compose ($C$). Any multi-reference prompt can then be represented as a compositional formula over these operators, whose structural complexity is quantified by the number of operator slots. Building on this formulation, we construct TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis for recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies in disentanglement ($g$) and attribute binding ($\oplus$) rather than scene-level composition ($C$), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench

1 Introduction

TRACE-Bench reframes multi-reference image-generation evaluation around four atomic operators and compositional formulas, enabling systematic complexity control and operator-level diagnosis. Across 9 models, it shows that disentanglement and attribute binding are stronger bottlenecks than scene composition.

  • Motivation: Existing multi-reference benchmarks cover complex reference settings but retain predefined task structures, limiting systematic coverage and diagnostic value.This motivates a capability-oriented evaluation framework rather than ad-hoc task labels.
  • Capability-oriented formulation: TRACE-Bench represents diverse multi-reference prompts with four atomic operators—Anchor, Disentangle, Apply, and Compose—and compositional formulas over them.The formulation targets the combinatorial space of real-world usage while supporting capability-specific failure diagnosis.
  • Benchmark construction: Approximately 1,600 cases span slot counts 1–8, using 631 formula templates and around 4,000 reference images.The references cover diverse artistic styles and real-world subjects, with slots providing systematic structural-complexity control.
  • Evaluation and findings: 9 leading proprietary and open-source models were evaluated with operator-level analysis that reveals failures hidden by holistic scoring.The protocol is paired with diagnostic tree analysis for recursive failure localization.
  • Evaluation and findings: Attribute disentanglement (g) and attribute binding (⊕) are the primary bottlenecks, whereas scene-level composition (C) is comparatively less difficult.The result indicates that precise reference transfer is substantially harder than producing a plausible scene arrangement.

2 Related Work

Prior work progressed from text-to-image and single-reference personalization toward unified multi-reference generation, but existing benchmarks remain organized around predefined tasks and coarse holistic metrics. TRACE-Bench instead uses four atomic capability operators as a unified basis for construction, evaluation, and diagnostic failure analysis.

  • Reference-based generation: Text-to-image models offer limited fine-grained control, while DreamBooth, Textual Inversion, ControlNet, and IP-Adapter add reference-based conditioning for subject identity or spatial and semantic signals.These approaches primarily address fine-grained control and reference conditioning in text-to-image generation.
  • Multi-reference generation: Single-reference methods are not inherently designed for jointly processing several images, while early multi-subject approaches require per-subject optimization or fixed localized attention.Custom Diffusion and FastComposer extend personalization to multiple concepts but retain these constraints.
  • Multi-reference generation: Recent systems demonstrate strong multi-reference capabilities through unified multimodal architectures or instruction-based generation and editing conditioned on reference images.The passage names GPT-Image-1.5, Nano Banana, OmniGen2, BAGEL, Emu3.5, FLUX.1 Kontext, Qwen-Image-Edit, FireRed Image Edit, DeepGen, and UniReason.
  • Benchmarking: Existing benchmarks organize multi-reference evaluation around predefined task types or surface-level difficulty factors and assess results with holistic scores or coarse metrics such as FID and CLIP similarity.TRACE-Bench addresses these gaps by decomposing generation into four atomic capability operators and using their compositional structure for benchmark construction, operator-aligned evaluation, and diagnostic failure analysis.

3 TRACE-Bench

TRACE-Bench represents multi-reference generation through four atomic capabilities—Anchor, Disentangle, Apply, and Compose—and organizes prompts as compositional formulas over them. This structure supports controllable benchmark construction and operator-aligned evaluation rather than relying only on holistic scores.

  • Core operators: TRACE-Bench decomposes reference ability into Anchor, Disentangle, Apply, and Compose to expose fine-grained failures hidden by holistic evaluation.Anchor preserves an entity’s identity, Disentangle extracts attributes while removing irrelevant properties, Apply binds attributes to entities, and Compose arranges contents into coherent scenes.
  • Compositional formulas: Compositional formulas encode reference-conditioned structure from entity expressions through scene expressions to complete prompt formulas.Formula structure makes referenced entities, interactions, and scene-level conditions explicit for benchmark construction and evaluation.
  • Benchmark construction: TRACE-Bench covers slots 1–8, using operator-slot count as a controllable measure of structural complexity.Slot count measures formula structure rather than fully determining case difficulty.
  • Benchmark statistics: Approximately 1,600 cases are built from 631 distinct formula templates and around 4,000 reference images.The general benchmark contains 180 cases for each slot level, alongside application-specific cases.
  • Operator-aligned evaluation: Operator-aligned question sets are derived automatically from formula terms, replacing a single holistic score with capability-specific evaluation.Each referenced term is grounded to its source image, enabling questions to follow the formula structure.

4 Experiments

TRACE-Bench evaluation shows that proprietary models lead overall, but disentanglement and attribute application remain the main bottlenecks. Operator-aligned scoring and diagnostic trees reveal specific failures caused by reference complexity, style interference, and multi-reference composition.

  • Overall Results: 0.7724 anchor, 0.7384 disentangle, 0.7989 apply, and 0.9100 compose are Nano Banana 2’s operator scores, with proprietary models outperforming open-source baselines.All scores are normalized to an ideal value of 1, leaving the task far from solved.
  • Capability Bottlenecks: Disentangle and apply are the largest gaps because correct attribute extraction and target assignment remain difficult, while GPT-Image-1.5 reaches 0.9259 on compose.Anchor is comparatively stable, but its best score remains 0.7724.
  • Model Comparison: Among open-source models, Emu3.5 performs best overall, and Qwen-Image-Edit-2511 improves over version 2509 on all four metrics.Open-source baselines remain substantially behind leading proprietary systems, especially on disentangle and apply.
  • Difficulty Factors: Anchor performance declines more clearly with increasing reference-image entity count than with slot level, indicating that image complexity is the more direct difficulty source.This analysis compares anchor performance across template slot counts and reference-image entity counts.
  • Diagnostic Analysis: Diagnostic trees localize style-induced interference for f2 and intrinsic anchor difficulty for f1, with composition partially reinforcing f1’s identity.The tree decomposes a failing formula into simpler sub-cases and evaluates the model at each node.
  • Aggregate Diagnostic Patterns: Joint-composition interference dominates localized failures for f, g, and ⊕, whereas global-reference interference accounts for 29.0% of composition failures in Emu3.5.These patterns indicate that isolated reference content is often preserved but lost when multiple reference-conditioned entities are composed.

5 Conclusion · A Task Formulation and Symbolic Representation Details

TRACE-Bench frames multi-reference image generation around four atomic operators whose compositional structure unifies benchmark construction, evaluation, and diagnostic analysis. Its evaluation identifies disentanglement and attribute binding as primary bottlenecks rather than scene-level composition.

  • 5 Conclusion: TRACE-Bench is a capability-oriented benchmark for multi-reference image generation.
  • 5 Conclusion: The framework replaces predefined task types with four atomic operators: Anchor, Disentangle, Apply, and Compose.
  • 5 Conclusion: Operator composition provides a unified basis for benchmark construction, operator-aligned evaluation, and diagnostic tree analysis.
  • 5 Conclusion: Evaluation of 9 leading models identifies disentanglement and attribute binding as the primary bottlenecks.
  • 5 Conclusion: The best model achieves only 0.74 on attribute fidelity, while scene-level composition is less limiting.

A.1 Comparison with Existing Benchmarks

TRACE-Bench differs from existing benchmarks through capability-oriented organization, compositional case construction with aligned evaluation, and richer reference content that makes grounding more challenging.

  • Capability-oriented decomposition: TRACE-Bench organizes cases around four atomic capabilities—Anchor (f), Disentangle (g), Apply (⊕), and Compose (C)—rather than predefined task categories.This design places different generation scenarios within a shared capability space.
  • Compositional case construction with aligned evaluation: TRACE-Bench constructs cases from compositional formulas and uses the same underlying structure to define evaluation targets, unlike benchmarks separating case design from evaluation.The formula connects construction directly with evaluation.
  • Richer reference content and harder grounding: More diverse, information-rich images and comprehensive tagging allow each image to provide multiple reference contents, making Anchor and Disentangle more challenging.References may encode multiple usable contents rather than only one object or simple global attribute, increasing grounding difficulty.

B Benchmark Construction Details … C Evaluation and Diagnostic Details

TRACE-Bench constructs an operator-aligned benchmark through fine-grained attribute tagging, structured reference representations, controlled formula sampling, natural-language realization, and checklist-based evaluation. These components preserve explicit grounding and enable systematic diagnosis of operator-level failures.

  • B.1 Attribute Taxonomy and Tagging Criteria: The attribute taxonomy spans Appearance, Form, Dynamics, and Global, with finer-grained human and humanoid annotations because identity and appearance errors are especially perceptually salient.The taxonomy is designed to balance transferability, perceptual salience, and annotation stability while aligning with the operator-based benchmark formulation.
  • B.2 Structured Tagging Format: Each reference image is represented through ent_list, background, and global_tag, attaching transferable attributes explicitly to grounded foreground instances.The format also preserves salient background content, image-level attributes, and relations when useful for later case construction.
  • B.2 Structured Tagging Format: The structured format additionally supports g_attach for holistic attached content and g_ip for holistic IP-style design transfer.These types preserve clothing, accessories, silhouettes, color schemes, or iconic motifs as unified transferable content rather than isolated local attributes.
  • B.3 Template Construction and Sampling Strategy: Templates are organized from entity-level targets to scene-level composition and full-prompt global references, while deliberately limiting the formula to reference-content combination structure.Scene representations use at most two nested composition levels, with text-specified relations included only when needed for disambiguation.
  • B.3 Template Construction and Sampling Strategy: Template sampling controls slot levels and the proportions of g_rel and g_global terms, yielding both simple transfers and structured relational or global-reference compositions.As slots increase from 1–8, dominant patterns shift toward more complex compositions of the same atomic operators.
  • B.4 Prompt Realization Format: Natural-language realization uses a readable target-scene paragraph followed by explicit source–target assignments for every reference operation.This design retains natural prompting while specifying source images, extracted content, and target objects for visually complex references.
  • B.5 Checklist Construction from Structured Prompts: Evaluation checklists are generated from preserved prompt structure and use multiple binary questions per operator to distinguish object presence, source fidelity, attribute transfer, and binding errors.Questions record their corresponding target parts, enabling traceability and aggregation by operator, entity, or failure mode.
  • C Evaluation and Diagnostic Details: The resulting evaluation is fine-grained and traceable because each checklist question maps back to the original formula, reference prompt, and structured target representation.This supports systematic operator-level analysis rather than relying on a single holistic judgment.

C.1 Reliability of the Evaluation Protocol · C.2 Complete Diagnostic Tree Rules

The evaluation protocol agrees closely with human annotations across four VLM judges, while the diagnostic tree recursively simplifies complex formulas into targeted sub-cases that preserve scene context and localize failures.

  • C.1 Reliability of the Evaluation Protocol: 400 generated outputs from 200 sampled benchmark cases were human-audited using the same binary checklist questions as the VLM judges.The cases were generated with Nano Banana 2 and Emu3.5, and checklist decisions were aggregated into normalized operator-level scores.
  • C.1 Reliability of the Evaluation Protocol: 85.4–88.4% checklist-level agreement with human annotations was achieved by all four VLM judges, with broadly comparable operator-level correlations and errors.G25P was selected for full-benchmark evaluation as a practical reliability–cost trade-off, while the ensemble provided higher-confidence agreement and Pearson correlation.
  • C.2 Complete Diagnostic Tree Rules: The diagnostic tree starts from the full formula and recursively compares model behavior across progressively easier sub-cases to localize failure sources.Each child reduces reference-conditioned complexity while retaining the overall scene; removed reference content is downgraded to text-only descriptions when needed.
  • C.2 Complete Diagnostic Tree Rules: Global-reference stripping removes 𝑔global terms one by one to test whether style or scene-level constraints interfere with anchor or attribute fidelity.This isolates the effect of global references on lower-level capabilities.
  • C.2 Complete Diagnostic Tree Rules: Composition flattening replaces all but one reference-conditioned branch with text-only scene descriptions, isolating failures that arise under joint composition.When multiple anchors cooccur, branches can be examined one anchor at a time while preserving the main scene context.
  • C.2 Complete Diagnostic Tree Rules: Relation simplification converts explicit interactions into text-only relation descriptions before removing participating branches, separating relation grounding from appearance or attribute-transfer failures.Attribute removal likewise preserves carrier entity 𝐸 while removing added attributes in 𝐸⊕𝑔, 𝐸⊕𝑔ip, or 𝐸⊕𝑔attach expressions.
  • C.2 Complete Diagnostic Tree Rules: Decomposition stops at a single informative reference-conditioned unit or when further simplification would not isolate a more specific failure source.Leaf nodes usually represent one anchor, one attribute transfer, or a minimally composed scene, and each node uses only checklist items for its retained operator targets.
  • C.2 Complete Diagnostic Tree Rules: The example decomposition strips a global term, flattens composition, removes a relation, and eliminates transferred attributes to produce simpler branches while preserving scene context.The sequence proceeds through nodes N0, N1, N2a/N2b, N3b1–N3b3, and N4a/N4b.

C.3 Quantitative Validation of Diagnostic Trees

Diagnostic-tree localization agrees with human judgments in 82.6% of Emu3.5 cases. Simplification progressively resolves operator requirements, while persistent endpoint failures indicate both interaction-driven and intrinsic failure sources.

  • Reliability: 82.6% of 200 Emu3.5 cases show agreement between automatic diagnostic-tree localization and human identification of the originating failure node.A VLM evaluates every generated tree node, while human annotators independently inspect the same trees.
  • Solvability Through Simplification: Relative-depth analysis records the earliest point where each operator requirement changes from failure to success along every root-to-leaf path.Depth 0 is the complete case, and depth 1 is the maximally simplified node; cumulative pass rate is measured at each depth.
  • Failure Diagnosis: Cumulative pass rates rise consistently with relative depth for all four operators, but remain below 100% at endpoints.This indicates that removing interfering reference-conditioned components resolves many failures, while some requirements fail even in the simplest diagnostic nodes.

C.4 Additional Diagnostic Tree Examples

The diagnostic tree separates persistent failure to realize a structural reference attribute from unstable behavior that shifts between attribute mismatch and carrier-entity collapse. In this example, GPT-Image-1.5 consistently misses the horse’s head shape, while bear-hat errors vary across simplified nodes.

  • Horse branch: GPT-Image-1.5 never realizes the chess-piece-inspired horse head shape from the root through N3a2, indicating failure in disentanglement-and-apply even after simplification.The error is not introduced only by additional scene complexity.
  • Bear branch: The bear branch is unstable: hat-color mismatch at N0 shifts to bear-generation failure at N2b, while the hat becomes more accurate at N1 and N3b1.The failure therefore involves both attribute fidelity and collapse of the carrier entity.
  • Diagnostic interpretation: The diagnostic tree distinguishes consistently unrealized reference requirements from unstable errors that alternate between attribute mismatch and carrier-level failure.The example uses different node behaviors to localize these qualitatively distinct failure patterns.

D Additional Experimental Results … D.4 More Application-Oriented Formula Abstractions

TRACE-Bench’s additional results show that increasing formula complexity generally reduces performance, with sharper degradation for open-source models, while qualitative comparisons and attribute-subtype analysis expose capability-specific behavior. Application-oriented mappings further demonstrate that diverse practical tasks fit the same operator-aligned formula space.

  • D.1 Performance across Slot Levels: Performance generally declines from slot 1 to slot 8, supporting slot count as a controllable measure of formula structure.The decline is not strictly monotonic for every model because empirical difficulty also depends on reference clutter, entity composition, and attribute granularity.
  • D.1 Performance across Slot Levels: The performance decline is substantially sharper for open-source models, whereas leading proprietary models remain comparatively stable as reference-conditioned elements increase.
  • D.2 Cross-Model Qualitative Comparisons: Under high-complexity slot-8 cases, open-source models often retain multiple reference-conditioned contents, while closed-source models more often drop references and revert to generic text-to-image generation.
  • D.3 Performance across Attribute Subtypes: Overall disentangle performance can hide differences across attribute types, motivating evaluation by attribute subtype.Figure 14 breaks down disentangle performance across fine-grained attribute subtypes defined in the tagging taxonomy.
  • D.4 More Application-Oriented Formula Abstractions: Table 8 maps benchmark-defined task categories and common application-oriented settings into the TRACE-Bench formula space.
  • D.4 More Application-Oriented Formula Abstractions: Diverse realistic task categories can share one formula space by assigning different reference-conditioned content types to the same compositional structure.This extends the abstraction beyond benchmark-specific categories to a broad range of practical generation settings within an operator-aligned framework.

E Potential Extensions

TRACE-Bench’s operator formulation suggests extensions that preserve alignment among benchmark construction, operator targets, and diagnostic questions. Proposed directions include localized model-supervision targets, scalable verification and curation, and explicit difficulty axes for rare concepts and underrepresented combinations.

  • The operator formulation suggests extensions preserving alignment among case construction, operator targets, and diagnostic questions.
  • Localized targets such as direct mask generation could provide structured feedback for reward-guided updates, iterative refinement, and evidence-based revisiting of initial diagnoses.
  • Detector-verifiable properties—including object co-occurrence, position, count, and color—could enable scalable checks for newly sampled formula templates.
  • Future releases could make rare concepts and underrepresented operator–attribute combinations explicit difficulty axes, expanding them through generation with quality filtering.
Loading 2608.16765v1…