PaperlayerView exact revision on arXiv

Source-linked AI summary

ExBind: A Controlled Diagnostic Benchmark for Visual-to-Executable Correspondence

Ziqian Wang, Yuxiao Cheng, Tingxiong Xiao, Jinli Suo

arXiv:2609.01344v1cs.CV

TL;DR

Multimodal coding systems can identify a visible referent yet select the wrong executable object, and final execution success does not diagnose that correspondence failure. ExBind isolates this layer with deterministic latent-to-surface benchmark mappings, finding that candidate validity can mask structural mismatches while table errors localize to correct-row/wrong-column selections. The benchmark is intended for controlled diagnosis rather than population-scale ranking or end-to-end editing evaluation.

  • Problem

    Visual-to-executable correspondence remains distinct from semantic localization and action execution, but end-to-end outcomes do not reveal which structural binding constraint failed.

  • Method

    ExBind compiles representation-independent latent binding instances into SVG, DOM, canvas, tree, graph, and table cases, then scores strict references against deterministic structural constraints.

  • Results

    Qwen2.5-VL-3B has 98.4% candidate validity but 76.4% exact accuracy, while Qwen3-VL-4B reaches 98.8% exact accuracy; targeted-table residuals preserve a correct-row/wrong-column pattern.

  • Takeaways & Limitations

    ExBind separates invalid outputs, identity and relation mismatches, hierarchy and role errors, structured-selection errors, and final-reference errors within a controlled diagnostic framework.

  • Takeaways & Limitations

    The benchmark does not measure full multi-turn coding trajectories, patch generation, execution, or population-scale error prevalence, and candidate serialization can change case outcomes.

Abstract

from arXiv · show

Multimodal coding and editing systems must map a visible or semantic referent to the exact executable object that can be edited. A wrong reference may select a valid but incorrect DOM node, SVG element, graph endpoint, hierarchy member, or table cell, while final execution success alone does not reveal the source of the failure. ExBind isolates this visual-to-executable correspondence layer as a controlled diagnostic benchmark between semantic localization and action execution. It samples representation-independent latent binding instances and compiles them into SVG, DOM, canvas, tree, graph, and table cases with deterministic mappings to executable references. Models output only a strict reference; the evaluator maps predictions back to latent structure and scores structural constraints without requiring reasoning traces. The release contains a 250-case broad suite, a disjoint 240-case targeted suite, and 50 paired latent groups. Qwen2.5-VL-3B achieves 98.4% candidate validity but 76.4% exact accuracy, while Qwen3-VL-4B achieves 100.0% validity and 98.8% exact accuracy. In the targeted table suite, all Qwen2.5-VL-3B residual errors are valid correct-row/wrong-column selections. Candidate-order perturbations change case-level outcomes while preserving this error pattern. ExBind is designed for controlled diagnosis rather than population-scale ranking or end-to-end editing evaluation. Code and benchmark records are available at https://github.com/Daerwang2020/Exbind and https://huggingface.co/datasets/Ziqianwwww/ExBind.

1 Introduction

ExBind isolates visual-to-executable correspondence as a measurable layer between referent localization and action or code execution. Its controlled benchmark construction and diagnostic results distinguish valid outputs from structurally incorrect bindings and expose localized residual error patterns.

  • Motivation: Visual-feedback editing requires mapping a clear visible referent to the exact executable object that a patch should modify.A wrong correspondence can apply a sensible patch to the wrong DOM node, SVG child, component, or code-controlled object.
  • Motivation: End-to-end editing failures conflate localization, executable correspondence, patch generation, execution, and accumulated multi-turn errors.ExBind therefore measures the correspondence layer independently rather than benchmarking the complete editing loop.
  • Motivation: Correct table binding requires resolving the target row, requested attribute column, and their intersection, so a valid candidate can still be structurally wrong.The same distinction applies to wrong executable children, reversed graph roles, and valid DOM elements from the wrong hierarchy.
  • Benchmark design: ExBind compiles representation-independent latent binding objects into SVG, DOM, canvas, tree, graph, and table surfaces with deterministic executable mappings.Models emit only strict references, while evaluators map predictions back to latent structure and report violated constraints without requiring reasoning traces.
  • Benchmark design: 250 broad cases, 240 disjoint targeted cases, and 50 paired latent groups support identifiable structural diagnosis rather than population-scale ranking.The controlled size reflects the need for known latent targets, unique executable gold, deterministic mappings, structural-oracle recovery, and shortcut checks.
  • Results: 98.4% candidate validity versus 76.4% exact accuracy for Qwen2.5-VL-3B shows that valid executable outputs can conceal binding errors.Qwen3-VL-4B reaches 98.8% exact accuracy, while targeted-table residuals for Qwen2.5-VL-3B preserve a correct-row/wrong-column pattern under candidate-order perturbations.

2 Related Work

ExBind extends structured and diagnostic evaluation traditions by making referent-to-executable correspondence the primary object of measurement. It complements grounding and GUI-agent benchmarks without extending the environment or agent horizon.

  • Benchmark distinction: ExBind evaluates whether an environment-specific executable reference denotes an identified referent, with diagnostic labels mechanically recovered from latent targets and executable structure.These labels are not model-generated reasoning traces or process-supervision signals.
  • Grounding and executable correspondence: Prior grounding evaluations commonly stop at regions, coordinates, elements, or actions rather than diagnosing referent-to-executable structural correspondence.ExBind studies the subsequent resolution step after the relevant visual entity has been recognized.
  • Grounding and executable correspondence: ExBind complements GUI-agent evaluations by isolating the binding decision instead of extending the environment or agent horizon.Its decomposition makes hierarchy, ordered roles, and symbolic identity verifiable after semantic or visual localization.
  • Diagnostic evaluation: Diagnostic benchmarks such as CLEVR and GQA motivate separating error sources that aggregate task metrics conflate through controlled scenes, annotations, and structured analyses.ExBind applies this diagnostic orientation to executable correspondence.
  • Structured artifacts: Webpages, interfaces, diagrams, and tables couple visual layout with symbolic structure, motivating evaluations that connect rendered observations to executable representations.Related work includes webpage screenshot-HTML alignment and structured screen annotation, while ExBind targets executable reference resolution.

3 Problem Formulation

ExBind represents executable correspondence through a shared latent binding object and format-specific surface cases. It evaluates strict references by mapping predictions back to latent structure and checking deterministic structural constraints.

  • A latent binding instance contains entities, edges, attributes, hierarchy, relations, and a target binding specification.
  • Format-specific compilers transform the latent object into observations, instructions, candidate sets, surface gold references, and mappings to executable references.
  • Diagnostic labels report applicable structural constraints and, for ordered task families, the first violated constraint under an evaluator-defined decomposition.
  • ExBind distinguishes surface-specific executable targets, structural labels, and paired latent cases across formats from prior benchmark positioning.
  • The model returns one strict surface reference, which ExBind maps back to latent structure when possible for deterministic diagnostic scoring.

4 Benchmark Construction and Validation

ExBind-v2 constructs controlled, disjoint benchmark suites from latent binding instances and validates them with deterministic gates, shortcut checks, interface controls, and sampled taxonomy coverage.

  • The primary protocol contains a 250-case broad suite and a disjoint 240-case targeted structural diagnostic, while the inherited 5.9K inventory remains a legacy resource.
  • The broad suite spans canvas, graph, table, tree, SVG, and HTML-DOM cases, while targeted table cases expose correct-row/wrong-column selections.
  • Cases begin as latent binding instances and are compiled into representation-specific observations and executable candidates, with structural variables explicit in serialized formats.
  • Validation: Acceptance gates require unique gold targets, non-empty candidate sets, deterministic mappings, and a structural oracle that recovers gold targets.
  • Validation: Shortcut checks vary candidate order and exclude target identifiers from instructions, while lexical-only and type-only baselines must fail the targeted diagnostic.
  • Cross-format validation: Cross-format groups support semantic-consistency and paired diagnostic analyses only for explicitly compatible compiler pairs.
  • Taxonomy coverage: In a sampled set of 20 cached errors, all were covered by the existing taxonomy, including 16 wrong-identity and four other structural error cases.

5 Evaluation Protocol

The evaluation uses frozen vision-language models under a strict reference-output contract and measures validity, exact binding, structural failures, survival, and cross-format consistency. It organizes analysis around whether validity hides binding errors and which constraints fail first.

  • Two primary frozen VLMs, Qwen2.5-VL-3B and Qwen3-VL-4B, are evaluated under the same strict reference-output contract.
  • The broad diagnostic suite contains 250 records and 50 cross-format latent groups, while the targeted structural diagnostic contains 240 records split evenly between hierarchy and table cases.
  • Metrics: Metrics include exact executable-reference accuracy, candidate validity, dimension accuracy, violated-constraint sets, task-specific survival, first violated constraints, and cross-format semantic consistency.
  • Metrics: Proportions retain their case denominator, and non-applicable dimensions are excluded rather than counted as incorrect.
  • Evaluation questions: The empirical questions test whether validity hides binding errors, which constraints fail first, whether failures differ across models, and what the targeted table diagnostic reveals.

6 Results

ExBind results show that candidate validity can substantially exceed exact executable binding, while stage-level profiles reveal distinct structural error patterns and interface sensitivity. Across controlled suites, the benchmark distinguishes persistent latent mismatches from surface-dependent outcomes without supporting claims about unique internal mechanisms.

  • 6.1 RQ1: Validity Is Not Binding Correctness: 98.4% validity versus 76.4% exact accuracy for Qwen2.5-VL-3B shows that valid executable references can still bind incorrectly.Qwen3-VL-4B reaches 100.0% validity and 98.8% exact accuracy, with three remaining valid-but-wrong cases.
  • 6.2 RQ2: ExBind Localizes Evaluator-Defined Structural Mismatches: Dimension-level diagnostics expose hierarchy, graph, and table mismatches that aggregate exact accuracy conceals.Figure 2 reports the first violated evaluator-defined constraint, while released score files retain the complete violation set.
  • 6.3 RQ3: Failure Profiles Differ Across Frozen VLMs: Qwen3-VL-4B’s three broad-suite errors are valid outputs involving one canvas relational distractor and two wrong-group HTML-DOM selections.This residual profile differs from Qwen2.5-VL-3B’s concentration in hierarchy, graph, and table binding errors.
  • 6.4 RQ4: Table Binding Localizes a Consistent Column-Misbinding Pattern: 52.5% exact accuracy on the targeted table split leaves every Qwen2.5-VL-3B error as a valid correct-row/wrong-column selection.The declared row–column–cell order identifies table_column as the first violated constraint; Qwen3-VL-4B is exact on all 120 cases.
  • 6.4 RQ4: Table Binding Localizes a Consistent Column-Misbinding Pattern: Candidate-order permutations alter case outcomes while preserving the correct-row/wrong-column residual form, indicating interface sensitivity rather than a reliable accuracy improvement.Across permutations, exact accuracy ranges from 68/120 to 77/120 versus 63/120 originally; the paired bootstrap interval is [-1.1, 16.1] percentage points and crosses zero.
  • 6.5 RQ5: Reliability and Cross-Representation Checks: Both models are exact on all 120 hierarchy-depth cases, while paired cross-format results show stable SVG–canvas behavior but limited tree–DOM consistency.Tree–DOM semantic consistency is 0.80 for Qwen2.5-VL-3B and 0.92 for Qwen3-VL-4B; some hierarchy mismatches persist across surfaces while others are surface-sensitive.

7 Limitations

ExBind’s scope is controlled structural diagnosis rather than full editing evaluation or population-scale ranking. Its measurements are also sensitive to candidate serialization and limited cross-format and prevalence checks.

  • ExBind does not measure error accumulation across complete multi-turn coding trajectories, patch generation, or execution.
  • The benchmark’s primary model comparison uses two related Qwen-family checkpoints, while LLaVA-Phi-3-mini has 55.8% targeted-table validity, limiting matched conditional comparison.
  • Candidate serialization can change case outcomes, so comparisons require the released order or an explicitly matched alternative.
  • Cross-format pairing is limited to compatible compiler pairs and 50 latent groups.
  • The existing-task check is a fixed 20-error sample rather than an estimate of real-agent prevalence.

8 Conclusion

ExBind treats visual or semantic-to-executable correspondence as a structured diagnostic object, separating failure types beyond a single exact-match outcome. Its results show that interface serialization can alter case outcomes while preserving localized error forms across paired latent constructions.

  • ExBind separates latent targets, surface references, and deterministic structural constraints to distinguish invalid outputs from multiple executable mismatch types.
  • High candidate validity can conceal executable mismatches, motivating separate measurement of validity and exact correspondence.
  • Interface serialization can alter case-level outcomes without necessarily changing the localized error form.
  • Paired latent construction extends the analysis across compatible surfaces, revealing both persistent and surface-sensitive transitions.

Reproducibility Statement

The ExBind v2.0 release provides versioned benchmark artifacts, source code, cached predictions, coverage annotations, and reproduction instructions. These materials support regeneration, evaluation, and artifact-level verification.

  • All paper results correspond to the ExBind v2.0 release manifest in the artifact directory.
  • The source tree contains the benchmark schema and evaluation code.
  • Versioned artifacts include the broad suite, targeted diagnostic, table audit, adjudication pack, explicit-column ablation, cached predictions, coverage annotations, and legacy bridge subset.
  • REPRODUCE_EXBIND.md and Appendix A list tested command lines for generation and frozen-model execution.

A.1 Data Generation

ExBind data generation samples latent binding instances and target specifications, compiles them into representation-specific observations, constructs executable candidates and gold references, and derives deterministic stage labels. Integrity gates reject cases that fail structural or shortcut checks before storing complete records.

  • Data-generation configuration: The targeted structural diagnostic uses seed 20260824, generator version exbind_v2.hard_pilot.v1, and 40 examples per family-level cell.
  • Algorithm 1: ExBind Case Generation: The generator compiles each latent instance into an observation Of = Kf(G) for the selected representation family.
  • Algorithm 1: ExBind Case Generation: Algorithm 1 samples a latent binding instance G = (V, E, A, H, R, Y ) and a target specification containing the canonical semantic target.
  • Algorithm 1: ExBind Case Generation: It constructs the executable candidate inventory from surface structure and derives surface gold through the latent-to-surface mapping Mf.
  • Algorithm 1: ExBind Case Generation: Deterministic stage labels are derived from the latent instance, candidates, target specifications, surface gold, and mapping.
  • Integrity and storage: Integrity gates check unique gold, non-empty candidates, structural-oracle recovery, target-ID leakage, lexical-only and type-only baselines, and candidate-order shortcuts.

A.2 Benchmark Record Schema

ExBind represents generated cases as sparse BindingRecord objects and evaluates strict executable references through deterministic structural checks. Its release artifacts preserve stage-level diagnostics, applicability denominators, paired transitions, and candidate-order controls for reproducible analysis.

  • BindingRecord objects store generated and converted cases with null values for task dimensions that do not apply.A simplified table record includes case, latent, format, instruction, observation, candidates, gold, binding, mappings, and applicable stages.
  • The evaluator parses strict JSON, normalizes aliases, validates candidate identifiers, maps surface references to latent targets, and emits deterministic diagnostic scores.Unknown candidate identifiers are invalid, while reversed graph endpoints are scored as role/order failures.
  • The frozen-model protocol uses one ordinary reference-output contract across checkpoints, with format-specific schemas for single targets, graph pairs, DOM elements, and table cells.Rendered images use the model processor chat template; non-image observations are serialized as SVG, HTML, DOM, graph, or table source text.
  • Candidate-order controls and paired latent-group analyses preserve permutations, predictions, scores, transitions, rank mappings, and experiment manifests as versioned artifacts.Tree–DOM transition counts are reported for 25 groups, while applicability tables expose evaluator-defined opportunity denominators.
  • Constraint Sets and Declared-Order Summaries: The release records stage-level score vectors and violated constraints, while first-violation labels follow declared task-family orders such as row/entity → column → cell → exact reference.The main paper uses declared order for concise summaries and retains full possible first-constraint sets for order-independent analysis.
  • The benchmark’s diagnostic categories support only fixed-sample localization claims, not causal probabilities, model-internal reasoning stages, real-agent failure frequencies, or all open-ended editing trajectories.Declared first-violation labels are not invariant to every alternative decomposition, and the coverage artifact documents the fixed sampled subset.
Loading 2609.01344v1…