PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

Zhuchenyang Liu, Yao Zhang, Yu Xiao

arXiv:2604.00913v2cs.CVcs.CL

TL;DR

Assembly assistants need to align abstract 2D diagrams with live video, but their differing visual depictions create an under-evaluated challenge. This paper introduces IKEA-Bench and evaluates VLMs across alignment strategies, finding that text aids instruction comprehension yet does not solve cross-depiction alignment, whose main bottleneck lies in visual encoding and video understanding.

  • Problem

    Abstract diagrams and photorealistic assembly videos share few visual features, making their alignment for mixed-reality guidance an under-evaluated research problem.

  • Method

    The paper builds IKEA-Bench with 1,623 questions across six task types and 29 IKEA products, evaluates 19 VLMs under three alignment strategies, and performs three-layer mechanistic analysis.

  • Results

    Text improves instruction comprehension by 23.6 percentage points but reduces T1 diagram-to-video alignment by 3.1 points on average; architecture family matters more than parameter count, while video discrimination remains low.

  • Takeaways & Limitations

    Improving cross-depiction robustness should target visual encoding, while practical systems may select alignment strategies according to the subtask.

  • Takeaways & Limitations

    The benchmark covers only IKEA furniture assembly, and whether its findings generalize to other assembly domains is untested.

Abstract

from arXiv · show

2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions. Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-video alignment; (2) architecture family predicts alignment accuracy more strongly than parameter count; (3) video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: https://ryenhails.github.io/IKEA-Bench/

1 Introduction

IKEA-Bench addresses the depiction gap between abstract assembly diagrams and photorealistic assembly video by benchmarking VLM alignment and analyzing its mechanisms. The evaluation shows that text helps instruction comprehension but can hurt cross-depiction alignment, while visual encoding and video understanding remain central challenges.

  • Motivation: 2D furniture diagrams are abstract and difficult to follow, motivating assistants that monitor progress, detect errors, and provide step-by-step guidance.Mixed-reality systems must align diagram instructions with the user’s current step in a live video feed.
  • The depiction gap: The depiction gap arises because diagrams use arrows, exploded views, and silhouettes, whereas video shows hands assembling furniture in cluttered photorealistic scenes.Unlike static cross-depiction recognition, assembly alignment also depends on the temporal stage of the procedure.
  • Benchmark: IKEA-Bench contains 1,623 questions across six task types, 29 IKEA products, and three alignment strategies for evaluating foundational diagram-to-video alignment.The tasks include four alignment tasks plus video discrimination and instruction comprehension diagnostics.
  • Findings: +23.6 pp on D2 shows that assembly instruction comprehension is recoverable via text, but −3.1 pp on T1 shows that adding text degrades diagram-to-video alignment on average.Architecture family predicts alignment accuracy more strongly than parameter count, and video discrimination remains a hard ceiling across strategies.
  • Mechanistic analysis: Diagrams and video occupy disjoint ViT representational subspaces (CKA ≈0), while text shifts reasoning away from visual representations toward text.The shift is confirmed in both LLM hidden states and attention routing.
  • Contributions: The paper proposes a benchmark, mechanistic analysis, and design guidelines for alignment strategy selection, architecture choice, and future improvement.Its analysis localizes cross-depiction failure to the visual encoder and identifies text as a distractor from visual processing.

2 Related Work

Prior work studies cross-depiction recognition, procedural video reasoning, and technical-image understanding largely in isolation. This paper differs by evaluating alignment between visual assembly instructions and temporal assembly video, then analyzing the failure mechanism across modalities.

  • Cross-depiction recognition: Cross-depiction research recognizes objects across photographs, paintings, and drawings, with spatial-part models helping handle differing visual styles.These studies focus on static object recognition rather than temporal procedures.
  • Procedural video: Procedural-video benchmarks examine temporal and causal reasoning, but use text instructions as the procedural specification.They therefore do not test visual instructions such as assembly diagrams.
  • Technical images: Technical-image benchmarks evaluate diagram comprehension or multi-image reasoning in isolation, whereas IKEA-Bench tests diagram-to-video alignment.The distinction is the cross-depiction relationship between schematic instructions and video.
  • Benchmark positioning: Figure 2 organizes the benchmark into four increasingly difficult cross-depiction tasks and two unimodal diagnostic tasks.D1 isolates video discrimination, while D2 isolates assembly-instruction understanding.
  • Mechanistic analysis: Prior mechanistic studies trace visual-information degradation or multimodal attention within a single depiction; this work analyzes failure across two depictions of the same content.The analysis targets why diagrams and video fail to align rather than how one depiction is processed internally.

3 IKEA-Bench

IKEA-Bench is constructed from temporally aligned IKEA assembly manuals and videos, with tasks and distractors designed to test genuine cross-depiction reasoning. It compares visual, text-mediated, and text-only inputs while isolating video and instruction bottlenecks.

  • Source data: IKEA Manuals at Work uniquely supplies manual step diagrams, assembly videos, and dense temporal annotations linking each step to its video interval.This makes it suitable for constructing diagram-to-video alignment questions.
  • Source data: The benchmark retains 29 of 36 products and extracts 2,569 video frames across 97 assembly videos after excluding short or low-quality products.The retained data support temporally grounded task construction.
  • Source data: Wordless diagrams and fine-grained sequential structure create hard negatives because adjacent steps show the same parts at different assembly stages.This design makes coarse part-presence shortcuts less reliable.
  • Task design: The six tasks cover step recognition, verification, progress tracking, next-step prediction, video discrimination, and instruction comprehension.The first four target assembly guidance prerequisites, while D1 and D2 isolate unimodal bottlenecks.
  • Task design: T1 and T4 use diagram thumbnails as answer options, T3 presents all product diagrams, and D2 uses 65 consecutive triplets to test ordering.Consecutive triplets preserve visual similarity and prevent ordering through coarse part matching.
  • Anti-shortcut design: Adjacent-step distractors and randomized binary options are used to prevent shortcuts and balance response distributions.Video frames are uniformly sampled from each ground-truth temporal segment, with adjacent diagrams serving as visually similar distractors.
  • Alignment strategies: Visual presents diagrams and video, Visual+Text adds descriptions to diagrams, and Text Only replaces diagrams with descriptions.Text Only is a diagnostic control rather than a cross-depiction alignment condition.
  • Alignment strategies: Step descriptions are generated across eight dimensions and cross-validated against ground-truth part and connection annotations.This supports the text-mediated alignment strategies.

4 Experiments

The experiments evaluate VLMs across model families, scales, alignment strategies, and diagnostic probes to locate sources of cross-depiction failure. They combine benchmark evaluation with representation, retrieval, modality-influence, and attention analyses.

  • Evaluation setup: 17 open-source VLMs from 8 families spanning 2B–38B parameters, plus two proprietary Gemini models, are evaluated under three alignment strategies.The open-source models cover small, medium, and large size tiers; Gemini models are evaluated in the Visual setting via API.
  • Evaluation setup: All models are evaluated zero-shot with greedy decoding across 51 runs covering six tasks, with accuracy as the primary metric.The overall answer parse rate is 94.1%, and parsing failures are counted as incorrect.
  • Layer 1: Visual representations: Frozen ViT features from 113 matched diagram–video step pairs are compared to test whether the visual encoder provides a shared representational space.The analysis extracts mean-pooled diagram vectors and video vectors averaged across sampled frames, then computes CKA at ViT-output and post-merger levels.
  • Layer 2: Video features: A linear probe tests whether video features encode temporal structure by classifying whether two frames depict the same assembly step.The classifier uses concatenated frame representations, their absolute difference, and negative pairs from different steps of the same product.
  • Layer 1: Cross-modal alignment: Cross-modal retrieval measures diagram–video alignment by retrieving nearest video frames from a 2,546-frame gallery and reporting Recall@1 and Recall@10.Each retrieved frame is checked against the matching assembly step.
  • Layer 2: Modality influence: Modality-influence analysis compares the final prediction vector with average diagram, video, and text representations to quantify each modality’s presence.A decrease in diagram influence from Visual to V+T indicates reduced reliance on diagram information when text is available.
  • Layer 3: Attention allocation: Attention analysis sums last-token attention over modality-specific tokens across six decoder layers to examine how processing is allocated across inputs.Decreasing diagram and video attention alongside increasing text attention would confirm that text diverts processing from visual inputs.

5 Results and Analysis

Results show that cross-depiction alignment remains difficult, with architecture family mattering more than parameter count and video understanding forming the main bottleneck. Text improves instruction comprehension but can reduce diagram-to-video alignment, consistent with mechanistic evidence that models shift away from visual processing.

  • Benchmark performance: 45.6% average T1 accuracy among open-source models falls to 33.5% on T4, showing that next-step prediction is harder than current-step recognition.The average T1→T4 drop is 12.2 percentage points; proprietary models show similar declines.
  • Scaling and architecture: Architecture family predicts T1 alignment more reliably than parameter count, with Qwen3.5-9B reaching 57.8% versus 49.1% for Qwen2.5-VL-7B at similar scale.Upgrading model families can outperform substantially increasing parameters within one generation, although the causes cannot be isolated from architecture, data, and training procedure.
  • Scaling and architecture: Qwen3-VL-8B scores 53.1% on T1, exceeding the 48.8% of Qwen3-VL-30B-A3B despite the latter’s larger total parameter count.Gemma3 also shows nonmonotonic scaling, with its 12B model below its 4B model.
  • Diagnostic bottlenecks: D1 remains a hard video ceiling: 11 of 17 open-source models score below 53%, while the best proprietary and open-source scores reach only 71.1% and 63.7%.By contrast, D2 is substantially more recoverable through text, although its per-model comparisons use only N = 65 examples.
  • Alignment strategies: Text improves D2 instruction comprehension by +23.6 pp, but 12 of 17 models show negative T1 deltas under V+T.All 17 models improve on D2 with text, while most decline on T1, so comprehension gains do not transfer to alignment.
  • Mechanistic analysis: ViT-level CKA is 0.001–0.101, indicating disjoint diagram-video representations; adding text reduces diagram influence in 3 of 4 models and redirects attention from visual tokens.For Qwen3-VL-8B, diagram attention falls from 8.1% to 3.9% and video attention from 5.0% to 3.3% when text is added.

6 Discussion

The results support task-specific alignment strategies and identify visual encoding as the main target for improving cross-depiction assembly guidance, while generalization beyond IKEA furniture remains untested.

  • Visual input performs best for cross-depiction matching, while Text Only is preferable for assembly instruction comprehension.The authors recommend routing different subtasks to different alignment strategies.
  • Architecture family predicts alignment accuracy more reliably than parameter count, with newer dense architectures outperforming older families and larger MoE variants at similar scales.The comparison is presented as offering better cross-depiction alignment without additional compute cost.
  • Mechanistic analysis localizes the bottleneck to visual encoding: diagrams and video occupy disjoint representational subspaces, while added text distracts models from visual processing.The reported representational separation is CKA ≈0.
  • The benchmark covers IKEA furniture assembly only, so whether its findings generalize to other assembly domains is untested.It also evaluates foundational alignment rather than higher-level functions such as error detection.

7 Conclusion

IKEA-Bench evaluates VLM alignment with assembly diagrams and videos through a benchmark and mechanistic analysis, while its task prompts vary the available visual and textual inputs.

  • IKEA-Bench evaluates 19 VLMs across 1,623 questions using three alignment strategies and a three-layer mechanistic analysis.
  • The Visual baseline presents diagram images directly, whereas Visual+Text appends text descriptions and Text Only replaces diagram images with descriptions.
  • The benchmark includes video-to-diagram tasks for recognizing, verifying, tracking, and predicting assembly steps.These prompts require comparing video frames with instruction diagrams or selecting the relevant step from a sequence.
  • The next-step task asks models to identify the instruction diagram following the action shown in the video.
  • The dataset excludes products with too few steps or data-quality problems, and reports source-dataset statistics separately.

B.2 Text Description Generation and Quality

The benchmark pipeline generates structured diagram descriptions, validates them against source annotations, cleans ordering leakage, and deterministically constructs task questions from temporal alignments.

  • Each of 132 manual diagrams is independently described along eight dimensions, including parts, actions, tools, spatial orientation, results, warnings, fasteners, and arrow directions.
  • 952 characters is the average description length, with descriptions generated in six batches totaling approximately 296K tokens.
  • 127 of 132 descriptions, or 96.2%, are fully consistent with ground-truth parts and connection annotations.The five flagged cases are attributed to source-data issues.
  • 23 of 132 descriptions contained cross-step references, which were cleaned through regex replacement; a final audit found zero remaining references.
  • Video frames are uniformly sampled from temporal step segments, excluding the first and last 10% to avoid ambiguous transition frames.
  • T1 uses adjacent-step diagrams as distractors, while T2, T3, T4, D1, and D2 construct task-specific positive, negative, sequential, or shuffled options.D2 restricts ordering questions to consecutive triplets, and the benchmark reports question counts by product.

C Full Per-Model Results

The full results cover 19 models across six task types and three strategies for open-source systems, with response parsing generally successful across evaluation runs.

  • Tables 9–13 report results for all 19 models across three alignment strategies and six task types.The two proprietary Gemini models are evaluated under Visual only.
  • 94.1% is the overall parse rate across 51 open-source evaluation runs.Unparseable responses are counted as incorrect after multi-priority answer extraction.
  • Most models achieve 100% parse rates across all settings, while Gemini models achieve a 99.6% parse rate.Lower parse rates occur for Gemma3-12B, Gemma3-27B, and GLM-4.1V-9B.

C.2 Implementation Details

The evaluation uses controlled inference settings across open-source and proprietary VLMs, while Table 14 compares alignment strategies across representative models.

  • Evaluation setup: Open-source models use bfloat16 precision, greedy decoding, and a maximum of 1,024 new tokens on a single NVIDIA H200.Models are loaded with automatic device mapping.
  • Evaluation setup: Attention implementations vary by architecture, including Flash Attention 2 for Qwen2.5-VL, Qwen3-VL, and Gemma3, and SDPA for Qwen3.5.InternVL, MiniCPM, and GLM use trusted remote code.
  • Evaluation setup: Model weights are cached locally without quantization.
  • Evaluation setup: Gemini 3.1 Pro and 3 Flash are evaluated through the Google AI Studio API using temperature 0 and the same answer-extraction pipeline.The proprietary models use a maximum of 256 output tokens.
  • Strategy comparison: Table 14 compares T1 and D2 accuracy across three alignment strategies for five representative models spanning four families and three size tiers.The table includes reversed-preference cases such as InternVL3.5-8B.

C.4 Diagram Blindness Analysis

Visual assembly-instruction understanding consistently trails text-based reasoning across evaluated open-source models, independent of model scale as shown by marker size.

  • Visual versus text reasoning: Every one of the 17 open-source models falls below the y = x diagonal in D2 Visual versus Text Only accuracy.The comparison indicates lower visual than text-based assembly-instruction accuracy across architectures and scales.
  • Benchmark coverage: The benchmark statistics include per-product results with products sorted by category.
  • Visual versus text reasoning: Marker size in Figure 7 represents parameter count, enabling visual comparison of model scale against the visual–text accuracy relationship.

D Mechanistic Analysis Details

The mechanistic analysis extracts representations and probes retrieval, temporal discrimination, modality influence, and attention to characterize diagram–video alignment.

  • Feature extraction: ViT representations are forward-hook outputs from the final transformer block, mean-pooled across image patch tokens into one vector per image.Tile-based Qwen inputs are flattened before averaging, while InternVL3.5 removes each tile’s CLS token.
  • Feature extraction: Merger representations are extracted after the visual-to-LLM projection layer, with dimensions matching each model’s LLM hidden size.
  • Representation alignment: Linear CKA compares centered diagram and video representation matrices built from 113 valid step pairs, with bootstrap 95% confidence intervals from 1,000 resamples.CKA is dimension-invariant, enabling cross-architecture comparison.
  • Video temporal probe: The video temporal probe uses logistic regression on paired frame features, with same-step positives and different-step, same-product hard negatives split by product.The positive-to-negative ratio is 1:4, with deterministic seed 42.
  • Representation alignment: Cross-modal retrieval evaluates whether a matching video frame appears among the top K results for each diagram representation.Recall@K is the fraction of diagrams meeting this criterion.
  • Modality influence: For T1, final decoder hidden states are extracted under Visual and V+T conditions, then mean-pooled separately over video, diagram, and text tokens.The modality influence score is computed from these per-modality representations.
  • Attention analysis: Attention analysis probes six evenly spaced layers of Qwen3-VL-8B and aggregates last-token attention by modality across heads and layers.Grouped Query Attention expands 8 KV heads to 32 query heads, while float32 softmax supports numerical stability.
  • Attention analysis: Attention shares sum weights over modality-specific token positions after video spans, diagram spans, and remaining prompt tokens are categorized.The implementation uses O(seq) memory per layer rather than O(seq^2).
Loading 2604.00913v2…