Source-linked AI summary
Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
Aditya Chetan, Eric Cai, Peeyush Kushwaha, Bharath Raj Nagoor Kani, Utkarsh Mall, Qianqian Wang, Noah Snavely, Bharath Hariharan
TL;DR
Existing video benchmarks provide limited evidence about the fine-grained spatio-temporal reasoning required for complex, step-by-step activities. This paper introduces FLAT-PACK BENCH, a furniture-assembly benchmark using annotated videos, visual prompts, and multiple-choice tasks, and finds that current LVLMs remain far below human performance.
Problem
Existing benchmarks insufficiently evaluate the fine-grained spatio-temporal reasoning needed for complex activities such as furniture assembly.
Method
FLAT-PACK BENCH augments furniture-assembly data with part annotations, visual prompts, and curated multiple-choice tasks covering temporal order, localization, tracking, and mating.
Results
Current LVLMs struggle substantially: GPT-5 reaches ∼38% accuracy versus 94.18% for humans, and proprietary models remain only modestly above the 26.74% chance baseline.
Takeaways & Limitations
The benchmark identifies fine-grained spatio-temporal reasoning, tracking, contact detection, and region-specific grounding as persistent bottlenecks for current LVLMs and vision tools.
Abstract
from arXiv · showhide
The emergence of Large Vision-Language Models (LVLMs) has significantly advanced video understanding capabilities. However, existing benchmarks focus predominantly on coarse-grained tasks such as action segmentation, classification, captioning, and retrieval. Furthermore, these benchmarks often rely on entities that can be easily identified verbally, like household objects, animals, human subjects, etc., limiting their applicability to complex, in-the-wild video scenarios. But, many applications such as furniture assembly, cooking, etc., require step-by-step fine-grained spatio-temporal understanding of the video, which is not sufficiently evaluated in current benchmarks. To address this gap, we introduce Flat-Pack Bench, a novel benchmark centered on furniture assembly tasks. Our benchmark evaluates LVLMs on nuanced tasks, including temporal ordering of assembly actions, temporal localization of assembly state, understanding part mating, and tracking, using multiple-choice questions paired with visual prompts highlighting relevant parts as references for fine-grained questions. Our experiments reveal that state-of-the-art LVLMs struggle significantly with fine-grained spatio-temporal reasoning, highlighting their limitations in effectively leveraging temporal information from videos, limited tracking ability, and understanding of spatial interactions like physical contact.
1. Introduction
Current LVLM benchmarks do not adequately test the fine-grained spatio-temporal reasoning needed for practical, step-by-step activities. FLAT-PACK BENCH addresses this gap with furniture-assembly questions targeting temporal order, localization, tracking, and part mating, where leading models perform poorly.
- Motivation: Practical assistants must identify individual assembly steps, determine how and when to execute them, and communicate that understanding.Overall video recognition is insufficient for tasks such as cooking, repairing equipment, or assembling furniture.
- Gap: Existing video benchmarks often emphasize short or coarse-grained videos, uncluttered scenes, and limited object interactions.These settings can avoid the temporal understanding, tracking, and interaction reasoning required by complex everyday activities.
- Benchmark rationale: Furniture assembly provides a controlled sandbox for testing multi-step object tracking and interactions because rigid parts retain their shape and identity.The benchmark uses this simplified domain to articulate skills needed for more complex settings involving changing object states and interactions.
- Contribution: FLAT-PACK BENCH adds part segmentations, individual-part connections, and curated multiple-choice questions to evaluate tracking, mating, and connection order.The benchmark tests multiple axes of temporal understanding through fine-grained annotations and natural-language questions.
- Findings: ∼38% accuracy for GPT-5 trails 94.18% human performance, while models struggle with tracking and contact detection.An agentic approach using tools such as SAM2 also finds that the tools struggle on these challenging videos.
2. Related Work
Related benchmarks cover coarse video semantics, physical understanding, spatial intelligence, temporal sensitivity, and assembly reasoning, but often simplify scenes or omit dynamic object interactions. FLAT-PACK BENCH instead targets fine-grained temporal evolution and complex interactions among multiple similar parts in real-world videos.
- Video understanding benchmarks: Prior VidQA benchmarks generally focus on coarse-grained questions about high-level scene semantics.Other physical-scene benchmarks often use synthetic videos without clutter or occlusion.
- Dynamic-scene reasoning: Some spatial-intelligence benchmarks use real-world videos but mainly study static scenes with camera motion and ignore dynamic objects.Related work also evaluates temporal sensitivity and single-frame bias, whereas this benchmark focuses on fine-grained temporal evolution.
- Assembly reasoning: LEGO-Puzzles simplifies multi-step assembly by providing only 2–3 relevant images, unlike long demonstrations requiring models to select salient frames.FLAT-PACK BENCH is designed to represent real-world demonstrations with many video frames.
- Regional understanding: Existing regional-understanding datasets typically contain one or two easily tracked objects, whereas FLAT-PACK BENCH involves multiple visually similar parts and complex interactions.Its manually curated and verified segmentations make tracking substantially more difficult.
3. FLAT-PACK BENCH
FLAT-PACK BENCH builds a furniture-assembly video question-answering benchmark from in-the-wild demonstrations, combining fine-grained annotations, visual prompts, and curated multiple-choice questions. It evaluates whether models can track parts, reason about mating, and infer when connection events occur.
- Source data: IMaW provides furniture models, part models, key-frame segmentations, 6DoF poses, and sub-assembly annotations for in-the-wild IKEA assembly videos.The benchmark builds on these annotations while addressing their limited granularity.
- Visual prompts: Visual prompts mark each referenced furniture part with a label and segmentation to avoid ambiguity in symmetric structures.Each question uses an assembly video and one or two prompts constructed from frames in the same video.
- Visual prompts: The benchmark adds manual segmentations on 343 frames from 50 videos because the original IMaW segmentation annotations were incomplete.These frames serve as visual prompts for the questions.
- Question types: MATE, TRACK, TORD, and TLOC test final connectivity, part correspondence, connection order, and near-state temporal localization.The task design reflects the need to identify which parts connect, when they connect, and how parts move through the video.
- Question construction: Fine-grained part assembly annotations specify which parts connect, to which other parts, and when those connections occur.These annotations extend IMaW's sub-assembly-level information to support question construction.
- Question curation: Questions were manually curated because automatic generation often allowed video-ignoring shortcuts through obvious options or already-positioned parts.Curation was used to reduce questions answerable without the intended video reasoning.
- Benchmark construction: The benchmark contains 602 multiple-choice questions covering 50 furniture-assembly videos sourced from IMaW.The dataset composition reports videos, questions, templates, and their averages across categories.
- Prompt construction: Each prompt combines a video, one or two visual prompts, a question, and fixed task instructions, with mixed-media, collage, and concatenated prompt settings.The three settings vary how visual prompts are supplied alongside the assembly video.
4. Evaluation on FLAT-PACK BENCH
FLAT-PACK BENCH evaluates proprietary, open, and specialized LVLMs on fine-grained assembly video questions using varied video and visual-prompt configurations. Models remain far below human performance, with substantial variation across model families and evidence that training-task alignment matters.
- Evaluation Setup: The benchmark evaluates proprietary, open, and specialized LVLMs across video-question-answering settings involving visual prompts and different video-processing configurations.The setup reports multiple model families, prompt formats, video types, chance baselines, human performance, and accuracy-based evaluation.
- Human Performance: Humans achieve over 90% accuracy on every question category, and 80% of questions receive unanimous responses.Participants answered using the assembly video, visual prompt, multiple-choice question, and task instruction.
- Proprietary Models: GPT-5 reaches 37.71% accuracy and Gemini 2.5/3.1 Pro reaches 33.72/32.89%, only modestly above the 26.74% chance baseline and far below humans.Selecting question-relevant frames with GenS did not improve Gemini 2.5 Pro’s performance.
- Open Models: InternVL3 performs best among open models at 41.03% accuracy, while performance across open models is highly variable and sometimes only slightly above chance.The reported 95% bootstrap confidence interval for InternVL3 is [36.21, 45.64].
- Specialized Models: Specialized models generally perform poorly on the benchmark, but PerceptionLM-8B is competitive with much larger models and temporal sensitivity helps ArrowRL outperform its base checkpoint, especially on TORD.The authors associate poor specialized-model performance with a mismatch between training data and FLAT-PACK BENCH’s complex, multi-part interactions.
- Overall Results: Across proprietary, open, and specialized systems, current LVLMs remain far from the strong and reliable spatio-temporal understanding demonstrated by humans.This conclusion summarizes the benchmark-wide model comparison.
5. Analysis
The analysis tests prompting, visual-prompt rendering, temporal-context use, and agentic decomposition to identify why LVLMs struggle. Results implicate ineffective video use, object grounding, tracking, and contact reasoning as persistent bottlenecks.
- Linguistic Prompt Engineering: Zero-shot and self-consistency chain-of-thought prompting both reduce performance, indicating that linguistic prompting does not improve the benchmark’s spatio-temporal visual understanding.The comparison uses key-frame videos and mixed-media prompts against vanilla prompts.
- Visual Data Ablation: Video type has little effect, while mixed-media visual prompts perform best among the tested prompt formats.The authors attribute the mixed-media advantage potentially to Qwen2.5-VL’s mixed image-and-video training data.
- Image-only Prompt: Image-based and commonsense shortcuts improve MATE and TLOC more than TORD, which requires longer temporal context.The analysis links these improvements to recognizing part types, positioning, and hand proximity in prompt images.
- Part ID Bias: Shuffling part IDs causes TORD performance to decline, supporting the authors’ hypothesis that integer-valued part-ID bias affected the unshuffled results.The authors tested three random shuffling runs and reported average performance.
- Overall Analysis: Overall, the results suggest that LVLMs are not using the videos effectively, including for simpler contact and tracking subproblems.The conclusion is supported by the image-only analysis, rationale-based error categories, and tool evaluations.
- Self-probing Explanations: A Gemini self-explanation correctly localizes the final connection and orders two parts, but mistracks the first connected part and produces a wrong answer.The example illustrates a failure to constrain later reasoning to the correctly localized timestamp.
- Error Analysis: Object grounding accounts for 37.28% of analyzed errors and spatio-temporal reasoning for 32.45%, making them the two major error categories.The error analysis selected 200 incorrectly answered questions sampled equally across the four task categories.
- Agentic Decomposition: The agentic TVA approach answers 11.48% of questions missed by the LVLM, but inaccurate tools remain its primary error source.SAM2 achieves a 0.28 average IoU when tracking annotated parts across frames.
6. Conclusion
FLAT-PACK BENCH exposes bottlenecks in fine-grained video understanding, especially spatio-temporal reasoning and region-specific grounding. The authors identify task-specific training, improved regional prompting, and richer agentic pipelines as future directions.
- Conclusion: FLAT-PACK BENCH evaluates LVLM spatio-temporal understanding and reveals bottlenecks in tracking parts through occlusions or scene cuts and in region-specific grounding.The benchmark also exposes limitations in contact reasoning and in specialized tracking tools.
- Conclusion: Agentic decomposition does not resolve these limitations because current vision tools also struggle with tracking and LVLMs fail on simpler contact-reasoning subproblems.The authors propose synthetic-data fine-tuning, improved visual prompting, and pipelines using signals such as 3D geometry and depth as future directions.
A.1. Additional Benchmark Statistics
The benchmark combines diverse furniture-assembly videos with curated questions and annotations designed to preserve temporal and spatial reasoning demands.
- The dataset covers 24 furniture items with 3–19 parts each, averaging 7 parts per item.
- Keyframe videos are sampled at 1 FPS, while trimmed videos retain variable frame rates averaging 28.98 FPS.
- Existing IMaW segmentations cover only parts being connected in selected keyframes, limiting both temporal coverage and part-specific questions.
- Auto-generated questions can be answered from visual prompts and commonsense reasoning without temporal video cues.
- Manual question templates require challenging options, including similar-looking parts and parts connected much later than the prompted state.
B. Evaluation Details
Evaluation uses resized assembly videos, task-specific multiple-choice templates, visual-prompt variants, and model-specific frame-processing procedures.
- Questions cover mating, temporal localization, and temporal ordering of physically connected furniture parts.
- Videos are resized to 480 × 640, and the standard pipeline initially exposes frames at their stored frame rate.
- Open models generally receive 32 subsampled frames, while Video-LLaVA receives 8 frames under its evaluation setting.
- Concat prompts retain the first two frames during subsampling so the visual prompts remain available.
- Mixed-Media, Collage, and Concat settings place the labeled and jumbled prompts in different positions within the visual input.
- Specialized models use customized procedures, including 32-frame PerceptionLM inputs, direct segmentation inputs for VideoRefer, and chunked filtering for GenS.
- Human evaluation includes standard and image-only tasks, with a broader Prolific study collecting at least three responses per question for majority voting.
C. Additional Results
Additional evaluations show strong human understanding of the benchmark, large image-only degradation, weak model–difficulty correlations, and substantial cognitive demands for human participants.
- 80% of human responses were unanimous overall; agreement was highest for TORD at 88% and lowest for TLOC at 70%.
- Image-only human agreement was 25%, with task-wise rates of 17% for TORD, 36% for TLOC, 27% for TRACK, and 20% for MATE.
- Prolific participants performed significantly better than reference models despite low agreement, indicating that the questions are solvable but cognitively demanding.
- For Qwen2.5-VL 72B, human-perceived video difficulty and duration do not significantly predict model performance.The reported correlations were Spearman’s ρ=-0.20, p=0.17 for difficulty and ρ=-0.21, p=0.13 for duration.
D.2. Experimental Details for TVA
The supplementary procedures specify prompt formats, contact definitions, tracking propagation, and reasoning-prompt settings used in the benchmark’s experiments.
- Contact-reasoning questions define “connected” as direct physical contact matching the fully assembled furniture, not proximity or partial alignment.
- Collage prompts place a fixed labeled frame beside each assembly-video frame, while Concat prompts place the labeled frame first in the video.
- The appendix reports contact-reasoning results for Qwen2.5-VL 32B and 72B across two question templates, with poor performance across settings.
- SAM2 tracking uses 256-frame clips with one-frame overlaps and propagates masks forward and backward from the clip containing the prompt frame.
- Image-only human evaluation provides instructions for answering from visual prompts and questions without the assembly video.
- The reasoning-prompt experiments replaced generic step-by-step requests with explicit reasoning instructions after models merely paraphrased their selected options.
E. Analysis on InternVL3
InternVL3-78B broadly replicates Qwen2.5-VL-72B’s findings: linguistic prompting does not help, visual-prompt choices matter, and image-only inputs degrade performance across tasks. These results suggest the benchmark’s conclusions are not specific to one model, while persistent TRACK declines indicate limited use of video context.
- Linguistic Prompt Engineering: ZS-CoT slightly improves TRACK but not overall performance, whereas SC-CoT significantly decreases accuracy across all tasks.Both strategies therefore fail to improve FLAT-PACK BENCH performance for InternVL3-78B.
- Visual Prompt Ablation: Concat prompts perform significantly better than mixed-media prompts for InternVL3-78B, while video type does not significantly affect performance.The authors relate the prompt-type effect to InternVL3’s training on image-text and video-text sequences rather than all three modalities together.
- Visual Prompt Ablation: With concat prompts and key-frame videos, color scheme, mark type, and marker size show trends similar to those observed for Qwen2.5-VL-72B.InternVL3-78B performs better with concat prompts and key-frame videos than with the alternatives considered for this model.
- Image-only Results: Image-only prompts cause clear performance degradation across all tasks, indicating that InternVL3-78B uses video context more effectively than Qwen2.5-VL-72B.This aligns with InternVL3-78B outperforming Qwen2.5-VL-72B on the main benchmark.
- Image-only Results: Despite stronger use of video context than Qwen2.5-VL-72B, most of the performance decline still stems from TRACK, showing that InternVL3-78B does not use video context very effectively.The comparison is based on the image-only ablation and the persistent TRACK trend.
- Replication of Qwen2.5-VL-72B Findings: InternVL3-78B displays largely similar trends to Qwen2.5-VL-72B across the analyzed questions, suggesting the findings extend beyond one model family or architecture.The experiments replicate the analyses for the benchmark’s best model, InternVL3-78B.