PaperlayerView exact revision on arXiv

Source-linked AI summary

CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates

Shresth Grover, Priyank Pathak, Akash Kumar, Yogesh S Rawat

arXiv:2512.10342v3cs.CV

TL;DR

Visual sequential planning remains underexplored relative to text-domain reasoning. CoSPlan evaluates multi-step visual planning with erroneous actions, and SGI iteratively refines textual scene graphs. SGI consistently outperforms vanilla Scene Graph across tasks, with a 4.4% average gain for Step Completion.

  • Problem

    Visual-language sequential planning is underexplored compared with text-domain sequence reasoning, despite VLMs' demonstrated text-domain planning capabilities.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces CoSPlan, which tests visual action sequences with intentional errors, and SGI, which iteratively refines textual scene graphs after each action.

    Source: §1 · pp. 2–3

  • Results

    SGI consistently outperforms vanilla Scene Graph across all tasks, with a 4.4% average gain for Step Completion.

    Source: §4.2 · p. 13

  • Takeaways & Limitations

    SGI is reported to boost performance across error-prone, error-free, and VQA settings, extending beyond corrective planning.

    Source: §5 · p. 15

    The benchmark isolates one erroneous step per sequence, a constraint the authors call somewhat artificial relative to real-world multi-error settings.

    Source: §6 · p. 15

Abstract

from arXiv · show

Vision Language Models (VLMs) have shown promising planning capabilities, yet their success remains confined to the text domain, leaving visual decision-making relatively underexplored. Addressing this gap, we introduce Corrective Sequence Planning (CoSPlan) benchmark, where VLMs must plan a sequence of visual actions from an initial scene to a target scene. CoSPlan evaluates models on their ability to imagine and execute a coherent set of visual steps required to reach the goal (Step Completion). To prevent any shortcuts that simply describe the final scene, we introduce an erroneous action in decision-making, which must be detected (Error Detection) and corrected to reach the goal, enabling a deeper understanding of the task. CoSPlan spans across 4 tasks: maze navigation, block re-arrangement, image reconstruction, and object re-organization. Despite using advanced reasoning strategies such as Chain-of-Thought and Scene Graphs, VLMs struggle on CoSPlan, while still showing promising performance in the text domain. Addressing this, we propose Scene Graph Incremental updates (SGI), a novel training-free method to transform images into `textual' scene graphs, enabling step-by-step reasoning through iterative scene graph refinement. SGI yields an average of ~4.4% improvement on CoSPlan w/ generalization on PlanBench and VQA. Link for solving puzzles on the project page.

1 Introduction

Visual sequential planning is underexplored compared with text-domain reasoning. CoSPlan tests multi-step visual action planning with errors, while SGI refines scene graphs step-by-step to support corrective reasoning.

1 Introduction

  • CoSPlan tests VLMs on multi-step visual planning from an initial scene and action sequence, requiring scene updates after each action to reach a goal.Source: §1 · p. 2
  • An intentional erroneous action requires models to detect and correct mistakes, preventing success by simply matching the final state.Source: §1 · p. 2
  • The benchmark covers maze navigation, block rearrangement, image reconstruction, and real-world reorganization, evaluating Error Detection and Step Completion.Source: §1 · p. 3
  • Vanilla VLM performance is near random, and CoT and Scene Graph strategies struggle because they compress initial-to-final reasoning into one step.Source: §1 · p. 3
  • SGI converts images into textual scene graphs and refines them step-by-step, explicitly generating intermediate states to track scenes and correct errors.Source: §1 · p. 3
  • The authors report SGI enhances robustness on CoSPlan and datasets including VQA and PlanBench.Source: §1 · p. 3

2 Related work

CoSPlan addresses the limited exploration of visual sequential planning by evaluating whether VLMs can detect and correct erroneous actions while completing visual plans. The section describes its benchmark design and finds that models struggle with error-prone visual reasoning despite stronger performance with structured strategies and on text-only tasks.

2 Related work

  • Prior sequential-planning benchmarks often assume ideal instructions and focus on text, while visual-puzzle benchmarks lack complex decision-making.CoSPlan is presented as a vision-language benchmark for temporal planning with error-prone instructions.Source: §2 · p. 4

3 CoSPlan Benchmark

  • CoSPlan asks models to move from an initial state to a goal by detecting an erroneous action and correcting it before completing the remaining steps.The setup spans reconstruction, rearrangement, reorganization, and maze-navigation scenarios.Source: §3 · pp. 4–5

3.2 Erroneous Step AE

  • Errors range from suboptimal moves to invalid actions, but Shuffle-E errors are excluded because one bad swap can cascade into later swaps.Maze-E and block errors can violate movement or stacking constraints; Robo-VQA-E includes suboptimal or invalid object interactions and placements.Source: §3 · p. 6, §3.2 · pp. 6–7

3.1 Benchmark Datasets

  • The four tasks are Maze-E navigation, Blocks-World-E block rearrangement, Shuffle-E image reconstruction, and Robo-VQA-E real-world object reorganization.Shuffle-E uses shuffled ImageNet image patches; Robo-VQA-E contains curated image-text pairs adapted from ROM.Source: §3.1 · p. 6

3.3 Safeguard against Shortcuts and Cheating

  • A distractor can reach the goal without correcting the error, so selecting the correct option requires correcting the action rather than matching the final scene.In Blocks-World-E, both the correct option and the distractor match the target configuration, but only one repairs the erroneous move.Source: §3.3 · p. 7

3.4 Evaluation

  • CoSPlan is evaluated with Top-1(%) accuracy in a multiple-choice framework: Step Completion selects one correct option from five (random accuracy 1/5), while Error Detection includes a “none of the above” option.Error Detection’s random-accuracy baseline depends on initial-context length.Source: §3.4 · p. 7

3.5 Models & Techniques

  • CoT prompts models to identify constraints, describe context actions step-by-step, and verify them; SG represents objects, attributes, and spatial relations as nodes and edges.The prompts are shared across models, while scene-graph structures may differ by model.Source: §3.5 · p. 8

3.6 Results & Benchmark Analysis

  • SG outperforms CoT, which in turn outperforms vanilla models in most cases; GPT-4o and GPT-5.1 make informed decisions while lighter open-source models perform near or below random.VLMs excel in error-free settings but struggle when errors are introduced; text-only descriptions significantly boost reasoning accuracy, while visual step generation can remain brittle across long horizons.Source: §3.6 · pp. 8–11

4 Scene Graph Incremental update (SGI)

SGI converts scenes into textual scene graphs and incrementally updates them as actions unfold, making intermediate states explicit. It consistently improves on vanilla scene graphs across CoSPlan tasks, with reported generalization to VQA, PlanBench, open-text error detection, and error-free planning.

4.1 Algorithm

  • SGI queries initial and goal scene graphs, simulates context and candidate actions through incremental graph updates, then selects the option whose resulting graph best matches the goal.The method compares model-specific graphs using the VLM as a judge because no universal graph-comparison metric applies across models.Source: §4 · p. 12, §4.1 · p. 12

4.2 Results

  • Oracle scene graphs outperform VLM-generated and noisy graphs, but remain below text-only reasoning; Qwen 3's robustness to noisy graphs is consistent with evaluating options independently.The paper identifies scene understanding as a challenge and notes that visual-step simulation remains difficult even with oracle graphs.Source: §4.2 · pp. 13–14, Table 6 · p. 14
  • SGI beats vanilla SG across tasks: Step Completion averages +4.4%, with gains up to 10.3%, 4.8%, and 10.0% for Intern-VLM 2, InternVLM-3, and GPT-4o.Error Detection gains average 4.1%, 1.7%, and 9.2%, respectively; on Blocks-World-E, Gemini-2.5-pro rises from 67% with CoT to 70% with SG and 71.5% with SGI.Source: §4.2 · p. 13, Table 5 · p. 14
  • On VQA, iteratively evaluating options outperforms single-shot SG or CoT; on PlanBench Blocksworld Task 8, Qwen2-VL-8B with textual SGI achieves the best performance.PlanBench requires generating a full action sequence validated by the planning environment, unlike the MCQ setup.Source: §4.2 · p. 14, Table 7 · p. 14, Table 8 · p. 15
  • SGI also outperforms baselines on open-text error detection, where models generate the erroneous step instead of selecting an MCQ option.A Qwen-2 8B model judges similarity between generated text and the ground-truth error.Source: §4.2 · p. 14, Table 9 · p. 15
  • SGI outperforms SG on both error-prone and error-free sequence-planning scenarios.The authors describe this as evidence of SGI's generalizability.Source: §4.2 · p. 14, Figure 9 · p. 15

5 Conclusion and Future Work

CoSPlan evaluates VLMs on error-prone sequential planning and identifies weaknesses in recovering plans from mistakes. SGI refines scene representations step by step and improves performance across error-prone, error-free, and VQA settings; the current setup uses static image pairs.

5 Conclusion and Future Work

  • SGI refines scene representations after each action and substantially boosts performance across error-prone, error-free, and VQA settings compared with vanilla scene graphs.The paper describes this as evidence of SGI’s robustness in corrective sequence planning.Source: §5 · p. 15
  • CoSPlan exposes near-random behavior after even one error, poor handling of in-context mistakes, and a bias toward text-based reasoning over grounded multimodal understanding.Even GPT-4o and GPT-5.1 fail to reliably use visual context to recover correct plans.Source: §5 · p. 15
  • The current setup uses static image pairs; video-based planning and interactive execution with updated-state observations are future directions.Source: §5 · p. 15

6 Limitations

The benchmark isolates one erroneous step per sequence for controlled evaluation, while noting that this constraint is artificial relative to real-world multi-error settings.

6 Limitations

  • Each sequence contains exactly one isolated erroneous step to facilitate controlled evaluation.Source: §6 · p. 15
  • The authors acknowledge that the single-error constraint is somewhat artificial.Source: §6 · p. 15
  • Real-world alignment requires evaluating models in more complex, multi-error settings.Source: §6 · p. 15

Overview

The paper outlines CoSPlan design choices, SGI, model evaluations, implementation details, and future generalization experiments.

Paper organization

  • Section 1 describes CoSPlan’s design choices, while Section 2 gives more in-depth details of the SGI algorithm.Source: Overview · p. 24
  • Section 3 covers human baseline scores and models, and Section 4 reports results and additional CoSPlan experiments.Source: Overview · p. 24
  • The paper also covers implementation and ethical responsibility, then presents experiments on SGI generalization.These topics are listed in Sections 5, 6, and 7, respectively.Source: Overview · p. 24

1 CoSPlan Design Choices

CoSPlan introduces errors into visual action sequences and separately examines whether models detect and recover from them. Its design also specifies completion choices, scene-graph structure, reasoning baselines, and benchmark annotation.

1.1 Error Correction Design

  • CoSPlan introduces an erroneous step so agents must recognize a suboptimal action and correct course; Error Detection and Step Completion are evaluated separately to distinguish failure types.The separate scores help prevent error-spotting ability from masking difficulty completing the task.Source: §1.1 · pp. 24–26

1.2 Sequence Completion Design

  • A correct completion option may start from the erroneous state but must provide a recovery sequence to the goal without further errors.Incorrect options may perpetuate the error or introduce new ones.Source: §1.2 · p. 26

1.3 One error Design

  • VLMs make near-random predictions with one error; multi-error evaluation is limited by exponential correction possibilities and current automation capabilities.The paper notes that the evaluated models cannot yet handle even one error.Source: §1.3 · p. 26

1.4 Scene Graph Design (SG)

  • Scene-graph prompts standardize object-node attributes and relation edges, reject invalid formats, and apply identical schemas and prompts across models.The ground-truth Oracle SG is common across models.Source: §1.4 · p. 26

1.5 Difference between SGI vs SG & CoT

  • The paper positions SGI between CoT and SG: CoT is stepwise, SG structures scenes, and SGI supports reasoning across evolving scenes.The stated hierarchy is CoT ⊆ SG ⊆ SGI.Source: §1.5 · p. 26

1.6 Chain of Though prompt Details

  • The VisualCoT baseline prompts models to produce intermediate reasoning from visual input before selecting a corrective answer.The chain is intended to assess the current state against the goal and identify the erroneous step.Source: §1.6 · p. 27

1.7 Annotation Generation

  • Synthetic Maze, Blockworld, and Shuffle sequences use DFS-generated optimal paths with systematically introduced errors; Robo-VQA-E uses manually annotated Robo-VQA samples.For synthetic tasks, correction reverses or rectifies the invalid step.Source: §1.7 · p. 27

2 Scene Graph Incremental update (SGI) Details

SGI incrementally simulates actions in scene graphs, then compares resulting graphs with the goal to select a continuation or locate an erroneous context action.

Error Detection

  • SGI structures error detection for incomplete plans or embedded errors as incremental, interpretable updates rather than a single transformation from initial to goal state.The error-detection adaptation is presented in Algorithm 2.Source: §2 · pp. 27–28

Incremental Simulation

  • SGI represents the initial and goal images as scene graphs, then incrementally simulates context actions to produce an intermediate graph.For error detection, it applies each candidate action to that intermediate graph to generate a hypothetical result.Source: §2 · pp. 27–28

Error Detection Similarity

  • For error detection, SGI compares each intermediate context graph with the goal and identifies the action with the greatest deviation.If the least similarity exceeds the 0.75 threshold, it returns “None of the above”; otherwise it returns the action associated with that minimum.Source: §2 · p. 28

Similarity Comparison

  • For step completion, SGI selects the candidate whose resulting scene graph is most similar to the goal graph.The VLM judges similarity by checking relationships, object placement, and unmet constraints; scene graphs are model-specific, so no universal metric is used.Source: §2 · pp. 28–29

3 Contenders for solving CoSPlan

The section introduces the VLM contenders, gives brief model descriptions, reports a human-baseline collection, and outlines experimental input and output handling.

3 Contenders for solving CoSPlan

  • The evaluated suite uses a range of state-of-the-art vision-language models to address visual reasoning tasks, including proprietary and open-source solutions with diverse architectural characteristics for multimodal understanding.Descriptions include LLaMA-3 [52], GPT-4o & GPT-5.1 [1], CoG-VLM [61], InternVL2-26B, and Qwen2-VL [60].Source: §3 · p. 29, §3.1 · p. 29, §3.2 · p. 29

3.1 LLaMA-3

  • LLaMA-3 [52] is described as a general-purpose VLM combining textual and visual inputs in one architecture.GPT-4o and GPT-5.1 [1] are described as general-purpose models integrating text, image, and audio processing.Source: §3.1 · p. 29, §3.2 · p. 29

3.3 CoG-VLM

  • CoG-VLM [61] uses an EVA2-CLIP-E vision encoder and Vicuna1.5-7B language model, while InternVL2-26B combines InternViT-300M-448px with internlm2_5-7b-chat.The CoG-VLM vision backbone removes its final aggregation layer to preserve spatial information.Source: §3.3 · p. 29, §3.4 · p. 29

3.5 Qwen2-VL

  • Qwen2-VL [60] is represented by a 7B variant based on Qwen2-7B, and Janus uses SigLIP for vision understanding with a transformer language model.The Janus description identifies Janus-Pro as its multimodal-reasoning variant.Source: §3.5 · p. 29, §3.6 · p. 30

3.7 Human baseline

  • The human-baseline collection comprised 58 submissions, with each contestant solving a subset of puzzles and statistics reported in Table 1.“Unique” in Table 1 counts unique puzzles solved despite overlapping repeated submissions.Source: §3.7 · p. 30, Table 1 · p. 30

3.8 Experimental Setup

  • Experimental setup converts images and prompts to model-specific formats, concatenates initial and target images, and parses model responses into structured formats.Queries are structured as environment information followed by the task.Source: §3.8 · p. 30

4 Results

Results cover external planning and spatial-reasoning evaluations, CoSPlan comparisons and behavioral patterns, and performance as errors accumulate.

4.2 Experiments

  • GPT-4o achieves the highest performance across all CoSPlan tasks, while InternVLM and GPT-4o outperform prior baselines on Step Completion and Error Detection.Table 2 reports performance across the benchmark; GPT-5.1’s error-detection results are omitted because of budget constraints.Source: Table 2 · p. 31, §4.2 · p. 32

4.1 External Dataset Description

  • On text-only PlanBench, performance is almost perfect, especially for GPT-4o, while the evaluation tests whether generated plans are executable and satisfy goal conditions.The external VQA datasets described include Spatial-Map, Maze-Nav, and Spatial-Grid tasks without initial-context sequence planning.Source: §4.1 · pp. 30–32

4.2 Experiments

  • Maze-E accuracy remains roughly constant as obstacle count increases, and the authors suggest models may not understand or attend to the obstacles.The figure caption reports stable accuracy across VLMs as obstacles increase.Source: §4.2 · p. 32, Figure 3 · p. 32
  • Janus selects option A in 100% of Blocks-World-E predictions, while Qwen2 also shows a strong option-A bias regardless of option content.Because the correct answer is uniformly distributed among five options, this bias partially explains near-random accuracy.Source: §4.2 · p. 33, Figure 4 · p. 33
  • Intern-VLM sometimes selects the option matching the final state without correcting the error, and SGI cheats less often than simple Scene Graph prompting.The authors describe the latter comparison as an improvement in model prediction.Source: §4.2 · p. 33
  • Models maintain stable accuracy as additional context grows in Shuffle-E and Maze-E, while increasing the number of multiple-choice options lowers accuracy.The context-length setup keeps two context actions and requires nine steps to reach the goal, including one error correction.Source: §4.2 · p. 34, Figure 7 · p. 34

4.3 Multi Error Analysis

  • In the fixed-correction-order Maze-E setting, performance monotonically degrades as the number of errors increases.The analysis treats multiple errors as introducing exponential complexity and uses a fixed correction order to keep the setup tractable.Source: §4.3 · pp. 34–35, Table 3 · p. 35

5 Implementation Details

The implementation specifies the error-detection similarity threshold, hardware and batch size, and planned code release.

5 Implementation Details

  • Error detection uses a similarity threshold of 0.75, batch size 1, and one 48GB NVIDIA RTX A6000 GPU.The authors also state that they will release the codebase for task generation, evaluation, and SGI.Source: §5 · p. 35

6 Ethical Statement

CoSPlan is intended for research rather than safety-critical deployment. Future work targets broader reasoning settings and evaluation beyond the current multiple-choice setup.

6 Ethical Statement

  • CoSPlan is not intended for safety-critical applications; its dataset and code are for research use, and evaluated models may inherit pretraining biases.The benchmark includes synthetic and real-world task settings, with real-world images synthetic or sourced under permissible licenses without identifiable people or private information.Source: §6 · p. 35

7 Future work

  • SGI’s step/action simulation is not fundamentally dependent on scene graphs, and future work targets skipping states and extending it to video and other reasoning algorithms.Source: §7 · p. 35
  • Current analysis uses single-choice MCQ; despite this simplification, VLMs perform near randomly, while open-ended generation remains future work.Source: §7 · p. 35
Loading 2512.10342v3…