PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection

Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, He Wang

arXiv:2412.04455v3cs.ROcs.AIcs.CVcs.LG

TL;DR

Open-set robotic failure detection must address both unexpected failures after they occur and foreseeable failures before they occur, while existing approaches struggle to provide both modes with precise real-time monitoring. CaM uses VLM-generated code to evaluate spatio-temporal constraints over compact geometric elements. Under severe disturbances, it reports higher success rates and reduced execution time than baselines.

  • Problem

    Open-set robotic systems need to detect failures after they occur and prevent foreseeable failures, but existing VLM approaches face speed and coarse-grained accuracy challenges and proactive detection is less explored.

    Source: §1 · p. 2

  • Method

    CaM formulates reactive and proactive detection as spatio-temporal constraint satisfaction, using VLM-generated code to monitor compact geometric elements representing relevant entities or parts.

    Source: §1 · p. 2

  • Results

    28.7% higher success rates and 31.8% reduced execution time under severe disturbances are reported for CaM compared with baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CaM can be integrated with existing open-loop policies to form closed-loop systems for long-horizon tasks in cluttered scenes with environment dynamics and human disturbances.

    Source: §1 · p. 2

    Constraint elements primarily address failures detectable through explicit displacement and rotation, limiting their effectiveness for other failure types.

    Source: §A.3 · p. 14

Abstract

from arXiv · show

Automatic detection and prevention of open-set failures are crucial in closed-loop robotic systems. Recent studies often struggle to simultaneously identify unexpected failures reactively after they occur and prevent foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a novel paradigm leveraging the vision-language model (VLM) for both open-set reactive and proactive failure detection. The core of our method is to formulate both tasks as a unified set of spatio-temporal constraint satisfaction problems and use VLM-generated code to evaluate them for real-time monitoring. To enhance the accuracy and efficiency of monitoring, we further introduce constraint elements that abstract constraint-related entities or their parts into compact geometric elements. This approach offers greater generality, simplifies tracking, and facilitates constraint-aware visual programming by leveraging these elements as visual prompts. Experiments show that CaM achieves a 28.7% higher success rate and reduces execution time by 31.8% under severe disturbances compared to baselines across three simulators and a real-world setting. Moreover, CaM can be integrated with open-loop control policies to form closed-loop systems, enabling long-horizon tasks in cluttered scenes with dynamic environments.

1. Introduction

CaM addresses open-set robotic failure detection by unifying reactive and proactive monitoring through spatio-temporal constraints. The introduction reports higher success rates and reduced execution time under severe disturbances compared with baselines.

1. Introduction

  • Open-set failure detection must identify failures after they occur and prevent foreseeable failures, although the failures are not predefined.Reactive detection recognizes an occurred failure; proactive detection anticipates a potential cause before failure.Source: §1 · p. 2
  • Existing VLM-based reactive detectors face speed and coarse-grained accuracy challenges, while open-set proactive detection is less explored and requires precise real-time monitoring.The introduction attributes these challenges to computational costs and limited 3D spatio-temporal perception.Source: §1 · p. 2
  • CaM formulates both detection modes as spatio-temporal constraint satisfaction problems evaluated by VLM-translated executable programs.The programs verify whether entities or parts maintain or achieve required states during or after execution.Source: §1 · p. 2
  • Compact geometric constraint elements abstract relevant entities or parts, while segmentation and tracking models support their detection and monitoring.The monitor code is visually prompted by subgoal starting frames, constraint elements, and textual constraints, then runs on tracked elements without further VLM calls.Source: §1 · p. 2
  • 28.7% higher success rates and 31.8% reduced execution time under severe disturbances are reported for CaM compared with baselines.Experiments span three simulators and one real-world setting.Source: §1 · p. 2

2. Related work

Prior work addresses open-set reactive detection and visual prompting, but proactive detection and precise real-time constraint monitoring remain less developed. CaM positions constraint elements and code-based monitoring against these approaches.

2. Related work

  • VLM-based reactive detectors provide binary success indicators or textual explanations through VQA, but can have compromised speed and coarse-grained accuracy.Other LLM-based approaches convert visual inputs into text or rely on ground-truth feedback, which the paper describes as impractical in real-world scenarios.Source: §2 · pp. 2–3
  • Open-set proactive detection is rarely explored, and prior methods are confined to predefined failures.The paper frames proactive detection as forecasting failure causes and monitoring them precisely in real time.Source: §2 · p. 3
  • Unlike point-based constraint prompting such as ReKep [21], which may miss precise constraint keypoints, CaM introduces elements representing relevant entities or parts for real-time monitoring.CoPa [19] uses basic elements for robotic control; CaM applies elements to constraint satisfaction monitoring.Source: §2 · p. 3
  • Prior VLM visual-programming methods use raw RGB inputs and predefined primitives, whereas CaM uses constraint elements and code arithmetic for spatio-temporal dynamics.The paper notes that combining element associations with arithmetic operations presents greater challenges.Source: §2 · p. 3

3. Method

CaM cycles through constraint generation, visual annotation of compact elements, and code-based monitoring. Its method uses multi-view segmentation and tracking to evaluate constraints during execution and at subgoal completion.

3.1. Overview

  • The Constraint Generator uses observations, task instructions, prior subgoal, and monitor feedback to derive a next subgoal and textual constraints.Constraints may need to hold during execution or be met upon subgoal completion.Source: §3.1 · pp. 3–4
  • The Painter maps constraints to 3D-point-based elements and annotates them across views, while the Monitor evaluates generated code and returns a failure flag and reason for replanning.A False result halts policy execution; otherwise, the subgoal is considered completed and the cycle repeats.Source: §3.1 · p. 4

3.2. Constraint Element

  • Constraint elements reduce entities or parts to compact geometric forms, removing irrelevant visual detail to simplify tracking and support generalization.The pipeline generates instance- and part-level masks across views, projects them into 3D, selects points through voxelization and filtering, and annotates the resulting elements on images.Source: §3.2 · p. 4, Figure 3 · p. 4
  • ConSeg reasons over a constraint and image to produce an element type and part mask; its training data combines annotated BridgeData V2 trajectories with LISA data.The described BridgeData V2 collection contains 10,181 trajectories and 219,356 images.Source: §3.2 · p. 5

3.3. Real-time Monitor Module

  • GPT-4o generates monitor code from annotated visual prompts and textual constraints; the code checks tracked elements against spatio-temporal constraints and reports potential or actual failures.The monitor returns a boolean and explanation, and tracking with CoTracker avoids frequent VLM calls; an example tolerance allows up to 15° deviation of a pan’s surface normal from the z-axis.Source: §3.1 · p. 4, §3.3 · p. 5
  • Current positions and historical trajectories let the monitor check precise movements and rotations, while structured element associations and VLM prior knowledge support generalization to some unseen common-skill tasks.The paper gives 2 cm movement and 180° rotation as examples of monitored requirements.Source: §3.3 · p. 5

4. Experiments

Experiments evaluate CaM across three simulators and a real-world setting, using varied robots, tasks, and disturbances. They assess failure detection, constraint-aware segmentation, and design choices through task success and efficiency outcomes.

4.2.1 Results in CLIPort

  • 38.7% and 14.4% less execution time than Inner Monologue and DoReMi, respectively, in disturbed Stack in Order; CaM also achieves 4.5× DoReMi’s average success rate in Sweep Half the Blocks.For Stack in Order, CaM anticipates collapse and stabilizes a block; in Sweep Half the Blocks, it counts block points in the target region to improve halting precision.Source: Table 1 · p. 6, §4.2.1 · p. 6

4.2.2 Results in Omnigibson

  • 34.8% less execution time and 52.2% fewer tokens than DoReMi in Omnigibson; CaM alone detects the tested surface-level Pour Tea failures.Monitoring teapot tilt in real time avoids frequent VLM checks and supports timely replanning.Source: §4.2.2 · pp. 6–7, Table 2 · p. 7

4.3. Main Results in Real World

  • 20.4% higher success rates than DoReMi with a Leap Hand, while only CaM successfully handles the tested long-horizon tasks in cluttered scenes.The long-horizon tasks use an open-loop policy and include human disturbances.Source: Table 4 · p. 7, §4.3 · pp. 7–8, Table 3 · p. 7

4.4. Main Results of Segmentation

  • ConSeg performs comparably to LISA and PixelLM on ReasonSeg and achieves nearly 40% improvement at the part level on ConstraintSeg.Visual comparisons are reported to show generalization to unseen objects, scenes, and tasks.Source: Table 5 · p. 7, §4.4 · p. 8

4.2.3 Results in RLBench

  • RLBench experiments cover articulated objects, rotational manipulation, and tool use, and report generalization across these diverse manipulation tasks.The supplied results passage gives no task-level numerical comparisons.Source: §4.2.3 · p. 7

4.5. Ablation Study

  • 65% to 40% average success when using only front-view images with constraint elements, as occlusion and dimensional-reduction errors impair element generation and visual programming.Surfaces may be misinterpreted as lines, or lines as points.Source: Table 6 · p. 8, §4.5 · p. 8
  • 65% to 42.5% with DINOv2 semantic points and 65% to 55% with unconnected points, compared with constraint-aware segmentation and formed elements, respectively.DINOv2 points may fail to represent a constraint precisely; formed elements provide prior constraint information that standalone points require code to recombine.Source: Table 6 · p. 8, §4.5 · pp. 8–9
  • 60% to 20% success when entities or parts are not transformed into elements; instance-level and part-level training subsets both improve ConstraintSeg performance and reinforce each other.Without elements, inaccurate or slow pose tracking and arithmetic on 3D positions or poses hinder constraint encoding.Source: Table 6 · p. 8, Table 7 · p. 8, §4.5 · p. 9

5. Conclusion

Code-as-Monitor uses VLM-generated code to evaluate spatio-temporal constraints for reactive and proactive failure detection. Constraint elements abstract entities or parts into compact geometric forms to improve monitoring precision and efficiency.

5. Conclusion

  • CaM formulates reactive and proactive failure detection as spatio-temporal constraint satisfaction problems and uses VLM-generated code for real-time monitoring.Constraint elements abstract constraint-related entities or parts into compact geometric elements.Source: §5 · p. 9

Supplementary Material

The supplement collects methodology, implementation, and evaluation details. Its discussions explain the framework’s generalization, long-horizon planning approach, code reliability, and threshold-selection boundary.

Supplementary Material

  • The supplement covers technical details, limitations, future work, Painter implementation, environment configuration, evaluation, ablations, and demonstrations.It organizes these materials into Sections A–F.Source: Supplementary Material · p. 13

A.1. Methodology discussions

  • VLM reasoning supports task-level generalization, while geometric abstraction supports generalization across scenes and entities.The constraint-based formulation integrates these two aspects.Source: §A.1 · p. 13
  • For long-horizon tasks, the Constraint Generator decomposes tasks into subgoals, with proactive constraints shaping future states and failed subgoals replanned.The authors describe this as a potential way to handle long-term dynamics.Source: §A.1 · p. 13
  • Unlike ReKep’s DINOv2 keypoints, CaM combines ConSeg object/part extraction with point clouds and uses geometry-rich visual prompts for code generation.Code is generated at each subgoal’s beginning to support integration with different policies in closed-loop systems.Source: §A.1 · p. 13
  • ConSeg handles ambiguous or missing part instructions, but its performance on unseen objects is weaker than on objects within the training distribution.The dataset pipeline does not require textual instructions to specify parts.Source: §A.1 · p. 13

A.2. Technical detail discussions

  • Code is tested for path coverage at each subgoal’s beginning, and errors trigger regeneration; VLM prompting also targets specification adherence.The authors invoke the VLM once per subgoal because frequent calls increase hallucinations.Source: §A.2 · p. 13
  • Thresholds are initialized manually for common tasks or generated by VLM knowledge for unseen tasks, but variation across diverse conditions remains unexplored.The authors report robustness to threshold selection in common settings.Source: §A.2 · p. 13
  • Segmentation runs at subgoal beginnings, not during execution, so monitoring tracks elements without that model’s high computational cost.Segmentation models are run in parallel to accelerate this stage.Source: §A.2 · p. 13

A.3. Limitations and Future Work

The authors identify hallucination and representation limits in CaM, then describe a BridgeData V2-based process for assembling and annotating the ConSeg dataset.

A.3. Limitations and Future Work

  • VLM code-generation inaccuracies and segmentation biases may persist; geometric elements are less effective for force-only failures and may omit visual or multimodal cues.The paper gives a gripper failing to open a drawer and water flow or faucet sounds as examples of information the representation may miss.Source: §A.3 · p. 14

B. Constraint Painter

  • The Constraint Painter section covers dataset collection and annotation, ConSeg training, and the element pipeline.The following data-collection subsection details the first of these components.Source: B. Constraint Painter · p. 14

B.1. ConSeg Data Collection

  • BridgeData V2 trajectories are decomposed into subgoals and constraints, assigned frame-level labels using gripper-state information, then segmented with off-the-shelf models and manually filtered.The final dataset contains 10,181 trajectories and 219,356 images.Source: §B.1 · p. 14

B.2. ConSeg Training Details

ConSeg is trained on a mixture of segmentation, referring, VQA, reasoning, and constraint-aware data, with a simulator-finetuned variant and a separate point-cloud element pipeline. The environment configuration section emphasizes that CaM is policy agnostic.

B.2. ConSeg Training Details

  • ConSeg uses next-token prediction for text and per-pixel BCE plus DICE losses for masks, trained on multiple datasets including ConstraintSeg.The ConSeg-13B training uses eight 80G H800 GPUs for two days with batch size four.Source: §B.2 · p. 14
  • Simulation experiments use ConSeg-ft, finetuned on 100 trajectories per simulator, while real-world experiments use ConSeg-base, demonstrating generalization across scenarios.Simulator frames are sampled at 1 Hz and labeled with simulator ground-truth masks or annotations from Grounded SAM and Semantic SAM.Source: §B.2 · pp. 14–15

B.3. Element Pipeline Details

  • The element pipeline filters point-cloud outliers, sets voxel size by element type, then clusters points with DBSCAN, which supports arbitrary shapes without a preset cluster count.The passage also notes DBSCAN’s effectiveness in high-density regions.Source: §B.3 · p. 15

C. Environment Configuration

  • CaM is policy agnostic and can adapt to any control policy without requiring modifications.The section also introduces the simulator and real-world setups, low-level policies, and baselines.Source: C. Environment Configuration · p. 15

C.1. Environmental Setup

The study evaluates CaM across three simulators and a real-world setup using different robots, camera views, and low-level policies. It also describes environment-specific baselines and the scope of the evaluation details.

C.1. Environmental Setup

  • CLIPort uses a UR5 with front and top views, OmniGibson a Fetch with front and top views, RLBench a Franka with four views, and the real-world setup a UR5 with wrist and front views.The real-world robot uses a Leap Hand.Source: Figure 8 · p. 16, §C.1 · p. 16

C.2. Control Policy

  • CLIPort and DexGraspNet 2.0 are open-loop policies, whereas ReKep provides closed-loop control and ARP selects actions from historical observations and action sequences.An open-loop policy does not adjust actions in response to environmental changes during execution.Source: §C.2 · pp. 16–17

C.3. Baseline Details

  • Baselines pair CLIPort, ReKep, or DexGraspNet 2.0 with failure-detection or planning methods including Inner Monologue and DoReMi, depending on the environment.The CLIPort comparisons include CLIPort with Inner Monologue and with DoReMi; other environments compare their control policy with and without DoReMi.Source: §C.3 · p. 17

D. Evaluation Details

  • The evaluation-details section covers simulator and real-world task specifications, introduced disturbances, metrics, detailed results, and analyses.The text notes that some additional results are omitted from the main text for space constraints.Source: D. Evaluation Details · p. 17

D.1.1 Task, Disturbance and Metric Details

The CLIPort, OmniGibson, and RLBench evaluations cover varied manipulation tasks and disturbances, with results reported across several task-specific comparisons. In CLIPort, CaM improves success rates and reduces execution time against baselines under severe interference.

D.1.1 Task, Disturbance and Metric Details

  • CLIPort evaluates ordered block stacking, sweeping approximately half of 40 blocks, and using a rope to sufficiently enclose an open square; disturbances include suction release and placement noise.Success criteria include task-specific time limits and, for sweeping, 16–24 blocks in the target area.Source: §D.1.1 · pp. 17–18, Table 8 · p. 18

D.1.2 Detailed Experiment Results

  • 18.33% and 17.5% success-rate gains, plus 38.7% and 14.4% shorter execution times, over Inner Monologue and DoReMi, respectively, occur in severe-interference Stack in Order.The gains are reported in that baseline order.Source: §D.1.2 · p. 18
  • 4.1 and 4.5 times higher success rates than Inner Monologue and DoReMi, respectively, are reported for Sweep Half the Blocks.Success remains limited even without distractions; dense tracking points raise the likelihood of confusion and counting errors.Source: §D.1.2 · p. 18
  • 8.34% and 18.34% higher success rates, with 16.82% and 47.43% shorter execution times, respectively, are reported over Inner Monologue and DoReMi for Use Rope to Close the Opening Square.Measuring rope-end distances to the opening edges is reported as more accurate than querying a VLM image directly, allowing earlier policy termination.Source: §D.1.2 · p. 19

D.2.1 Task, Disturbance and Metric Details

  • OmniGibson covers pen insertion, vertical book stowing, and tea pouring under point-, line-, and surface-level disturbances; each task has one undisturbed and three disturbed trials, repeated 10 times.Evaluation includes success rate, execution time including VLM computation, and token count.Source: §D.2.1 · p. 19

D.2.2 Detailed Experiment Results

  • OmniGibson's detailed results are presented in the main text, while this section points to additional demonstrations in Sec. F.2.Source: §D.2.2 · p. 19

D.3.1 Task, Disturbance and Metric Details

  • RLBench tests six tasks in articulated-object interaction, rotational manipulation, and tool use, over 1,000 valid episodes per task of at most 25 steps.The policy naturally generates diverse failures, and flagged episodes are treated as invalid because the RLBench-trained policy has no recovery mechanism.Source: §D.3.1 · p. 19

D.3.2 Detailed Experiment Results

  • 98.1% success for Open Drawer and 98.3% for Put in Drawer are reported, versus DRM's 90.6% and 87.7%; CaM also outperforms baselines on rotational-manipulation and tool-use tasks.The authors attribute generalization to pre-trained VLM world knowledge and abstracting articulated objects into geometric components; the articulated objects were absent from training data.Source: §D.3.2 · p. 19

D.4.1 Task, Disturbance and Metric Details

The real-world evaluation includes object pick-and-place and long-horizon reasoning tasks under object-motion and removal disturbances. CaM's reported success-rate advantage over DoReMi is for Simple Pick & Place.

D.4.1 Task, Disturbance and Metric Details

  • Simple Pick & Place covers deformable, transparent, small rigid, and large geometric objects; reasoning tasks involve ambiguous categories and long-horizon instructions in cluttered scenes.Source: §D.4.1 · pp. 19–20
  • Both real-world task types include objects moving during grasping or being removed from the hand during movement; trials are evaluated by success rate and execution time including VLM computation.Each task, and each object in Simple Pick & Place, is tested in 10 trials; reasoning tasks have a four-minute or two-minute limit, depending on the task.Source: §D.4.1 · p. 20

D.4.2 Detailed Experiment Results

  • 20.4% higher success rates than DoReMi are reported for Simple Pick & Place across different object types.Source: §D.4.2 · p. 20

E. More ablation studies

The ablations examine the segmentation model and the combination of reactive and proactive failure detection.

E. More ablation studies

  • ConSeg yields superior overall framework performance to LISA and PixelLM in the reported segmentation-model ablations.Source: E. More ablation studies · p. 21
  • Proactive-only detection has lower success under unforeseen failures, reactive-only detection takes longer, and combining both modes achieves the best performance.Reactive-only detection responds after failure; the authors describe the combined modes as addressing the limitations of each.Source: E. More ablation studies · p. 21

F. More Demonstrations and Prompts

The appendix demonstrates failure detection and recovery across simulated and real-world tasks, shows segmentation examples, and provides the monitor-code prompt.

F. More Demonstrations and Prompts

F.1. CLIPort

F.2. OmniGibson

F.3. Real-world Evaluation

  • The real-world animal-clearing demonstration shows reactive detection after an object is removed from the robot's grasp and proactive detection of target movement during grasping.The passage states that this enhances task success rate and reduces execution time.Source: §F.3 · p. 22, Figure 19 · p. 30

F.4. Constraint-aware segmentation

F.5. Prompts

  • The monitor-code prompt combines task instructions, the current subgoal, and images from two perspectives to prompt GPT-4o to generate Python monitoring code.Source: §F.5 · p. 22, Figure 24 · p. 35

Related papers

Loading 2412.04455v3…