PaperlayerView exact revision on arXiv

Source-linked AI summary

TrySee: Modular Perception-Enhanced VLM Grounding for Closed-Loop Robot Task Planning in Specialized Domains

Elias Hofele, Thomas Trapp, David Timmermann, Tristan Schnell, Georg Heppner, Arne Roennau

arXiv:2610.35687v1cs.ROcs.AI

TL;DR

VLMs can struggle to recognize specialized-domain objects and produce reliable geometric targets for robot execution. TrySee combines general-purpose VLM reasoning with modular perception in a closed-loop planning pipeline. Across five specialized tasks, the complete system outperforms the ungrounded baseline on planning, verification, and overall accuracy.

  • Problem

    VLMs may fail to recognize specialized-domain objects, and their image-level outputs may not provide reliable geometric targets for robot execution.

    Source: §I · p. 1

  • Method

    TrySee connects general-purpose VLM reasoning to domain-specific perception and robot execution, using short-horizon planning and verification with semantic labels, masks, and target poses.

    Source: A. Components · p. 4, §VI · p. 7

  • Results

    Across five specialized tasks, 81.8% Planning Accuracy, 91.2% Verification Accuracy, and 67.3% Overall Accuracy compare with 3.6%, 50.2%, and 0.0% for the ungrounded baseline, respectively.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The brownfield e-waste disassembly application demonstrates a practical route to specialized robot planning that reuses existing software and targeted perception without task-specific VLM fine-tuning.

    Source: §VI · pp. 7–8

Abstract

from arXiv · show

Vision Language Models (VLMs) have shown impressive capabilities in high-level reasoning and task planning for robots. However, specialized domains remain a challenge for VLMs' image interpretation and grounding abilities. We introduce TrySee, a modular closed-loop robot task planning pipeline integrating general-purpose VLM reasoning with domain-specific perception and established robotics components. The system consists of an inherently reactive cyclic planning pipeline, augmented with domain-specific modular vision preprocessing, to provide online grounding. Finally, a dynamic preprocessing selection mechanism contributes to the scalability of the pipeline. The system requires neither VLM fine-tuning nor end-to-end demonstrations and adapts through replaceable perception modules and motion planning. The approach is evaluated on planning tasks across 5 widely different domains, including a real-world robot disassembly task. The field tests in a brownfield environment demonstrate the approach's practicality in integrating VLM planning into specialized robotic systems with limited domain-specific training data.

I. INTRODUCTION

VLMs struggle to ground language in specialized-domain objects and executable robot targets. TrySee connects general-purpose VLM reasoning to replaceable perception modules and robot execution components, providing online grounding across varied tasks.

Motivation

  • Specialized-domain objects and features may be underrepresented in VLM training, while image-level outputs may not provide reliable geometric targets for robot execution.The paper identifies this grounding gap as a barrier to adopting VLMs in fields where data collection and annotation are costly.Source: §I · p. 1

Approach

  • TrySee links general-purpose VLM reasoning with replaceable domain-specific perception and established motion-planning components, without fine-tuning or end-to-end robot demonstrations.The modular interfaces support adaptation through targeted perception models rather than replacing the robot software stack.Source: §I · pp. 1–2
  • The VLM incrementally generates and verifies short-horizon action sequences, while perception provides semantic labels, visual masks, and executable target poses for online grounding.For object-dense scenes, the VLM can request preprocessing only for information relevant to the current verification step.Source: §I · p. 2

Evaluation scope

  • The pipeline is evaluated on complex, multi-step tasks across five domains, including a real-robot evaluation in an industrial recycling environment.The examples include e-waste recycling, workshop, and agricultural settings.Source: §I · p. 2

II. RELATED WORK

Prior work includes open-loop planners, reactive feedback systems, and explicit visual-grounding methods. TrySee integrates modular perception into the planning loop so grounding can be updated during execution and perception modules can be replaced across domains.

LLM- and VLM-based task planning

  • SAYCAN, PROGPROMPT [7], and SAYPLAN are described as open-loop planners, which can fail in dynamic environments.They respectively select library skills, express plans in Python syntax, and traverse a 3D scene graph.Source: §II · p. 2

Reactive task planning

  • Reactive approaches add execution feedback, replanning, or verification, but some rely on external grounding mechanisms or provide the LLM only textual prompts.Examples include INNERMONOLOGUE [8], ROSLLM, REFLECT, BETR-XP-LLM, and REPLAN.Source: §II · p. 2

Explicit visual grounding

  • Other work uses scene graphs, symbolic interfaces, segmentation, or marked image regions to connect visual information with language-conditioned planning and robot actions.Examples include ConceptGraphs, LISA, SoM [26], and MOKA.Source: §II · p. 2

TrySee’s approach

  • TrySee integrates modular perception directly into the planning loop, updating grounding during execution and adapting across domains by replacing individual perception modules.The approach operates at the task-planning level and reuses existing robotic software rather than relying on large-scale retraining.Source: §II · p. 2

III. TRYSEE

TrySee targets natural-language task planning in specialized domains, where objects or features may be unfamiliar to general-purpose VLMs. It presents the system as a cyclic pipeline whose operation is illustrated in Fig. 2.

Pipeline overview

  • Fig. 2 illustrates TrySee as a cyclic planning pipeline in which each iteration generates, executes, and verifies the next plan step.The Vision Preprocessing Module interacts with every other component; the figure images are illustrative, not experimental inputs.Source: Figure 2 · p. 3

Scope

  • A specialized domain comprises objects, features, or components absent from general-purpose datasets and therefore unlikely to be recognized by a VLM out of the box.The paper frames improving practical applicability of VLM-based planning as its aim.Source: §III · pp. 2–3

A. Components

TrySee alternates VLM planning and verification, supported by a modular vision interface that grounds language in objects and robot actions. Planning and verification use separate constrained prompts, while perception supplies labels, masks, and target poses.

A. Components

  • For planning, the VLM receives the task, available actions, current images, and interactable objects, then generates a rationale, action sequence, and goal for one major interaction.The overall plan is divided into short steps by the VLM itself.Source: A. Components · p. 3
  • The same VLM verifies execution by comparing its generated goal with current and pre-execution images, then records an observation for later planning steps.Past observations form the VLM’s memory and are appended to subsequent inputs.Source: A. Components · pp. 3–4
  • Separate planning and verification prompts give each VLM invocation a constrained objective and output format.The paper identifies this division as a design choice intended to keep the two complex tasks focused.Source: A. Components · p. 4
  • The VLM is treated as a black box without task-specific fine-tuning, separating domain adaptation from the reasoning-model choice.This allows newer VLMs to be integrated without modifying perception or execution interfaces.Source: A. Components · p. 4
  • The vision module finds task-relevant objects and labels them, masks specified objects in images, and generates target poses that enable robot interaction.Shared object labels let the VLM reason about objects and parameterize actions; masks ground spatial relations in the image.Source: A. Components · p. 4

B. Pipeline

TrySee turns VLM plans into robot actions through a perception and execution interface, then uses fresh images and VLM verification to update planning. Dynamic preprocessing selection lets the verifier request additional visual annotations as needed.

B. Pipeline

  • The VLM’s action sequence is parsed into a Behavior Tree whose leaves call robot-toolbox functions, with object labels mapped to poses using fresh images at action time.The Behavior Tree provides an inspectable interface between the generated plan and robot control.Source: B. Pipeline · p. 4
  • After execution, the VLM compares initial and new preprocessed images in verification mode, then appends its observation to the next planning iteration.The verifier can iteratively request additional preprocessed images, limiting simultaneous annotations in dense scenes.Source: B. Pipeline · p. 4
  • The evaluation’s example image sequences A–E correspond to the tasks summarized in Table II.Table II overviews the tasks, domains, and required object-recognition capabilities.Source: Figure 3 · p. 4, Table II · p. 5

IV. EXPERIMENTAL SETUP

The evaluation compares five grounding-interface configurations in a controlled image-sequence study and a real-robot case study. Five multi-step tasks span specialized domains, from computer assembly and crop handling to aircraft interaction and allocator disassembly.

IV. EXPERIMENTAL SETUP

  • Five configurations compare a baseline, domain descriptions, modular vision preprocessing with and without descriptions, and the full pipeline with dynamic preprocessing selection.The evaluations ask whether grounding improves planning and verification, whether preprocessing adds benefits beyond text, and whether VLM-directed preprocessing helps in dense scenes.Source: §IV · p. 5
  • The controlled image-sequence study isolates grounding-information effects, while the hardware case study demonstrates a complete implementation using learned perception, pose estimation, and an established execution stack.Both studies provide the VLM the same forms of visual and textual input.Source: §IV · p. 5

A. Evaluation Tasks

  • Task A requires installing specified PC components and removing any component already occupying a needed slot.Recognizing installed components is a core challenge.Source: A. Evaluation Tasks · p. 5
  • Task B requires recognizing crops, including fine-grained distinctions such as wheat versus barley, to carry out harvesting, watering, or fertilizing instructions.The tasks include both greenhouse plants and field crops.Source: A. Evaluation Tasks · p. 5
  • Task C asks the robot to accept an order and serve the correct customer, distinguished through facial recognition.Source: A. Evaluation Tasks · p. 5
  • Task D tests scalability by having the robot complete a simplified aircraft cockpit sequence involving many interactable objects.Source: A. Evaluation Tasks · p. 5
  • Task E is a real-robot disassembly task in which a milling spindle can remove built-in batteries from heat cost allocators.Destructive disassembly is allowed; the task explores real-world applicability.Source: A. Evaluation Tasks · p. 5
  • Each task is evaluated over 11 sequences with variations in initial setup or execution, including robot-action failures.Source: A. Evaluation Tasks · p. 5

B. Simulation

Tasks A–D use manually annotated perception inputs to isolate how the grounding interface affects VLM planning and verification. This controlled setup does not model perception-module recognition or segmentation errors.

B. Simulation

  • For tasks A–D, each time step’s images are manually annotated with the labels and segmentation masks a domain-specific perception system would produce.Source: B. Simulation · p. 5
  • Alternate inputs include failed actions and unexpected scene changes, allowing the evaluation to isolate grounding-interface effects on planning and verification.Source: B. Simulation · p. 5
  • The simulated perception setup does not model recognition or segmentation errors within perception modules.Source: B. Simulation · p. 5

C. Real-World Robot

The real-world evaluation implements TrySee on a UR10e robot with stereo cameras, learned object detection, segmentation, and pose estimation. The bulk of the evaluation uses LLaVA-OneVision, with a subset repeated using Gemma 3 to examine transfer across reasoning models.

C. Real-World Robot

  • The real-world setup combines a UR10e, custom-finger gripper, two stereo cameras, and a ROS 2/MoveIt 2 execution stack with Behavior Tree generation and execution.The cameras provide top-down and lateral views of the workspace; application-specific adaptation is concentrated in perception models and task-specific Behavior Tree nodes.Source: C. Real-World Robot · p. 5, Figure 4 · p. 6
  • YOLOv9 fine-tuned on 181 labeled instances detects objects and parts; Segment Anything 2 segments components using those detections as region proposals.Source: C. Real-World Robot · p. 6
  • For milling, the VLM selects a detected part from a masked close-up, and the vision module generates a trajectory for the robot to follow above the spindle.Source: C. Real-World Robot · p. 6

D. Choice of VLM

  • LLaVA-OneVision is used for most evaluation, while a subset is repeated with Gemma 3 to examine grounding-interface transfer and the effect of VLM choice.Source: D. Choice of VLM · p. 6

E. Evaluation Metrics

The section defines planning, verification, and end-to-end accuracy, then reports that TrySee improves grounding across tasks, with 67.3% Overall Accuracy and a shift in the primary failure mode to high-level reasoning.

E. Evaluation Metrics

  • Planning Accuracy measures correct generated sequences; Verification Accuracy measures correct decisions for correctly generated plans; Overall Accuracy requires both a correct sequence and all associated verification decisions.The ablation table covers five tasks and their combined average; D adds natural-language descriptions, V adds the Vision Preprocessing Module, and S adds dynamic preprocessing selection.Source: E. Evaluation Metrics · p. 6, §V · p. 6, Table III · p. 7

V. RESULTS

  • The Vision Preprocessing Module markedly increases Planning Accuracy across all tasks by improving object grounding in task-instruction semantics.Dynamic preprocessing selection affects verification only and does not affect Planning Accuracy.Source: §V · p. 6
  • The Vision Preprocessing Module greatly improves Verification Accuracy, while dynamic selection further improves it in object-dense environments without significantly increasing verification duration.Verification baseline performance is approximately 50.0% because individual verification decisions are binary; in most cases the VLM requested one image containing the goal-mentioned objects.Source: §V · p. 6
  • 67.3% Overall Accuracy is achieved by the complete pipeline, with the primary failure mode shifting from semantic grounding to high-level task reasoning.The reported planning errors involved instruction-following or logical correctness, rather than semantic object grounding.Source: §V · p. 6
  • For task C, vision-provided facial recognition changes performance from no successful baseline plans to perfect Planning Accuracy; language descriptions initially improve accuracy by 36.4%.More minute distinctions are difficult to convey using language alone.Source: §V · p. 6
  • Dynamic preprocessing selection increases task D's Verification Accuracy by 19.5 percentage points by showing masks only for objects relevant to the current goal.Task D involves many interactable cockpit objects.Source: §V · pp. 6–7

Cross-Model Evaluation

TrySee’s vision preprocessing and dynamic selection improve grounding-related planning and verification, with preliminary cross-model evidence that the gains are not specific to the primary VLM. The evaluation isolates downstream effects from perception-error sensitivity and identifies a structured application scope.

Cross-Model Evaluation

  • Gemma 3 results are generally consistent with LLaVA-OneVision, albeit slightly higher, in the Task A cross-model evaluation.Source: Cross-Model Evaluation · p. 7, Table IV · p. 7
  • Vision preprocessing substantially boosts planning and verification accuracy, while dynamic preprocessing selection further improves verification accuracy.Source: Cross-Model Evaluation · p. 7
  • Vision preprocessing shifts generalization requirements to dedicated vision models, which can use smaller datasets and be fine-tuned with a few examples, strengthening modularity and practical applicability.Source: Discussion · p. 7
  • The controlled image-sequence evaluation uses manually prepared perception outputs, so it does not measure sensitivity to perception errors; the robot case study demonstrates one learned-perception implementation.Source: Discussion · p. 7
  • Remaining failures concern instruction following and logical consistency; the stated scope is structured applications with available domain perception and robot skills, where environments change mainly between actions.Suggested avenues include action constraints, Behavior Tree validation, or structured memory for longer tasks.Source: Discussion · p. 7

VI. CONCLUSION

TrySee combines general-purpose VLM reasoning with modular domain-specific perception and established robot execution components. Across five specialized tasks it outperforms an ungrounded baseline, and a brownfield disassembly application demonstrates reuse of existing robotics software.

VI. CONCLUSION

  • TrySee connects general-purpose VLM reasoning with domain-specific perception and established robot execution, using short-horizon plans and dynamic preprocessing selection in object-dense scenes.Source: §VI · p. 7
  • 81.8% Planning Accuracy, 91.2% Verification Accuracy, and 67.3% Overall Accuracy compare with 3.6%, 50.2%, and 0.0% for the ungrounded baseline across five specialized tasks.Source: §VI · p. 7
  • The brownfield industrial e-waste disassembly application reuses ROS 2, MoveIt 2, Behavior Trees, and targeted perception models without task-specific VLM fine-tuning.Source: §VI · p. 7
  • The authors conclude that explicit, modular visual grounding offers a practical route to specialized robotic applications while retaining existing robot software and domain-specific capabilities.Source: §VI · pp. 7–8

Related papers

Loading 2610.35687v1…