PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language-Policy Model for Dynamic Robot Task Planning

Jin Wang, Kim Tien Ly, Jacques Cloete, Jin Jin, Nikos Tsagarakis, Ioannis Havoutis

arXiv:2512.19178v2cs.ROcs.AI

TL;DR

Traditional planning struggles to link high-level task reasoning with low-level execution and to update strategies when instructions change. This paper fine-tunes a vision-language model on real-world robot interactions to generate structured policies and adapt them during execution. Across multiple task scenarios, fine-tuned models show higher planning feasibility and execution success than the original model.

  • Problem

    Traditional task-planning approaches struggle to bridge high-level reasoning and low-level execution or dynamically update strategies when instructions change.

    Source: §I · pp. 1–2

  • Method

    The VLP framework fine-tunes a pretrained vision-language model on real-world robot interaction data and uses scene reasoning and behavioral primitives to generate structured policies that can be updated during execution.

    Source: §I · p. 2, B. System Framework · p. 3

  • Results

    Across three task categories, fine-tuned models exceeded 90% planning feasibility and achieved over 70% execution success, with over 20% higher success rates than the benchmark model.

    Source: 2) Multi-task scenario evaluation · pp. 6–7

  • Takeaways & Limitations

    Experiments across multiple platforms and scenarios support the framework's effectiveness and generalization for autonomous task planning and execution.

    Source: §V · p. 8

Abstract

from arXiv · show

Bridging the gap between natural language commands and autonomous execution in unstructured environments remains an open challenge for robotics. This requires robots to perceive and reason over the current task scene through multiple modalities, and to plan their behaviors to achieve their intended goals. Traditional robotic task-planning approaches often struggle to bridge low-level execution with high-level task reasoning, and cannot dynamically update task strategies when instructions change during execution, which ultimately limits their versatility and adaptability to new tasks. In this work, we propose a novel language model-based framework for dynamic robot task planning. Our Vision-Language-Policy (VLP) model, based on a vision-language model fine-tuned on real-world data, can interpret semantic instructions and integrate reasoning over the current task scene to generate behavior policies that control the robot to accomplish the task. Moreover, it can dynamically adjust the task strategy in response to changes in the task, enabling flexible adaptation to evolving task requirements. Experiments conducted with different robots and a variety of real-world tasks show that the trained model can efficiently adapt to novel scenarios and dynamically update its policy, demonstrating strong planning autonomy and cross-embodiment generalization. Videos: https://robovlp.github.io/

I. INTRODUCTION

Traditional planning systems struggle to combine semantic scene understanding with instruction-driven behavior in unstructured environments, while existing vision-language-action approaches face data, interpretability, and deployment challenges. The paper introduces a real-world-data-trained VLP framework for generating and updating robot policies, and reports validation across different mobile manipulators and diverse tasks.

Problem

  • Model-based planning struggles with semantic scene understanding and spatial reasoning needed for real-time, instruction-driven behavior in unstructured environments.Existing VLA approaches also face challenges from large-scale data requirements, the gap between high-level outputs and low-level control, interpretability, and local deployment.Source: §I · pp. 1–2

Framework

  • The VLP framework uses semantic scene reasoning and spatial understanding to generate task strategies that guide autonomous motion and manipulation.It is fine-tuned on real-world interaction data and coordinates predefined action and perception APIs to produce executable policies and refine them as requirements evolve.Source: §I · p. 2

Contributions

  • The work validates its approach across different mobile manipulators and diverse tasks, with the aim of demonstrating effectiveness and generality.The framework integrates a grounded multimodal language model with behavior primitives and affordance perception for real-time responses to changing task requirements.Source: §I · p. 2

II. RELATED WORK

Prior work spans model-based planning, learning-based control, language-model planners, and end-to-end vision-language-action systems, each with limitations for unstructured physical interaction. This paper instead uses interpretable structured policies for dynamic replanning and cross-embodiment task execution.

Traditional and learning-based planning

  • Traditional task planning and motion-planning methods address state transitions or feasible trajectories, while reinforcement and imitation learning acquire skills through trial and error or demonstrations.The paper identifies semantic understanding and dynamic adaptation as requirements for increasingly unstructured physical-interaction settings.Source: §II · p. 2

Foundation-model approaches

  • Language-model planners and VLA systems have limits including API-dependent inference, robot-specific configurations, opaque end-to-end policies, large data needs, computational cost, and safety concerns.These constraints can limit onboard real-time inference and generalization across embodiments.Source: §II · p. 2

Proposed approach

  • The proposed approach factors decisions into interpretable structured policies that support instruction-driven replanning and incremental updates when execution conditions change.Policies use object-centric affordances and a platform-agnostic primitive space to support multiple mobile manipulators and everyday tasks.Source: §II · p. 2

III. METHODOLOGY

The method uses a two-stage VLP pipeline: fine-tune a pretrained vision-language model on real-world interaction data, then deploy it with task memory and robot behavior primitives. During execution, state changes or new instructions trigger policy regeneration.

Two-stage pipeline

  • A pretrained VLM is fine-tuned on real-world robot interaction data with LoRA to produce a policy model for task reasoning, scene understanding, and policy generation.At deployment, the system combines semantic instructions, visual observations, robot state, behavior primitives, and task memory to generate a hierarchical policy.Source: A. Task Formulation · p. 3

Policy execution

  • The robot executes policy behaviors sequentially and updates perception and state feedback after each behavior.The execution loop terminates when the task is complete.Source: A. Task Formulation · p. 3

Policy updates

  • A new instruction or task-state change triggers updates to the instruction and memory, followed by regeneration of the policy.This loop is described as enabling real-time adaptation while connecting high-level semantic reasoning with low-level control.Source: A. Task Formulation · p. 3

B. System Framework

The system collects real-world demonstrations, fine-tunes a pretrained VLM, and deploys the resulting VLP model locally with predefined behavioral primitives. The model produces structured policies and can recompute them when instructions or task conditions change.

B. System Framework

  • Real-world demonstrations supply instructions, scene images, and action strategies for fine-tuning a pretrained VLM, which is then deployed locally for real-time inference.Predefined action and perception primitives can be invoked by the generated policy; execution monitoring and mid-task instructions support real-time replanning.Source: B. System Framework · p. 3, Figure 2 · p. 4

C. Vision-Language-Policy Model

  • The VLP model fine-tunes Qwen2.5-VL, using Qwen2.5 LLM as its backbone and a Vision Transformer as its visual encoder.Its LM decoder outputs hierarchical JSON-formatted policies for real-time robot behavior planning.Source: C. Vision-Language-Policy Model · p. 3

1) Model training:

  • The post-training dataset covers 15 environments and layouts, 12 mobile and manipulation task types, and varied targets and objects, with approximately 1,000 samples.Samples combine onboard-camera observations with manually annotated policies and were expanded through multi-view acquisition across embodiments and keyframes from demonstration videos.Source: 1) Model training · p. 3
  • The study compares Vision+Decoder LoRA with decoder-only LoRA, training for two epochs using a causal language-modeling objective applied only to JSON policy generation.Vision+Decoder LoRA adapts the last six visual-encoder layers and the decoder; decoder-only LoRA freezes the visual encoder.Source: 1) Model training · p. 4

2) Policy generation:

The VLP policy is a JSON-formatted sequence of action and perception behaviors that can be executed by the robot. Its hierarchical, sequential structure supports interpretable planning and updates when instructions or task requirements change.

2) Policy generation:

  • The VLP policy is a JSON-formatted sequence of action and perception behaviors that is converted into an executable Python script for robot control.Its top-down sequential execution scheme retains hierarchical representation and node-based logic inspired by behavior trees.Source: 2) Policy generation · p. 4
  • The policy structure propagates execution states and parameters through sequential behavior nodes.The structure is simplified from behavior-tree-style logic to top-down sequential execution.Source: 2) Policy generation · p. 4
  • The policy supports dynamic updates that adapt robot behavior to new instructions or task requirements during execution.Source: 2) Policy generation · p. 4

3) Motion primitives and visual perception:

The framework connects high-level plans to robot execution through reusable action and perception primitives. Visual grounding, affordance extraction, and depth images support object localization and grasp-point generation.

3) Motion primitives and visual perception:

  • Predefined action and perception primitives let the model decompose complex tasks into interpretable sequences of skills.The primitives are directly invoked by generated policies to connect high-level planning with low-level control.Source: 3) Motion primitives and visual… · p. 4
  • Fig. 4 contrasts LoRA applied to both the vision encoder and LM decoder with decoder-only LoRA, which freezes the vision encoder.Source: Figure 4 · p. 5
  • Actions such as grasp, lift, place, and handover are independent APIs that accept a target Cartesian pose.Preparatory poses such as homing and start pose support alignment and transitions between actions.Source: 3) Motion primitives and visual… · p. 5
  • Perception primitives combine visual object grounding with depth images to localize objects and generate grasp points.They also extract object-related and robot self-state information through onboard sensing modules.Source: 3) Motion primitives and visual… · p. 5

IV. EXPERIMENT AND EVALUATION

The evaluation uses real-world robot experiments in a laboratory setting with everyday household objects. The setup deploys two fine-tuning configurations on two distinct robotic platforms.

IV. EXPERIMENT AND EVALUATION

  • The study evaluates real-robot task completion after fine-tuning Qwen2.5-VL-3B-Instruct with decoder-only or vision-plus-decoder LoRA.Each configuration was trained for two epochs, then deployed locally; generated policies typically had subsecond latency.Source: §IV · p. 5, A. Experiment setup · p. 5

A. Experiment setup

  • The evaluation tests cross-embodiment capability on ANYmal D with a Unitree Z1 Arm and on the Toyota Human Support Robot.The platforms differ in configuration: a quadruped with a manipulator and a mobile robot with a single arm.Source: A. Experiment setup · p. 6
  • Experiments use laboratory scenes set up with everyday household objects.The real-world system deployment setup is illustrated in Fig. 5.Source: A. Experiment setup · p. 6

1) Comparison with other methods:

Across task scenarios, fine-tuned VLP models show higher planning feasibility and execution success than the original model, while dynamic policy generation exceeds the benchmark model. Comparisons also report advantages over existing methods in planning success, interpretability, and responsiveness.

1) Comparison with other methods:

  • VLP had higher planning success across task settings than other models, with visual input yielding significantly higher success when replanning after a new command than language-model-based planning.The comparison also reports better task decomposition and planning interpretability than VLA approaches, and greater adaptability and responsiveness than other language-model methods.Source: 1) Comparison with other methods · p. 6

2) Multi-task scenario evaluation:

  • Planning feasibility exceeded 90% and both fine-tuned models achieved execution success rates above 70% across three task categories.The categories were pick and place, object handover, and scene interaction; results are summarized in Table II.Source: 2) Multi-task scenario evaluation · p. 6, Table II · p. 6
  • The model achieved over 20% higher success rates than the benchmark model; fine-tuning both the vision encoder and LM decoder achieved 5–10% higher success rates than decoder-only fine-tuning.Source: 2) Multi-task scenario evaluation · p. 7

3) Cross-embodiment testing of the VLP model:

Across three tasks on two physical robot platforms, the VLP model achieved task success rates above 68% and showed cross-embodiment adaptability.

3) Cross-embodiment testing of the VLP model:

C. Results Analysis

The results analysis reports advantages in real-time replanning, feasibility, stability, and dynamic policy refinement. The conclusion summarizes cross-embodiment effectiveness while noting that predefined primitives constrain open-set planning flexibility.

C. Results Analysis

  • Post-training on real-world interaction data improves planning feasibility and execution stability relative to pretrained baselines in the evaluation.Jointly fine-tuning the visual encoder and LM decoder enhances scene reasoning and policy refinement, especially under dynamic conditions.Source: C. Results Analysis · p. 7
  • The results analysis reports advantages over existing task-planning methods, particularly for real-time replanning and adaptation to instruction and environment changes.Experiments across embodiments support adaptability and generalization; the paper describes the framework as promising for practical robotic deployment.Source: C. Results Analysis · p. 7

V. CONCLUSION

  • The framework combines semantic understanding, multimodal reasoning, and hierarchical policy generation for autonomous planning across robotic embodiments.The conclusion also reports interpretable planning, local inference, and real-time strategy adaptation in unstructured environments.Source: §V · p. 8
  • The predefined action and perception primitives may limit planning flexibility in open-set environments.Future work includes improving perception, adding autonomous navigation and long-horizon planning, and expanding the skill library.Source: §V · p. 8
Loading 2512.19178v2…