Source-linked AI summary
STEP: State-Aware Task Estimation and Planning with Multi-Modal LLMs for Human-Robot Collaboration
Maitrey Gramopadhye, Prakash Baskaran, Xiao Liu, Songpo Li, Soshi Iba
TL;DR
MM-LLMs lack explicit system-state tracking, and natural-language plans can leave action execution ambiguous. STEP estimates structured states, predicts state transitions, and supplies execution parameters; it improves executability and final error over the existing state-of-the-art in collaborative industrial assembly tasks.
Problem
MM-LLMs do not explicitly track system states, while cursory natural-language action plans can leave execution parameters ambiguous.
Method
STEP uses a multi-stage pipeline to estimate the task and structured workstation state, predict actions and state transitions, and derive assistance parameters for execution.
Results
STEP improved executability and final error over the existing state-of-the-art on a teleoperated-robot assembly dataset.
Takeaways & Limitations
Correct task and current-state estimation improve predicted-action performance, while state-distance tracking guides rollouts toward the goal state.
Takeaways & Limitations
STEP’s current implementation is specific to the block assembly scenario and requires modifications to prompts, state representations, and functions for new scenarios.
Abstract
from arXiv · showhide
Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.
I. INTRODUCTION
STEP addresses state-tracking and action-execution limitations in MM-LLM robot planning by forecasting structured states alongside actions and evaluating the approach in simulated block assembly.
- MM-LLMs struggle to track planned-action effects, causing executed actions to deviate from the intended goal state.
- Natural-language “verb, noun” plans omit execution parameters such as placement location and object orientation, creating ambiguity.
- STEP prompts an MM-LLM to estimate the operator’s task and workstation state, predict remaining actions, and propagate future states.
- Structured state propagation supplies assistance parameters and enables quantitative distance tracking from the inferred goal for iterative plan regeneration.
- 32.8% higher action executability and 14.8% lower final-state error were achieved against a state-of-the-art baseline in simulated block assembly.
II. RELATED WORK
Prior work applies LLMs and MM-LLMs to shared teleoperation, robot assistance, and long-term action anticipation, especially where domain-specific data are scarce.
- Shared teleoperation aims to increase production while enabling humans to assume supervisory roles.
- Prior shared-teleoperation systems infer or estimate user goals from observed actions before providing assistance.
- MM-LLMs support in-context generalization across domains when provided with few domain-specific examples during inference.
C. State estimation using language models
Language-model state estimation addresses state-tracking limitations, but existing approaches may rely on expressive natural-language descriptions, affordable queries, or restrictive assumptions.
- LLMs often hallucinate actions that diverge from the intended goal because they lack motivation to track system state.
- Natural-language state representations may not express robot-workspace states precisely enough to provide more than vague goal guidance.
- STEP propagates environment state as structured JSON, enabling quantitative goal-distance calculation during action planning.
- Statler updates structured state for one action without action planning, while Wall-E requires an initial ground-truth state and exploratory rule-learning phase.
III. APPROACH
STEP combines task estimation, structured workstation-state estimation, future-action prediction, state propagation, and goal-directed rollout planning from operator actions and workstation images.
- Task Estimation: STEP prompts an MM-LLM with operator action history, a workstation image, and examples to estimate the overall task.
- Action and State Prediction: Future actions are predicted from the estimated task and action history, after which the current state is propagated to predict future workstation states.
- State Representation Generation: The method estimates each block’s structured state from the workstation image and formats the collected information as JSON.
- State Representation Generation: The state representation records block location, orientation, top-face orientation, and relative placement, with modular descriptors for extension.
C. Action Prediction
STEP uses the estimated task and action history to generate future actions, then propagates the current workstation state to predict the states resulting from those actions.
- Action Prediction: STEP prompts the MM-LLM with the estimated task, recorded action history, and in-context examples to predict future actions.The examples use performed actions → future actions pairs from the pre-collected dataset.
- State Propagation: The method propagates the estimated current workstation state to track changes caused by executing each predicted action.State propagation is performed iteratively across the predicted action sequence.
- State Propagation: For each future action, STEP predicts a corresponding next workstation state as a JSON object.The prompt includes prior actions, the current state, and the next action to obtain the next state.
- State Propagation: A two-part chain-of-thought procedure first describes state changes in natural language and then uses that description for propagation.The authors report improved performance from breaking state propagation into these two parts.
E. Rollouts
STEP uses structured state distance to select progressively closer predicted states and iteratively regenerate action plans toward the inferred assembly goal.
- E. Rollouts: STEP computes dT as the total number of differences between each predicted workstation state and the estimated task-completion state.With five blocks and four characteristics per block, dT ranges from 0 to 20.
- E. Rollouts: Each rollout truncates the predicted sequence after the first state attaining the lowest dT, then uses that state as the next rollout’s current state.This retains the action prefix leading to the closest predicted state.
- E. Rollouts: Because each selected state has dT no greater than the rollout’s starting state, STEP guides plans toward the inferred goal.Rollouts continue until dT = 0 or the maximum rollout count is reached.
- Baseline: The baseline uses the same inputs for task estimation and one action-prediction rollout, but provides no state information.The comparison isolates the effect of explicitly estimating and tracking system state without interactive planning or human feedback.
- Experimental Setup and Dataset: The evaluation dataset contains 495 teleoperated instances from 19 users covering eight five-block assembly structures.The structures include Tuning fork, Bridge, Arch, Snake, Horse, Frame, Stacking, and Low base tuning fork.
B. Metrics
The evaluation measures action executability, final-state accuracy, action-sequence overlap, task estimation, and current-state estimation using defined state and action comparisons.
- B. Metrics: Quantitative metric calculation manually propagates the ground-truth workstation state using rules that incorporate predicted actions and states.Predicted states help determine assistance parameters needed to remove execution ambiguity.
- B. Metrics: Executability is the percentage of predicted actions that satisfy logical ordering and execution constraints before the sequence is truncated at the first failure.Checks include object poses and action preconditions.
- B. Metrics: Final-state error measures the difference between the expected task-completion state and the final state achievable by successfully executing predicted actions.The achievable state comes from the last predicted action that executes successfully.
- B. Metrics: LCS is the percentage obtained by dividing the longest common action subsequence by the longer of the predicted and remaining recorded action sequences.Gaps are allowed when action order is preserved, but LCS alone does not guarantee execution success.
- B. Metrics: Task correctness is one when the predicted task matches any possible task at inference time and zero otherwise.
- B. Metrics: Current state error compares the estimated current state with the ground-truth state at inference time using their distance.
V. RESULTS & DISCUSSION
The results section reports experiments using GPT-4o and a comparison table covering execution, state, task, and sequence metrics, with outputs selected by log probability.
- V. RESULTS & DISCUSSION: GPT-4o served as the MM-LLM for the main results.
- V. RESULTS & DISCUSSION: Table I compares STEP with the baseline using executability, final error, LCS, task correctness, and current state error across task-completion percentages.
- V. RESULTS & DISCUSSION: Each prompt generated five sampled outputs, and STEP selected the output with the highest returned log probability.
A. Comparison with Baseline
Across task-completion levels, STEP generated more executable plans and achieved lower final-state error than the baseline, while producing shorter plans with lower LCS.
- STEP produced more executable action plans than the baseline at every evaluated task-completion percentage.The experiments used 30%, 50%, 70%, and 90% completion levels.
- STEP generated states closer to the operator’s intended goal than the baseline.Its plans also had lower final error and were approximately three actions shorter on average.
- The baseline achieved higher LCS, partly because STEP truncated plans after reaching the state with least distance from the goal.The authors associate the baseline’s extra common actions with unnecessary, less executable actions and less correct resultant states.
- With increasing task completion, both methods achieved better final error because the robot began closer to the goal state.Task correctness first declined and then increased as longer action histories and changing visual discernibility affected estimation.
B. Ablations
Ablations show that rollout count, MM-LLM choice, task correctness, and current-state estimation affect STEP’s execution and final-state performance.
- Rollouts: More rollouts increased action-plan length by 0.7 actions on average per rollout and reduced final error at 50% task completion.Executability decreased as plans lengthened, while the additional executable actions brought the final state closer to the goal.
- MM-LLM choice: STEP outperformed the baseline on executability and final error across most evaluated MM-LLMs.The comparison used GPT-4o, GPT-4o mini, GPT-4.1, GPT-5.1, and GPT-5.2 on 45 randomly sampled instances.
- MM-LLM choice: GPT-4o mini was the exception, with the baseline slightly outperforming STEP because its low task correctness propagated errors through later stages.The authors attribute this behavior to the model’s relatively smaller size limiting task-estimation capability.
- Task and state estimation: Correct task prediction improved downstream predicted-action performance and demonstrated modularity across pipeline stages.The ablation reports that improvements earlier in the pipeline reliably transferred to better final predicted actions.
- Task and state estimation: Executability and final error improved when the current workstation state was predicted without error.This result supports independently improving state estimation as an avenue for improving downstream action prediction.
VI. CONCLUSION
STEP explicitly models state transitions to produce more accurate and executable robot action plans, using propagated states and goal distance to guide task-convergent planning. Evaluation showed improved executability and final-state error, while the current implementation remains scenario-specific and not yet real-time.
- STEP explicitly models state transitions to improve the accuracy and executability of robot action plans in collaborative industrial tasks.Its pipeline estimates the task and workstation state, predicts actions, and propagates the state representation to derive execution parameters.
- Propagated workstation states provide assistance parameters for executing predicted actions and support quantitative tracking of distance from the goal state.STEP uses this distance to guide predicted actions toward the goal.
- Evaluation on a teleoperated-robot assembly dataset demonstrated improved executability and final-state error over the existing state of the art.The conclusion reports extensive evaluation and improved performance on both measures.
- STEP’s current implementation is specific to the block assembly scenario and requires scenario-relevant prompts, state representations, and function implementations for extension.The authors identify generalization to varied scenarios with minimal modifications as future work.
- Additional work is required to make STEP real-time because its extra MM-LLM queries trade off speed for performance accuracy.The reported mean runtime was 148.07 seconds for STEP versus 18.24 seconds for the baseline.