Source-linked AI summary
ThinkLocoManip: A Whole-Body Control Agentic Harness for Legged Loco-Manipulation
Nuo Chen, Shizeng Zhang, Yujie Yang, Jinghong Wang, Zhenyang Qu, Xiao-Ming Wu, Yunfei Guo, Zhongwen Lu, Haojie Liu, Chunpeng Lu, Qiao Sun, Xueyi Zhang, Jiangwei Zhong
TL;DR
Long-horizon legged loco-manipulation remains challenging because direct action generation and plan-then-act approaches can struggle with generalization and error recovery. ThinkLocoManip interleaves reasoning and acting across specialized tools and skills, and achieves a 77% overall success rate compared with 37.1% for the strongest plan-then-act baseline.
Problem
Direct action generation and plan-then-act approaches can struggle with the generalization and error-recovery capabilities needed for complex, long-horizon tasks in large-scale environments.
Source: §I · p. 1Method
ThinkLocoManip interleaves reasoning and acting, coordinating specialized tools across perception, navigation, manipulation, and locomotion, with skills and Keyframe Thinking for long-horizon execution and recovery.
Source: §I · p. 2, C. ThinkLocoManip-Skill for… · p. 5, D. Keyframe Thinking for Error Recovery · p. 5Results
Across challenging long-horizon legged loco-manipulation tasks, ThinkLocoManip outperforms representative direct-action and plan-then-act methods, achieving 77% overall success versus 37.1% for the strongest plan-then-act baseline.
Source: §I · p. 2Takeaways & Limitations
Real-world experiments, ablations, interference tests, and in-the-wild evaluations support the framework’s robustness and generalization across long-horizon loco-manipulation tasks.
Source: §V · p. 8
Abstract
from arXiv · showhide
In this paper, we propose ThinkLocoManip, an agentic harness for legged loco-manipulation designed to tackle complex, long-horizon tasks in large-scale environments. Existing approaches typically follow either direct action generation or a plan-then-act paradigm, both of which can struggle with the generalization and error-recovery capabilities required for extended task execution. In contrast, ThinkLocoManip adopts an interleaved reasoning-and-acting paradigm, allowing the agent to continuously reason about its current state, execute actions, observe their outcomes, and adapt its strategy accordingly. Building on this paradigm, our agentic harness orchestrates a suite of specialized tools for perception, navigation, manipulation, and locomotion, enabling flexible composition of heterogeneous capabilities. Unlike existing robotic agentic harnesses, ThinkLocoManip explicitly incorporates locomotion as a tool domain for agent reasoning and execution. We further design a set of skills that explicitly account for both tool usage within each functional domain and coordination across domains, supporting effective execution of complex behaviors over long horizons. To tackle task failure during long-horizon interactions, we develop keyframe thinking mechanism to preserve critical execution context and support efficient error recovery. Extensive real-world experiments demonstrate that ThinkLocoManip substantially outperforms representative methods based on both direct action generation and plan-then-act paradigms across challenging long-horizon legged loco-manipulation tasks. Notably, ThinkLocoManip achieves a success rate of 77%, more than twice that of the strongest plan-then-act baseline (37.1%). Our experiments further validate the effectiveness of interleaved reasoning and acting, tool orchestration, and coordinated skill execution.
I. INTRODUCTION
Long-horizon legged loco-manipulation requires robots to coordinate navigation, physical interaction, and whole-body motion, but existing approaches can struggle to generalize and recover from errors. ThinkLocoManip interleaves reasoning and acting across specialized tools and reports strong real-world task performance.
I. INTRODUCTION
- Loco-manipulation integrates navigation, physical interaction, and whole-body motion over extended task horizons in large-scale environments.Legged mobility provides flexibility and robustness for traversing complex terrains.Source: §I · p. 1
- Direct action generation can struggle to generalize, while plan-then-act can be brittle when observations invalidate plans or execution errors require recovery.Both paradigms remain limited for robust execution of complex, long-horizon tasks in large-scale environments.Source: §I · p. 1
- ThinkLocoManip continuously reasons about state, invokes tools, observes outcomes, and adapts, with tools for perception, navigation, manipulation, and locomotion.The framework also coordinates skills across domains and uses Keyframe Thinking to identify execution failures and adapt behavior.Source: §I · p. 2
- 77% overall success, more than twice the strongest plan-then-act baseline’s 37.1%, across challenging long-horizon real-world tasks.Figure 1 includes a performance comparison of long-horizon real-world tasks.Source: Figure 1 · p. 1, §I · p. 2
II. RELATED WORKS
Prior loco-manipulation approaches include optimization-based control and learning-based methods, including direct action generation and plan-then-act. ThinkLocoManip instead uses an interleaved reasoning-and-acting harness.
A. Loco-Manipulation Policy
- Optimization-based methods – use whole-body optimization, including model predictive control, to coordinate legged locomotion and arm manipulation.They can incur high computational costs and encounter infeasibility under hard constraints.Source: A. Loco-Manipulation Policy · p. 2
- Direct action generation methods – learn loco-manipulation capabilities from data but can struggle in diverse real-world environments and long-horizon tasks because of limited training-data diversity and the sim-to-real gap.These methods include whole-body controllers, goal-conditioned policies, imitation learning, and vision-language-action models.Source: A. Loco-Manipulation Policy · p. 2
- Plan-then-act methods – compose reusable primitive tools into high-level plans, but separated reasoning and execution can hinder recovery from invalidated plans or errors.The framework comparison is shown in Figure 2.Source: A. Loco-Manipulation Policy · p. 2, Figure 2 · p. 2
- ThinkLocoManip interleaves reasoning and acting rather than committing to a fixed policy or pre-generated plan.The agent invokes tools, observes outcomes, and adapts its strategy; Figure 2 contrasts this with direct action generation and plan-then-act.Source: A. Loco-Manipulation Policy · p. 2, Figure 2 · p. 2
B. Agentic Harness for Robotics
Robotic agentic harnesses support closed-loop interaction and recovery, but extending them to legged loco-manipulation requires coordination between locomotion and manipulation. ThinkLocoManip addresses this through locomotion-aware tools and a hierarchy of tools, skills, and Keyframe Thinking.
B. Agentic Harness for Robotics
- Existing robotic agentic harnesses support task decomposition, closed-loop interaction, monitoring, and recovery, but legged loco-manipulation demands coordination of locomotion and manipulation in large, unstructured environments.Earlier harnesses often generate complete executable programs upfront and cannot adapt to intermediate execution outcomes.Source: B. Agentic Harness for Robotics · p. 3
- ThinkLocoManip explicitly treats locomotion as a first-class tool domain alongside perception, navigation, and manipulation for long-horizon legged tasks.The agent can reason about and invoke locomotion capabilities during task execution.Source: B. Agentic Harness for Robotics · p. 3
III. METHOD
- The method has three hierarchical components: primitive tools, ThinkLocoManip-Skill for within- and cross-skill coordination, and Keyframe Thinking for failure identification and replanning.Keyframe Thinking is introduced as an error-recovery mechanism for long-horizon execution.Source: §III · p. 3
A. Task Definition
The task is to coordinate locomotion and manipulation over an extended horizon to complete a natural-language instruction. The agent makes closed-loop decisions from the instruction and execution history, then navigates to and performs each sub-task while updating its plan as needed.
A. Task Definition
- Long-horizon legged loco-manipulation is framed as autonomously coordinating locomotion and manipulation over an extended horizon to complete a specified natural-language task.The agent’s action decision is conditioned on the task instruction and its execution history.Source: A. Task Definition · p. 3
- The agent-level decision determines locomotion and manipulator behaviors, with observations indexed by time step.The formulation describes closed-loop decision making.Source: A. Task Definition · p. 3
- For each sub-task, the agent navigates to its location and coordinates body posture with end-effector pose, then incorporates observations to detect failures and replan from the current state.Execution errors or external interference can disrupt the planned sub-task sequence.Source: A. Task Definition · p. 3
B. Library of Loco-Manipulaton Primitive Tools
ThinkLocoManip combines specialized tools for perception, navigation, manipulation, and locomotion. The agent selects and composes them to address the requirements of long-horizon sub-tasks.
Primitive tool domains
- The tool library spans perception, navigation, manipulation, and locomotion, enabling the agent to compose capabilities for diverse long-horizon tasks.The framework also coordinates tool use within functional domains and across domains.Source: B. Library of Loco-Manipulaton… · pp. 3, 5, Figure 3 · p. 4
Perceptive tools
- Head and end-effector cameras support object search and close-range grasping, while prompted SAM3 segmentation returns a bounding box, mask, and 3D position.The head camera captures RGB images for searching; the end-effector camera provides RGB and depth images for precise close-range grasping.Source: B. Library of Loco-Manipulaton… · p. 3
Navigational tools
- Navigation offers map-based planning to named waypoints and target-based straight-line approaches that do not require a pre-built map.The target-based approach first aligns the robot’s orientation, then moves straight toward the target.Source: B. Library of Loco-Manipulaton… · p. 4
Manipulative and locomotion tools
- Manipulation tools include grasp-pose detection, lifting, putting down, pushing, and pulling; locomotion tools control body velocity and posture.Velocity control uses a robust reinforcement-learning policy, while pose control computes leg joint angles for a desired posture with stable foot contacts.Source: B. Library of Loco-Manipulaton… · p. 4
C. ThinkLocoManip-Skill for Long-Horizon Tasks
ThinkLocoManip-Skill organizes primitive tools into workflows with invocation conditions and behavioral constraints. Its skills coordinate sub-task execution, body posture, manipulation, and recovery across long-horizon tasks.
Skill framework
- ThinkLocoManip-Skill defines tool-composition workflows within each skill and behavioral constraints for coordinating different skills.The structure addresses the challenge of choosing and ordering tools while coordinating posture and responding to execution failures.Source: C. ThinkLocoManip-Skill for… · p. 5
Skill set
- Five skills cover sub-task planning, path navigation, object searching, error recovery, and whole-body manipulation.Their descriptions help match skills to sub-tasks, and pre-rules specify when to invoke them.Source: C. ThinkLocoManip-Skill for… · p. 5
Planning, searching, and recovery
- Sub-task planning verifies each completed sub-task and replans the remaining order after failure; error recovery uses execution logs to analyze failures and replan.Object searching rotates the robot 45° clockwise until the target is detected or the attempt limit is reached.Source: C. ThinkLocoManip-Skill for… · p. 5
Whole-body manipulation
- Whole-body manipulation combines manipulation tools with locomotion tools to adjust body posture, maintain balance, and facilitate task execution.The body pose is computed from the target’s position in the robot’s body frame, and pose control orients the robot toward the object.Source: C. ThinkLocoManip-Skill for… · p. 5
Skill template and constraints
- The skill template requires object identity and 3D position; for grasping it computes a robot pose before grasping, while opening uses a pull followed by a short retreat.Velocity control is prohibited except for chest or cabinet opening.Source: C. ThinkLocoManip-Skill for… · p. 5
D. Keyframe Thinking for Error Recovery
Keyframe Thinking records critical task context at object-contact moments and uses execution logs to diagnose failures and replan sub-tasks. The section’s materials also identify the indoor-task execution sequence and the scope of the real-world comparison table.
D. Keyframe Thinking for Error Recovery
- Keyframe Thinking logs RGB images, robot states, the current sub-task, and primitive-tool name at object-contact moments, rather than recording every frame.After each sub-task, the resulting state is also provided to the agent to assess task completion.Source: D. Keyframe Thinking for Error Recovery · p. 5
- When execution failures disrupt a sub-task sequence, the agent analyzes execution logs, identifies potential failure causes, and replans the sequence.Recording only keyframes preserves essential task context while reducing reasoning time.Source: D. Keyframe Thinking for Error Recovery · p. 5
- Fig. 4 depicts the robot and objects, then shows five indoor task executions from navigation through task completion.Table II is identified as a real-world long-horizon task comparison, with average-success-rate improvement over the second-best method shown in parentheses.Source: Figure 4 · p. 6, Table II · p. 6
A. Experimental Setup
The evaluation uses five real-world indoor tasks on a wheeled quadruped with a front-mounted arm, controlled from natural-language instructions. The tasks range from single-object actions to multistage chest and cabinet manipulation.
A. Experimental Setup
- Five real-world indoor tasks evaluate autonomous execution from natural-language instructions on a Lenovo BOT MX wheeled quadruped with an Agilex PiPER 6-axis arm.The agent uses GPT-5.6 Sol as its LLM backbone; the tasks are grouped into three simple and two challenging tasks.Source: A. Experimental Setup · p. 6
- The simple tasks test table grasping at two heights, ground grasping, and color-specific button pressing followed by navigation back.The tasks require navigation and posture adjustment as well as manipulation.Source: A. Experimental Setup · p. 6
- The challenging chest and cabinet tasks require opening an enclosure, retrieving an object, and coordinating sequential navigation and manipulation.The chest task also requires placing the retrieved object on a table.Source: A. Experimental Setup · p. 6
- Table III is identified as an ablation study, while Table IV reports results for an in-the-wild task.The supplied captions do not state either table’s results.Source: Table III · p. 7, Table IV · p. 7
B. Indoor Long-Horizon Task
On the indoor long-horizon tasks, navigation-related pose deviations can shift learning-based methods’ visual and object-relative inputs, while ThinkLocoManip performs better overall. The ablation discussion highlights locomotion tools and Keyframe Thinking as important design components.
B. Indoor Long-Horizon Task
- Learning-based methods perform poorly when navigation-related pose deviations shift visual observations and object-relative configurations; ThinkLocoManip performs better overall.The passage notes particular sensitivity in Button Pressing, which requires high-precision manipulation.Source: B. Indoor Long-Horizon Task · p. 7
C. Ablation Study
- The ablation discussion says whole-body locomotion tools help handle varying manipulation heights, while removing Keyframe Thinking reduces performance on three tasks.The authors attribute the keyframe mechanism’s value to cues for diagnosing execution failures and choosing recovery actions.Source: C. Ablation Study · p. 7
- Fig. 5 is identified as an outdoor-scene ground-grasping task, and Table V as results on robustness to external interference.The supplied figure and table captions do not state their outcomes.Source: Figure 5 · p. 7, Table V · p. 7
E. Robustness to External Interference
ThinkLocoManip achieves a 77.0% overall success rate against 37.1% for the strongest baseline. In external-interference trials, the authors report that its error-recovery skill and Keyframe Thinking support failure diagnosis and behavioral adaptation.
E. Robustness to External Interference
- Under external interference, ThinkLocoManip achieves the best performance, which the authors attribute to its Error Recovery skill and Keyframe Thinking.The authors report that these designs help the agent diagnose failure causes and adapt its behavior; ACT and π0.5 resume execution but do not complete the tasks successfully.Source: E. Robustness to External Interference · p. 7
V. CONCLUSION
- The conclusion describes specialized perception, navigation, manipulation, and locomotion tools, with ThinkLocoManip-Skill structuring tool use and cross-domain coordination.Keyframe Thinking preserves critical execution context for error recovery during long-horizon interactions.Source: §V · pp. 7–8
- 77.0% overall success rate for ThinkLocoManip compares with 37.1% for the strongest baseline.The conclusion also reports that ablations, external-interference experiments, and in-the-wild evaluations validate whole-body control, Keyframe Thinking, and framework robustness and generalization.Source: §V · p. 8