Source-linked AI summary
KINO: A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation
Sitong Chen, Fatemeh Zargarbashi, Jin Cheng, Tianxu An, Stelian Coros
TL;DR
Humanoid loco-manipulation must connect semantic task planning with dynamically feasible whole-body control, yet VLMs and low-level controllers use different kinds of inputs and goals. KINO uses VLM-selected, scene-retargeted keyframes to guide a whole-body RL policy, and saliency-based training improves sparse-keyframe task success from 44% to 92%.
Problem
VLMs reason about semantic objectives, whereas low-level RL controllers typically require precise state observations and structured goals, leaving their interface a challenge for humanoid loco-manipulation.
Source: §I · p. 1Method
KINO uses a VLM to select task-relevant keyframes from a library, retargets them to the scene, and passes them to a keyframe-conditioned whole-body policy.
Source: §I · pp. 1–2Results
44% to 92%: saliency-based keyframe sampling improves end-to-end success under sparse VLM keyframe goals.
Source: §I · p. 2Takeaways & Limitations
KINO performs one- and two-handed manipulation and generalises to placement locations beyond the training reference data in simulation and on a G1 humanoid.
Source: §I · p. 2, §V · p. 7The manually constructed and labelled keyframe library limits scalability as the number of tasks increases, and deployment relies on third-view observations and external mocap for relative object pose.
Source: §V · p. 8
Abstract
from arXiv · showhide
Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control. Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals. We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system successfully performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.
I. INTRODUCTION
Humanoid loco-manipulation must combine semantic task reasoning with dynamically feasible whole-body motion, but connecting high-level VLM planning to low-level control remains challenging. KINO uses keyframes as an intermediate interface and reports improved success with saliency-based sampling, alongside simulation and hardware demonstrations.
I. INTRODUCTION
- Humanoid loco-manipulation requires task understanding and subtask decomposition alongside dynamically feasible whole-body motion.Integrating high-level task planning with low-level motion control remains a central challenge.Source: §I · p. 1
- Hierarchical control faces an interface mismatch: VLMs reason semantically, while RL controllers typically require precise state observations and structured goals.End-to-end VLA training requires large, diverse robot-specific interaction datasets, while scaling RL across long-horizon tasks remains challenging.Source: §I · p. 1
- Saliency-based keyframe sampling improves end-to-end success from 44% to 92% under sparse keyframe goals, while KINO completes tasks by following keyframes and generalizes to placement regions beyond training reference data.Source: §I · p. 2
- KINO uses motion keyframes to bridge VLM semantic planning and whole-body RL control.The VLM selects a task-relevant keyframe from a preconstructed library using the task command, scene observation, and robot context; the keyframe is adjusted to the scene and passed to a keyframe-conditioned policy.Source: §I · pp. 1–2
II. RELATED WORK
Prior work provides strong motion-tracking and foundation-model capabilities, but challenges remain in specifying desired behavior for dynamic whole-body humanoid control. KINO uses a VLM to select sparse keyframes as goals for a low-level policy.
A. Humanoid Loco-Manipulation
- Motion-tracking approaches can produce diverse humanoid behaviors, but require a complete kinematic reference trajectory at inference; KINO instead needs a sparse keyframe [6], –[24].The cited work includes HDMI [7], OmniRetarget [5], ResMimic [23], and SoftMimic [24].Source: A. Humanoid Loco-Manipulation · p. 2
B. Foundation Models for Robot Planning and Control
- VLM high-level reasoning avoids requiring extensive robot-specific data to learn end-to-end VLA motor outputs, while providing an explicit goal for a low-level policy, [4], [25]–[32].ReKep [32] uses spatio-temporal relational keypoint constraints but does not address dynamic whole-body humanoid control.Source: B. Foundation Models for Robot… · p. 2
C. Motion Keyframes
- Keyframes represent diverse behaviors without prescribing complete trajectories; KINO uses a foundation model to predict the next task keyframe rather than requiring manual inference-time specification –.Source: C. Motion Keyframes · p. 2
III. METHOD
KINO separates semantic planning from whole-body motion execution: a VLM selects and retargets a library keyframe, then a goal-conditioned policy generates robot actions. The policy is trained with reference, goal-reaching, and regularization objectives and uses privileged critic observations during training.
III. METHOD
- The VLM selects a task-relevant keyframe from a predefined library using the instruction, robot context, and scene observation; the keyframe is retargeted and passed to a low-level policy.Keyframes encode intermediate poses such as approaching, crouching, or picking an object.Source: §III · pp. 2–3, Figure 2 · p. 3
A. Keyframe-Conditioned Policy
- The goal-conditioned policy receives robot state, object state, previous action, and the next keyframe, then outputs whole-body actions to reach the pose.Training uses reference tracking, keyframe goal-reaching, and regularization rewards, with separate value functions for the reward groups; the action vector contains target joint positions tracked by a PD controller.Source: A. Keyframe-Conditioned Policy · p. 3
- During training, the critic receives privileged reference-state and goal-error information, while the deployed policy relies only on observations available in the real world.The critic also receives the remaining time to the goal and uses a temporal history of three observations.Source: A. Keyframe-Conditioned Policy · p. 3
1) Keyframe Sampling for Policy Training:
The training sampler combines random future-frame goals with saliency-based selection to represent distinct loco-manipulation stages. It filters periodic leg motion and incorporates object-motion events, producing keyframes that better correspond to semantically distinct stages.
1) Keyframe Sampling for Policy Training:
- Uniform sampling can underrepresent approaching, lifting, transporting, and placing stages among many intermediate walking frames.The approach augments random future-frame sampling with specialized saliency-based selection.Source: 1) Keyframe Sampling for Policy Training · p. 4
- Acceleration profiles of selected robot bodies and object motion provide salient keyframe candidates for training.The initial candidates come from robot-body acceleration; object-motion events further refine the set.Source: 1) Keyframe Sampling for Policy Training · p. 4
- Local maxima in filtered aggregate body acceleration mark motion changes, while periodic alternating leg-motion candidates are removed.The acceleration peaks often align with task-stage transitions such as picking or placing.Source: 1) Keyframe Sampling for Policy Training · p. 4
- After the object starts moving, a robot candidate is retained when it falls within δ of an object-event candidate.Object-event candidates are peaks in the object's vertical acceleration.Source: 1) Keyframe Sampling for Policy Training · p. 4
- Object-event candidates not coinciding with existing candidates are added to preserve important object transitions.The resulting salient set also includes the first and last motion frames.Source: 1) Keyframe Sampling for Policy Training · pp. 3–4
- Filtering periodic leg-motion candidates leaves keyframes that correspond more closely to semantically distinct stages, providing a better policy-training sampling strategy.Fig. 3 illustrates the removal of many intermediate walking keyframes while retaining salient ones.Source: 1) Keyframe Sampling for Policy Training · p. 4, Figure 3 · p. 4
2) Keyframe Library:
The keyframe library stores representative poses for semantic stages of loco-manipulation, pairing robot and, when applicable, object poses with language descriptions. During execution, the VLM selects a keyframe using its text label, and its states are retargeted to the current scene before policy control.
2) Keyframe Library:
- Each library keyframe represents a semantic motion stage, such as approaching, lifting, or carrying an object.The library contains representative poses drawn from the motion dataset.Source: 2) Keyframe Library · p. 4
- A keyframe stores the full robot pose and, when applicable, the object pose, specifying their nominal spatial relationship.Source: 2) Keyframe Library · p. 4
- Each keyframe has a natural-language description that the VLM uses to select it online before the robot and object states are retargeted to the current scene.Source: 2) Keyframe Library · p. 4
C. Scene Alignment
Scene alignment adapts a selected library keyframe to the target object's observed pose and dimensions. The retargeter transforms end-effector and robot-root positions, then uses inverse kinematics to recover the remaining robot configuration.
C. Scene Alignment
- Retargeting adapts a generic library keyframe to the target object's pose and dimensions before the low-level policy receives it.The retargeter uses inverse kinematics to adapt robot body positions to the target object.Source: 1) Keyframe Retargeting · p. 4
- The source object frame is rotated to align with the observed orientation of the target object.Source: 1) Keyframe Retargeting · p. 4
- The transform preserves each point's normalized position relative to the source object while adapting to target pose and dimensions; robot-root updates affect only horizontal coordinates.Damped least-squares inverse kinematics minimizes end-effector position errors and weighted vertical foot-position residuals to keep the feet at ground height.Source: 1) Keyframe Retargeting · p. 5
2) Path Planning:
KINO uses keyframes to connect semantic VLM planning with low-level control, while path planning inserts feasible intermediate locomotion goals. The planner uses task, scene, and execution context to select and update goals during execution.
2) Path Planning:
- Locomotion paths use tangent-aligned circular arcs or shortest valid Dubins paths, limited to 3 m^-1 curvature and sampled into goals with bounded heading change and displacement.The greedy selection limits accumulated heading change to 45° and displacement between consecutive goals to 2 m.Source: 2) Path Planning · p. 5
D. VLM-Based Planner
- The VLM interprets the instruction and scene, selects the next semantic keyframe, and decides whether the task is complete.The selected keyframe is retargeted into a goal for the low-level policy.Source: D. VLM-Based Planner · pp. 5–6
- Each planning request combines the user’s long-horizon instruction, a third-person scene image, and structured robot context.The context includes the previous keyframe, goal outcome, robot and object poses, target pose, and controller diagnostics; it supports advancing, retrying, or selecting recovery goals.Source: D. VLM-Based Planner · p. 5
- The planner sends a new VLM request after the robot and object become stationary; otherwise, it retains the current goal while the policy executes it.The resulting keyframe is retargeted before being passed to the low-level policy.Source: D. VLM-Based Planner · pp. 5–6, Figure 5 · p. 5
IV. RESULTS
The simulation evaluation tests closed-loop planning and whether keyframe goals support placements beyond the training reference data. It also describes the task and motion data used in evaluation.
IV. RESULTS
- The evaluation covers two-handed box and one-handed bucket pickup, transport, and placement, using locomotion and manipulation motion clips with object-pose augmentation.The reference data includes 52 AMASS locomotion clips, 105 box manipulation clips, and 120 in-house bucket manipulation clips.Source: §IV · p. 6
A. Sim-to-Sim
- In simulation, the planner selects keyframes from the task and robot context, and the system executes them through the retargeter and policy.The VLM chooses one- or two-handed keyframes according to the object named in the command.Source: A. Sim-to-Sim · p. 6
- The box placement evaluation varies target positions over a grid and compares success rates with the spatial coverage of the training data.A trial counts as successful when the final object position is within 0.3 m of the target; successful placements outside the training-data contour indicate generalisation beyond the reference trajectories.Source: A. Sim-to-Sim · p. 6, Figure 7 · p. 6
B. Real robot deployment
The G1 deployment demonstrates VLM-guided one- and two-handed object manipulation. The sampling comparison reports better reaching accuracy in most scenarios, with a placement-phase exception.
B. Real robot deployment
- On the Unitree G1, the VLM-selected keyframes guide successful two-handed box and one-handed bucket pickup and placement.The deployment uses onboard robot-state sensing and external motion capture to estimate the object-relative pose.Source: B. Real robot deployment · p. 6, Figure 8 · p. 7
C. Ablation on keyframe sampling
- The sampling comparison uses a uniform strategy with keyframes spaced at a fixed 1.5-second interval and evaluates task success over 100 experiments.Starting distance to the box varies from 0.5–2.3 m, and target distance from the initial box pose varies from 0.2–2.0 m.Source: C. Ablation on keyframe sampling · pp. 6–7
- Saliency-based sampling achieves better goal-reaching accuracy in most scenarios, except for object position error during placement.The reference placement keyframe puts the box too close to the robot, making the motion unstable; the RL policy instead matches root position while placing the object farther away.Source: Table II · p. 7, C. Ablation on keyframe sampling · p. 7
D. Failure recovery
The conclusion reports a success-rate increase with saliency-based sampling and describes VLM-guided recovery after failed subtasks. It also identifies library construction and perception as limits on broader deployment.
D. Failure recovery
- After a failed subtask, the VLM can use vision and planner-context errors to resend the previous keyframe and guide a retry.The described recovery is an example in which the VLM may guide the policy toward successful execution after a failure.Source: D. Failure recovery · p. 7, Figure 9 · p. 7
V. CONCLUSION
- Task success increased from 44% to 92% with saliency-based keyframe sampling compared with the uniform baseline.The conclusion also reports simulation and hardware experiments in which the VLM selects one- or two-handed keyframes and reattempts certain motions after detecting failures.Source: §V · p. 7
- The current library is manually constructed and labelled, limiting scalability as the number of tasks increases.The authors propose automatically extracting salient keyframes and assigning semantic descriptions as a way to reduce manual effort.Source: §V · p. 8
- The results suggest motion keyframes provide a practical, interpretable interface between semantic planning and whole-body control.Scaling to more diverse tasks and operating with onboard sensing remain steps toward general-purpose humanoid manipulation in real-world environments.Source: §V · p. 8