Source-linked AI summary
Instance-anchored interaction evidence: Grounding robot plans in human pointing and handling
Xinliang Xiao, Bowen Yang, Wenjing Zhang, Li Yang, Wei Zhou
TL;DR
Robots must infer from human videos which objects were indicated or moved, even when identical instances have changed position. IAE anchors interactions to final-scene identities and decodes them into plans, outperforming a 32B VLM on implicit-intent and episodic tasks in WatchAct.
Problem
Video-to-plan reasoning must connect earlier human interactions to specific execution-scene instances, including visually identical objects that may have moved.
Source: §1 · p. 2Method
IAE registers final-scene objects to public identifiers, tracks their identities backward through video, and combines hand–instance evidence with task-outcome learning and program decoding.
Source: §1 · p. 3Results
64.2% plan success on WatchAct implicit-intent tasks versus 27.5% for the 32B VLM; on restoration, reversal and imitation, IAE reaches 46.4% versus 27.0%.
Source: §1 · p. 3Takeaways & Limitations
Identity bookkeeping is largely addressed by registration and tracking, while reading pointing gestures remains the main source of errors.
Source: §7 · p. 19With one-grid-cell layout errors, a third of videos change identity assignment and overall success drops by 12 points; unavailable layouts or identical instances sharing a region require another anchor.
Source: §7 · p. 19
Abstract
from arXiv · showhide
A robot that assists people must often act on what a person has shown rather than said: which of several identical cartons was pointed at, or which box was handled. The plan is executed from the final scene, whereas the evidence occurs earlier, possibly on objects that have since moved. We propose instance-anchored interaction evidence (IAE), which registers every object of the final scene to its public identifier, keeps each identity through the video by backward mask propagation, and describes every frame by the geometry between hands, forearms and these instances. An evidence network trained only from task outcomes scores the instances. For pointing tasks, a grammar-constrained dynamic program trained with a structured loss decodes object-destination programs; symbolic programs handle reference disambiguation and, without learning, episodic tasks. On 1,255 WatchAct benchmark requests, scored by symbolic execution, IAE reaches 64.2% plan success on implicit-intent tasks against 27.5% for a 32B vision-language model (strict success 49.7% against 15.4%), and 46.4% against 27.0% on restoration, reversal and imitation without task-specific training. Controls with the same perception overlays, the same 32 frames, forward tracking, or a relation model trained on the same labels do not explain the gain. Given IAE's evidence as text with its meaning explained, the same language model reaches 57.4%: most of the gain comes from the instance-anchored evidence, and the explicit programs add 6.8 points at a fraction of the cost. Pointing remains the hardest case, with 16.9% strict success. The code is available at https://github.com/WeiZhou96/iae-watchact.
1. Introduction
WatchAct asks robots to infer from video which object instances a person indicated or moved, even when identical objects have since changed position. IAE makes that identity link explicit and combines learned pointing evidence with program decoding; it outperforms a 32B VLM on the reported tasks.
1. Introduction
- WatchAct tests video-to-plan reasoning where people act instead of speaking and the instruction omits information needed for the robot’s plan.Its requests pair human-action videos and instructions with simulated scenes, and score plans after execution.Source: §1 · p. 2
- A VLM’s direct video-to-plan approach struggles when several identical objects must be identified from earlier interactions and referenced by final-scene identifiers.The relevant gesture may occur seconds earlier, after the object has moved.Source: §1 · p. 2
- IAE registers final-scene objects to public identifiers, tracks their masks backward through the video, and represents hand and forearm geometry relative to each instance.Programs use this evidence to identify indicated objects and destinations, selected instances, or moved objects and their origins.Source: §1 · p. 3
- For pointing, task outcomes supervise a small evidence network, while a grammar-constrained dynamic program decodes object and destination cues into a complete program.A structured loss trains evidence to make the decoded program correct without gesture-episode annotations.Source: §1 · p. 3
- The evaluation covers 1,255 WatchAct requests and includes official and strict success metrics, cross-validation, controls, and VLM baselines.The evaluation includes models receiving the same perception overlays, IAE evidence, or prompts to verify ambiguous decisions.Source: §1 · p. 3
- 64.2% plan success on implicit-intent tasks versus 27.5% for the 32B VLM; strict success is 49.7% versus 15.4%.On restoration, reversal and imitation, IAE reaches 46.4% without task-specific training, compared with 27.0% for the 32B VLM; plans are scored by symbolic execution.Source: §1 · p. 3
2. Related work
Prior work studies video-to-plan generation, gesture grounding, weakly supervised temporal learning, and identity tracking. IAE differs by explicitly binding cues from third-person video to execution-scene instances and using task outcomes to learn evidence for fixed program structures.
Robot planning from human videos
- Video-to-plan systems and video-conditioned policies derive plans or skills from demonstrations, but mostly rely on VLMs to identify the involved object [2] [3] [10] [11] [12] [13].Benchmarks separately evaluate video-to-plan reasoning, egocentric planning, and learning from observation [1] [14] [15].Source: §2 · p. 3
Understanding pointing and deictic gestures
- Gesture-grounding methods recognize pointing or combine gestures with language, but most resolve a single act in the current view [4] [6] [7] [8] [9] [16] [17] [18] [19] [20] [21].IAE evaluates third-person cues, object movement after indication, final-scene identity binding, and outcome-only evidence learning together.Source: §2 · p. 3
Weak supervision and structured decoding over time
Tracking, identity and program-based reasoning
- Tracking methods and explicit trajectory reasoning address identity errors, while IAE binds identities to execution-scene instances without VLM fine-tuning [5] [25] [26] [27].IAE fixes program structure by task and learns only the evidence consumed by those programs; its program decoding margin supports selective prediction [28] [29] [30] [31] [32].Source: §2 · p. 4
3. Problem formulation
Each request combines a video, an instruction, and a public scene description that defines object identifiers and execution-state locations. The system predicts object–destination operations and evaluates the resulting state against the goal.
3. Problem formulation
- A request contains a video, an instruction, and a public scene description listing object identifiers and their initial regions.The execution scene is usually the video’s final frame, but is the first frame for imitation.Source: §3 · p. 4
- The system outputs object–destination operations, compiles them into pick-and-place actions, and symbolically compares the resulting state with the goal.Official success checks goal regions for objects whose goal differs from the execution state; strict success also requires all other objects to remain in place.Source: §3 · p. 4
- Nonverbal-cue tasks reproduce pointed object–destination pairs; reference-disambiguation tasks identify selected objects; episodic tasks restore, reverse, or imitate moves.Reference cues may be pointing, touching, or moving, and destinations can be defined relative to the person.Source: §3 · p. 4
4. Method
IAE binds execution-scene identities to earlier interactions, then maps learned or endpoint evidence into task-specific plans.
4.1. Final-scene registration and identity-preserving tracking
- Final-scene detections are assigned public identifiers by fitting the layout, then their masks are propagated backward to preserve identity through handling, occlusion, and containment.Registration also projects table regions into the image so empty regions can serve as candidate destinations.Source: §4.1 · p. 5, Figure 1 · p. 5
4.2. Hand–instance interaction evidence
- Each frame represents hand–instance geometry, including pointing rays, contact and motion, finger shape, and candidate context.Forearm and head rays use the eye–fingertip alignment observation to constrain target position despite unknown depth.Source: §4.2 · pp. 5–6
4.3. Learning deictic evidence from task outcomes
- Task outcomes provide candidate-level labels without frame annotations, while a grammar-constrained dynamic program decodes ordered object and destination cues into operations.The labels assume demonstrations contain no retracted or superseded cues; pointing follows an (O+D)+ sequence, assigning pending objects to each destination.Source: §4.3 · pp. 6–7
- A structured hinge loss trains evidence scores to support goal-consistent programs and penalizes programs with wrong or missing operations.A wrong destination costs two units, while an extra or missed object costs one; the program loss applies only when a goal-consistent program is admissible.Source: §4.3 · pp. 7–8
4.4. Programs over instance-anchored evidence
- The decoded-program margin ranks alternatives for selective prediction, but it is not a calibrated probability.A video with no valid program receives no operations and a zero margin.Source: §4.4 · pp. 8–9
- Reference disambiguation uses learned instance scores and instruction wording to select the intended set and the named or person-relative container.For person-relative references among matching containers, the median person ground point determines the closer or farther container.Source: §4.4 · p. 9
- Restoration, reversal, and imitation reuse registration and tracks without learning, comparing each instance’s places at opposite video ends to determine moves.Moves into and out of containers or drawers count regardless of image displacement.Source: §4.4 · p. 9
5. Experimental setup
The evaluation uses five WatchAct tasks, activity-grouped cross-validation, and direct-planning baselines with matched perception and evidence conditions.
Data and protocol
- The five tasks comprise nonverbal cue and reference disambiguation, plus restoration, reversal, and imitation, totaling 1,255 requests.Activities and requests from each activity stay within one fold; episodic programs are applied directly without learning.Source: §5 · pp. 9–10
Metrics
- The study reports benchmark plan success and strict success, which additionally requires all non-target objects to remain in place.Confidence intervals use activity-level bootstrap resampling, and paired comparisons use a two-sided sign test on per-activity differences.Source: §5 · p. 10
Baselines
- Direct-planning VLM baselines receive 32 sampled frames, instructions, and public-scene descriptions; overlay variants also receive IAE’s tracked boxes, identifiers, hand skeletons, and forearm lines.The evaluation also gives a 32B model IAE evidence as structured text, with and without an explanatory prompt.Source: §5 · p. 10
Examples
- Figure 2 presents randomly selected test examples with tracked identities, hand landmarks, forearm lines, and frame logits for three predefined request groups.The groups include strict IAE-only NC successes, NC tray–basket confusions, and RD successes involving selected instances moving more than 60 pixels.Source: Figure 2 · p. 11
6. Results
IAE outperforms direct VLM planning on implicit-intent tasks, while ablations and controls test the roles of tracking, evidence decoding, and sampling. Results also cover episodic tasks, camera and reference variation, failure modes, and sensitivity to data and layout.
6.1. Implicit-intent tasks
- 64.2% plan success with IAE’s ensemble exceeds every direct VLM baseline, which remains below 32% overall.IAE reaches 61.1% with a single model; paired comparisons cover 91 implicit-intent activities.Source: §6.1 · p. 11, Table 2 · p. 12
- 57.4% overall plan success with explained IAE evidence narrows the gap, but IAE remains 6.8 points higher.The same model reaches 37.4% with the plain prompt; providing evidence without explaining its semantics leads to poor RD selection.Source: §6.1 · pp. 11–12
6.2. Ablation
- Grammar-constrained decoding is critical for NC: pairing thresholded candidates by peak time lowers SR from 37.6% to 21.2%.Without grammar decoding, overall SR drops 7.0 points; backward tracking matters most for RD, where removing it lowers strict success by 3.1 points and overall SR by 3.9.Source: §6.2 · pp. 12–13
6.3. Episodic tasks
- 46.4% plan success on restoration, reversal, and imitation without task-specific training is 19.4 points above the 32B model.IAE succeeds at about 46% on each task; its successful plans also satisfy strict success.Source: §6.3 · p. 14, Table 4 · p. 14, §6.2 · p. 14
6.4. Cameras, verification and selective prediction
- IAE improves on every camera and varies by at most 1.7 points across spatial-reference frames, while the 32B VLM loses 5.1 points on person-centred NC.VLM verification changes few ambiguous destinations correctly, and confidence margins rank RD reliability more effectively than NC reliability.Source: §6.4 · pp. 15–16
6.5. Error analysis
- On NC, 52% of IAE goal pairs are placed correctly, but 28% receive a wrong destination and 62% of plans move a non-goal object.Extra-object operations account for the gap between plan SR (42.6%) and strict success (16.9%); supplying goal-object identities for analysis raises strict success to 48.7%.Source: §6.5 · pp. 16–17
6.6. Data efficiency and sensitivity
- With one quarter of training activities, overall SR is 52.2%, while NC SR rises from 26.0% to 37.6% across the tested training-data amounts.RD alone reaches 71.8% with a quarter of the activities.Source: §6.6 · p. 17
- Registration errors from random identity assignments reduce episodic plan success to 27.1%, while layout noise of a quarter cell changes 4.5% of assignments and lowers overall SR by 0.4 points.Decoding cost and program-loss weight trade NC plan SR against strict success; the study retains settings fixed before test-fold evaluation.Source: §6.6 · pp. 17–18
7. Discussion and limitations
IAE separates identity bookkeeping from the harder task of reading pointing gestures, where learned evidence and whole-program decoding help but strict success remains low. Its conclusions are bounded by layout and depth ambiguity, simulator-only evaluation, one benchmark and set of rooms, and extra-object errors.
7. Discussion and limitations
- Identity bookkeeping is largely handled by registration and tracking, while reading pointing gestures remains the main error source at 16.9% strict success on NC.The hand-crafted variant reaches 75.4% on RD, and episodic programs reach 46.4% without learning; explicit programs add 6.8 points over a VLM given IAE evidence and run about a hundred times faster.Source: §7 · p. 19
- Registration depends on the public execution-scene layout, depth-aligned destinations can be ambiguous in 2D, and plans are evaluated only by symbolic execution in the simulator.One-grid-cell layout errors change identity assignments in a third of videos and reduce overall SR by 12 points; physical grasping and placement are not evaluated.Source: §7 · p. 19
- Evidence comes from one benchmark and one set of rooms, while more than 60% of NC plans still move an extra object.Recording-order blocking reduces but does not remove train-test overlap; decoding cost and program-loss weight trade extra-object errors against plan success, and better evidence is needed to reduce spurious events without losing true ones.Source: §7 · p. 19
8. Conclusion
The paper introduces instance-anchored interaction evidence to bind demonstration interactions to the object instances a robot will manipulate. On WatchAct, it improves plan success over a 32B VLM across implicit-intent and episodic tasks, while pointing errors and depth-ambiguous destinations remain.
8. Conclusion
- IAE binds demonstration interactions to the object instances a robot will manipulate through scene registration, backward tracking, hand geometry, outcome-trained evidence, and grammar-constrained decoding.The approach turns pointing gestures into object–destination plans and uses symbolic programs for reference disambiguation and episodic tasks.Source: §8 · p. 19
- 64.2% plan success on implicit-intent tasks versus 27.5% for a 32B VLM, and 46.4% versus 27.0% on restoration, reversal, and imitation without task-specific training.Strict success on implicit-intent tasks is 49.7% versus 15.4%; plans are scored by symbolic execution.Source: §8 · p. 19
- Suppressing spurious object events and resolving depth-ambiguous destinations are the main remaining steps.Spurious object events account for most strict failures on pointing tasks.Source: §8 · p. 19
CRediT authorship contribution statement
The contribution statement assigns conceptualization, methodology, software, investigation, analysis, visualization, validation, data curation, supervision, funding, project administration, and writing across the named authors.
CRediT authorship contribution statement
- Xinliang Xiao is credited with conceptualization, methodology, software, investigation, formal analysis, visualization, and the original draft.Source: CRediT authorship contribution statement · p. 19
- Bowen Yang is credited with methodology, software, validation, data curation, investigation, and review and editing; Wenjing Zhang with validation, visualization, and review and editing.Source: CRediT authorship contribution statement · p. 19
- Li Yang is credited with conceptualization, validation, supervision, funding acquisition, and review and editing; Wei Zhou with conceptualization, methodology, supervision, project administration, funding acquisition, and review and editing.Source: CRediT authorship contribution statement · p. 19
Funding
The work acknowledges support from four Jiangsu and national science-funding programs.
Funding
- Support came from the Jiangsu Provincial Major Science and Technology Program (No. BG2025043).Source: Funding · p. 20
- Support came from the National Natural Science Foundation of China (No. 52502402).Source: Funding · p. 20
- Support came from the Basic Research Program of Jiangsu Province (No. BK20251462) and the Natural Science Foundation of Jiangsu Province (No. BK20232038).Source: Funding · p. 20
A. Prompts of the VLM baselines
The VLM baselines receive timestamped frames, an instruction, and public scene information, then directly return object–destination plans. Variants supply tracked-object overlays or structured interaction evidence, with score semantics specified for some prompts.
A. Prompts of the VLM baselines
- VLM baselines decode greedily from timestamped frames, an instruction without the final-scene prefix, and public scene data containing objects, states, regions, and descriptions.Predicted regions are mapped to the request’s canonical spatial frame before scoring.Source: A. Prompts of the VLM baselines · p. 20
- Direct-planning prompts ask for JSON object–destination pairs from the final scene, with episodic tasks requiring every moved object and overlay variants showing tracked IDs and hand skeletons.The ordinary prompt allows one to three pairs; the episodic version allows one to eight.Source: A. Prompts of the VLM baselines · p. 20
- Evidence prompts provide tracked interaction scores, with NC listing candidate events in temporal order and RD listing instance and container scores plus container distances.The explained-semantics variant adds guidance on how to read the evidence.Source: A. Prompts of the VLM baselines · p. 20
- Interaction scores indicate whether an instance was touched, pointed at, or handled, not whether it must move; instructions distinguish selected targets from “other” objects by high versus low scores.Destination-verification prompts ask the model to use finger and arm direction rather than fingertip proximity alone.Source: A. Prompts of the VLM baselines · pp. 20–21
B. Relation model
The same-supervision relation model provides a learned comparison using hand–candidate features and temporal object–destination scoring. It is trained on goal-operation labels and selects operations by thresholding each object’s best destination score.
B. Relation model
- The same-supervision relation model embeds hand–candidate features, combines object–destination sequences with temporal convolutions and pooling, and outputs operation scores.It uses a shared two-layer perceptron for hand–candidate features, then two temporal convolutions followed by maximum and mean pooling.Source: B. Relation model · p. 21
- The model is trained on NC training-fold videos with class-balanced binary cross-entropy, labeling goal operations as positive pairs.Training uses Adam for 150 full-batch epochs.Source: B. Relation model · p. 21
- At test time, each object takes its highest-scoring destination, and a fold-specific threshold keeps between one and three objects.The threshold is selected on training videos by matching predicted operations to the goal.Source: B. Relation model · p. 21