PaperlayerView exact revision on arXiv

Source-linked AI summary

GraphPoint: Semantic Entity Graphs and Point Trajectories for Compositional Robot Manipulation

Kang Luo, Hesheng Wang

arXiv:2609.18358v2cs.RO

TL;DR

Robot policies may rely on visual-action correlations rather than follow instructions when familiar objects are paired with new behaviors. CoMani evaluates compositional reuse, and GraphPoint connects role-structured semantic graphs to point-trajectory control. On CoMani, GraphPoint generalizes across atomic interactions and unseen multi-step instructions.

  • Problem

    Policies may learn fixed visual-action mappings, leaving open how they will respond when familiar entities are paired with new action types or modifiers and when learned subtasks must be reused in long-horizon tasks.

    Source: §I · p. 1

  • Method

    CoMani controls semantic recombination and subtask-reuse splits, while GraphPoint uses role-specific entity graphs to condition point-trajectory control and predicted progress to support subtask transitions.

    Source: §I · pp. 1–2, D. Role-Structured Entity Graph Encoder · p. 3, F. Subtask Progress and Long-Horizon… · p. 4

  • Results

    GraphPoint is the most accurate method on seen and held-out modifiers and the only method above 60% on every held-out task; experiments also report generalization across atomic and sequential tasks.

    Source: B. Can GraphPoint Generalize Across… · p. 7, §VI · p. 8

  • Takeaways & Limitations

    CoMani and GraphPoint support evaluation and study of instruction-dependent generalization within subtasks and across unseen long-horizon tasks.

    Source: §I · p. 1, §VI · p. 8

Abstract

from arXiv · show

Robot manipulation policies often struggle to generalize beyond their demonstrations, even when new instructions involve familiar objects and behaviors. When language and scenes are strongly correlated during training, a policy can learn a fixed visual-action mapping rather than respond to the requested behavior. We investigate compositional reuse at two levels: within a subtask, combining familiar entities, action types, and action modifiers; and across subtasks, reusing learned subtasks in unseen long-horizon tasks. We introduce CoMani, a benchmark with controlled splits for evaluating both capabilities. Matched initial scenes and controlled changes to a single semantic factor encourage reliance on language rather than visual shortcuts. We further propose GraphPoint, which connects semantic entity graphs to geometric control by predicting future gripper point trajectories and converting them into actions using robot geometry. The framework organizes the gripper and objects by semantic roles and conditions their interactions on action types and modifiers, while predicted progress guides transitions during execution. Experiments and ablations on CoMani validate the effectiveness of our method for instruction-dependent generalization at both levels. Code will be released at GraphPoint.

I. INTRODUCTION

Manipulation policies may learn fixed visual-action mappings instead of responding to instructions, limiting reuse of familiar behaviors within subtasks and across unseen long-horizon tasks. CoMani tests both forms of composition, while GraphPoint grounds instructions in role-specific entity graphs and point-trajectory control.

I. INTRODUCTION

  • Familiar objects paired with a new action type or modifier expose policies that rely on visual shortcuts rather than language-responsive behavior.The challenge grows in long-horizon execution, where each subtask begins from the state produced by the preceding one.Source: §I · p. 1
  • CoMani varies one semantic factor under matched initial scenes and tests both within-interaction recombination and reuse of atomic subtasks in unseen long-horizon instructions.Its controlled splits make the instruction necessary for selecting the intended behavior.Source: §I · pp. 1–2
  • GraphPoint represents the gripper, manipulated object, and reference entity as role-specific graph nodes, with action types and modifiers conditioning geometric information exchange.Observed entity points and predicted gripper trajectories are TCP-relative, and robot geometry recovers executable poses.Source: §I · pp. 1–2
  • During long-horizon execution, GraphPoint reuses the same atomic policy across subtasks and uses predicted progress to decide when to advance.This supports recombining learned subtasks into instructions never demonstrated as a whole.Source: §I · p. 2
  • Experiments on CoMani report strong gains over diverse image-, scene-point-, entity-point-, and graph-based baselines, with ablations supporting the framework design.Source: §I · p. 2

II. RELATED WORK

Prior work has used image, scene-point, entity-point, and graph representations for manipulation, but GraphPoint distinguishes entities by manipulation roles and conditions their interactions on action semantics.

A. Visual and Scene-Point Representations

  • Image and scene-point methods capture visual or geometric structure, while GraphPoint represents task-relevant objects as separate role-specific entities.The cited approaches include ACT, Diffusion Policy, VLA models, PerAct, RVT, and DP3.Source: A. Visual and Scene-Point… · p. 2

B. Entity-Centric and Graph Representations

  • Entity-centric methods use points or graphs, but GraphPoint couples role-specific entity graphs with action-type- and modifier-conditioned point-trajectory control.Point Policy, Point Bridge [9], GraphMimic, and CbF [10] are discussed as related approaches.Source: B. Entity-Centric and Graph… · p. 2

C. Compositional and Long-Horizon Benchmarks

CoMani addresses gaps in compositional and long-horizon evaluation, while GraphPoint constructs language-conditioned entity graphs for manipulation and subtask execution.

C. Compositional and Long-Horizon Benchmarks

  • CoMani isolates a single semantic change under matched initial scenes and tests composition of separately recorded atomic subtasks into unseen long-horizon tasks.The cited benchmark settings do not combine these controls: VIMA, CALVIN, LIBERO-Long, and VLABench.Source: C. Compositional and Long-Horizon… · p. 2

A. Overview

  • GraphPoint processes a language-conditioned entity graph to predict gripper trajectories and subtask progress, which determines when execution advances.The graph is instantiated for each subtask in a multi-step instruction.Source: A. Overview · p. 2, Figure 2 · p. 3

B. Task-to-Graph Construction

  • An LLM decomposes instructions into ordered subtasks with actor, patient, and target roles, action types, and modifiers; entity descriptions support visual grounding.A locator identifies patient and target objects or parts, and a tracker propagates their masks across frames for point sampling.Source: B. Task-to-Graph Construction · p. 2

C. Semantic Conditioning

GraphPoint conditions local entity encoding on semantic roles and global interaction on action type and modifier. Its role-structured encoder uses TCP-relative geometry and exchanges entity summaries through a constrained attention pattern.

C. Semantic Conditioning

  • Role embeddings condition local entity blocks, while encoded action types and modifiers condition global blocks, the flow decoder, and the progress head.These conditions let the same geometry be interpreted according to the instructed interaction.Source: C. Semantic Conditioning · p. 3

D. Role-Structured Entity Graph Encoder

  • Entity points share the current-frame TCP as their origin, reducing dependence on absolute scene placement while keeping camera axes.The encoder alternates entity-local processing with global attention over summaries; actor-target interaction is mediated through the patient along A ↔ P ↔ T.Source: D. Role-Structured Entity Graph Encoder · p. 3
  • Role-aware point collapse independently simplifies patient or target points with probability 0.05 to suppress shape-based shortcuts and encourage reliance on semantic conditions.Patient points become the mean of the four nearest to the TCP; target points become the entity centroid.Source: D. Role-Structured Entity Graph Encoder · pp. 3–4

E. Point-Trajectory Flow Policy

The policy predicts future gripper trajectories as point sequences and uses conditional flow matching to model them. Robot geometry then converts the predicted points into executable gripper poses.

E. Point-Trajectory Flow Policy

  • Each of ten predicted future steps contains six actor keypoints and a scalar gripper command, expressed as a 19-dimensional vector.The actor coordinates use the TCP-relative frame; arranging multiple points also encodes orientation.Source: E. Point-Trajectory Flow Policy · p. 4
  • A six-layer Transformer embeds observed actor history with a noisy future trajectory and predicts a velocity field using conditional flow matching.Decoder layers use self-attention across trajectory steps and cross-attention to retrieve entity geometry.Source: E. Point-Trajectory Flow Policy · p. 4
  • At inference, ten Euler steps integrate from noise into a trajectory, whose six keypoints are aligned to robot-geometry reference points to recover a gripper pose command.The subtracted TCP translation is added back before least-squares rigid alignment.Source: E. Point-Trajectory Flow Policy · p. 4
  • The policy runs at 7.8 Hz for a 10-action chunk, while the full pipeline including locator and tracker runs at 2.1 Hz.Source: E. Point-Trajectory Flow Policy · p. 4

F. Subtask Progress and Long-Horizon Composition

A progress head estimates subtask completion from the patient–target relation, and predicted progress governs transitions during long-horizon execution. The same atomic policy is reused across the sequence.

F. Subtask Progress and Long-Horizon Composition

  • A language-conditioned progress head estimates subtask completion from the patient–target relation.For actions without an explicit target, it uses the patient summary from the initial frame as a reference for state change.Source: F. Subtask Progress and Long-Horizon… · p. 4
  • The planner advances under self-switching when predicted progress exceeds 0.9 for two consecutive policy calls.Environment-based switching instead uses an environment-provided completion signal to separate execution performance from learned transition timing.Source: F. Subtask Progress and Long-Horizon… · p. 4
  • At each transition, the system releases and lifts the gripper, selects the next subtask’s entities and language conditions, resets policy history, and reuses the same atomic policy.Source: F. Subtask Progress and Long-Horizon… · p. 4

IV. COMANI BENCHMARK

CoMani evaluates semantic recombination within atomic interactions and reuse across unseen sequences, using matched initial scenes wherever feasible. Its three suites control which semantic factor changes and whether sequence-level demonstrations are available.

IV. COMANI BENCHMARK

  • CoMani has modifier, action-type, and sequential-composition suites; its task lists and ID/OOD splits are organized in Table I.The benchmark uses 50 trajectories for each atomic task, generated using MimicGen.Source: §IV · p. 4, Table I · p. 5
  • CoMani-Mod fixes put while varying modifiers across 18 tasks, with 12 ID and 6 OOD tasks that recombine familiar object–target pairs with the opposite modifier.Matched scenes make the instructed relation the distinguishing factor; the suite reports success rate.Source: §IV · p. 4
  • CoMani-Act fixes right while varying put, sweep, and rotate across 10 tasks, including held-out action combinations and rotation on an entity not rotated during training.Its action-consistent success rate requires both the final goal and the instructed action type; reaching the same position through another action does not count.Source: §IV · p. 4
  • CoMani-Seq tests three-step instructions using policies trained only on CoMani-Mod atomic demonstrations, with six fixed-order sequences and no sequence-level demonstrations.Pairs vary one modifier; the all-ID pair tests familiar composition, while other pairs include held-out atomic combinations.Source: §IV · p. 4, Figure 3 · p. 5

V. EXPERIMENTS

The experiments are organized around whether GraphPoint generalizes across modifiers and actions, composes atomic subtasks, and which components matter. The supplied table captions identify the corresponding metric and benchmark scope, but provide no outcome values.

V. EXPERIMENTS

  • The evaluation asks whether GraphPoint generalizes across modifiers, generalizes across actions, composes atomic subtasks, and depends on particular components.Source: §V · p. 5
  • The supplied result-table captions cover CoMani-Mod success rate, CoMani-Act action-consistent success rate, and CoMani-Seq SR2, SR3, and progress rate.The sequential table distinguishes environment-based switching with oracle goal signals from self-switching with predicted progress.Source: Table IV · p. 6, Table III · p. 6, Table II · p. 6
  • A separate CoMani-Mod ablation table reports success rate, with task indices following Table I.Source: Table V · p. 6

A. Experimental Setup

The experiments compare GraphPoint with representation-family baselines under shared CoMani scenes and evaluate both task performance and sequential execution. Training and rollout protocols are specified, and Fig. 4 illustrates successful self-switching sequences.

Baselines and adaptation

  • Baselines retain native language conditioning or receive a frozen full-instruction embedding, with π0.5 fine-tuned using LoRA.ACT, DP, DP3, and Point Policy receive architecture-specific language inputs; CoMani-Seq reuses atomic policies without sequence-level training under a shared environment-based switching protocol, while GraphPoint is also evaluated with self-switching.Source: A. Experimental Setup · p. 6

Training and evaluation

  • All models train for 30,000 steps with batch size 64; evaluations use shared scene initializations and 20 or 30 trials depending on the suite.CoMani-Mod and CoMani-Act use 30 trials per task, while CoMani-Seq uses 20 trials per sequence; OOD tasks are excluded from training and no EMA is used.Source: A. Experimental Setup · pp. 6–7
  • Fig. 4 shows successful GraphPoint self-switching rollouts for sequences containing 0, 1, and 2 OOD subtasks.Each row proceeds left to right, with three frames per subtask marking its start, execution, and completion; upper-right values show predicted subtask progress.Source: Figure 4 · p. 7

B. Can GraphPoint Generalize Across Modifiers?

GraphPoint generalizes across held-out modifiers and actions, and reuses atomic policies to complete unseen three-step instructions. Its learned self-switching matches environment-based switching in progress rate while trading some full-sequence success.

B. Can GraphPoint Generalize Across Modifiers?

  • GraphPoint is the only method above 60% on every held-out modifier task and is most accurate on both seen and held-out modifiers.Baselines either only partially acquire seen modifiers or lose seen-level performance on held-out modifiers; GraphPoint retains a high held-out success floor, though a residual gap remains.Source: B. Can GraphPoint Generalize Across… · p. 7

C. Can GraphPoint Generalize Across Actions?

  • GraphPoint achieves the highest held-out ACSR and is the only method with action-consistent success on rotation of an entity never previously rotated.DP3, the strongest baseline on action recombinations, records no success on that rotation task; GraphPoint also handles recombination of a familiar object–target pair under the other action.Source: C. Can GraphPoint Generalize Across… · p. 7

D. Can GraphPoint Compose Atomic Subtasks?

  • 82.5% SR2 and 50.8% SR3: GraphPoint completes unseen three-step instructions with environment-based switching, while every baseline records 0.0% SR2.Self-switching advances on predicted progress without an environment goal signal and matches environment-based switching in progress rate, while trading some full-sequence success.Source: D. Can GraphPoint Compose Atomic… · p. 7

E. Which Components Matter?

Ablations test GraphPoint’s graph structure, language conditioning, and coordinate and output representations. The conclusion reports instruction-dependent generalization across atomic and sequential tasks, while identifying perception errors and severe occlusion as limitations.

E. Which Components Matter?

  • Removing RPC or replacing chain connectivity with full connectivity preserves seen-modifier performance but lowers held-out performance; merging entity points is the most damaging structural variant.The merged-set variant removes role embeddings, local entity groups, and chain connectivity together, so it measures the structured encoder rather than one component.Source: E. Which Components Matter? · p. 7
  • Removing action-type and modifier conditions lowers held-out SR from 80.0% to 8.3%, while full-instruction conditioning without factor decomposition reaches 77.2%.The results support explicit instruction conditioning while showing that strong generalization does not require explicit action/modifier decomposition.Source: E. Which Components Matter? · pp. 7–8
  • Absolute point coordinates preserve seen-modifier performance but sharply reduce held-out performance, while point trajectories and Cartesian delta actions perform comparably.Point trajectories are retained because they share the input’s TCP-relative geometry and can be extracted from human videos, supporting transfer from human demonstrations.Source: E. Which Components Matter? · p. 8

VI. CONCLUSION

  • CoMani and GraphPoint support instruction-dependent generalization at atomic and sequential levels, with ablations supporting entity separation, chain-mediated interaction, language conditioning, and relative coordinates.The conclusion identifies perception parsing, localization, and segmentation errors, plus severe occlusion disrupting tracked entity points, as limitations.Source: §VI · p. 8

Related papers

Loading 2609.18358v2…