Source-linked AI summary
Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulation
Jianxiang Liu, Gaojing Zhang, Chuan Wen, Qipeng Liu, Yuxuan Zhao, Ning Guo, Wenzhao Lian
TL;DR
Long-horizon manipulation needs reliable plans despite changing observations, while existing VLA and hierarchical systems remain limited by data demands, opacity, weak grounding, or open-loop execution. TTS uses instance-grounded triplets for planning, continuous tracks for execution, and observation-based replanning; across seven settings, it reaches 74.8% average success.
Problem
Long-horizon manipulation remains difficult because end-to-end VLAs are data-heavy and opaque, while hierarchical plans can be weakly grounded and lack automatic error detection and recovery.
Method
TTS uses human videos to build instance-grounded triplet subgoals, translates triplet transitions into continuous track priors, and monitors observations for closed-loop replanning.
Results
74.8% average success rate across seven settings, compared with 30.4% for the best-performing baseline, SeeDo.
Takeaways & Limitations
TTS supports reliable long-horizon execution under limited data together with object-level and compositional generalization.
Abstract
from arXiv · showhide
Ensuring reliability in uncertain environments remains difficult for long-horizon robotic manipulation. End-to-end VLA models are data-heavy and opaque, making diagnosis and verification difficult. Hierarchical pipelines are more interpretable, but their plans are often weakly grounded in observations, weakly aligned with low-level actions, and computed without online feedback, leading to open-loop behavior and hallucinations. To address these issues, we introduce the Triplet-to-Track System (TTS), a closed-loop long-horizon imitation learning system that uses human videos to reduce reliance on robot-collected data. TTS represents high-level subgoals as instance-grounded triplets, translates them into continuous track priors for execution, and monitors task progress from observations for online replanning. Across diverse real-world long-horizon tasks, TTS achieves a 74.8\% average success rate and supports object-level and compositional generalization.
I. INTRODUCTION
Long-horizon manipulation requires plans that remain consistent with changing observations, while existing end-to-end and hierarchical approaches have complementary reliability and grounding limitations. TTS addresses these gaps with instance-grounded triplets, continuous track priors, and observation-based monitoring and replanning.
- Long-horizon tasks require plans that remain correct as observations evolve across intermediate interactions.
- End-to-end VLA systems are data-hungry and difficult to interpret or diagnose because perception, planning, and control are internalized in one network.
- Hierarchical methods reduce robot-data requirements but can produce weakly constrained goals, hallucinated objects or relations, and weak grounding in concrete observations.
- TTS represents subgoals as instance-grounded triplets and maps their transitions to continuous tracks that low-level controllers can consume.
- The system combines human videos, observation-based monitoring, and replanning to improve execution reliability and support verification.
II. RELATED WORK
Related work spans end-to-end VLA systems, hierarchical pipelines, semantic task decompositions, and video-based learning. These approaches improve scalability, interpretability, or data efficiency but remain limited by opacity, hallucination, weak instance grounding, interface mismatch, or inadequate temporal control.
- End-to-end VLA models map pixels and language directly to actions but remain opaque, data-hungry, and vulnerable to distribution shift.
- Hierarchical pipelines improve interpretability by separating planning and control, yet semantic sketches can hallucinate and mismatch low-level controllers.
- Semantic or relational decompositions improve data efficiency and modularity but often lack instance-level spatial grounding needed for concrete verification.
- Periodic replanning or feedback does not guarantee robust closed-loop behavior when predicates are not directly checkable from perception.
- Video-based learning methods use inverse-dynamics labels, feature pretraining, or video prediction, but face brittleness, weak temporal control, hallucinated geometry, or high inference cost.
III. PRELIMINARIES
The formulation separates high-level goal specification from low-level execution using object-relation triplets and entity-aligned motion tracks. A task planner selects subgoals, a track predictor produces continuous priors, and a policy converts them into actions.
- The task is modeled as a contextual POMDP in which observations arise from hidden states and actions induce probabilistic state transitions.
- Triplets c = (x, r, y) bind a relation from a predefined vocabulary to an ordered pair of concrete object instances.
- The function hold(ot, c) indicates whether a triplet relation currently holds in the observed scene.
- Track descriptors parameterize temporally extended, entity-aligned motion for low-level control.
- The Task Planner proposes the next triplet, the Track Predictor maps the transition to a continuous track, and the low-level policy generates actions from that prior.
A. Task Planner for discrete, abstract triplets
The task planner combines relation prediction with graph-constrained triplet selection. Temporal graphs extracted from human videos restrict feasible transitions, while guarded verification and event-triggered checking support responsive replanning with fewer false positives.
- Task Planner for discrete, abstract triplets: The planner uses a Relation Predictor and a Graph-Guarded Next Triplet Generator to infer current relations and propose the next discrete subgoal.
- Task Planner for discrete, abstract triplets: The Relation Predictor combines a lightweight Transformer relation classifier with a LoRA-tuned VLM branch to improve reliability without requiring a universal zero-shot detector.
- Task Planner for discrete, abstract triplets: The LTP predicts a relation for each ordered detected-object pair from a predefined relation vocabulary.
- Task Planner for discrete, abstract triplets: Spurious one-frame relation flips are the dominant LTP error when relations should remain stable over short horizons.
- Task Planner for discrete, abstract triplets: With observed per-frame accuracy p ∈ (0.80, 0.90) and change rate ρ ≈ 0.02, the analysis predicts that spurious flips dominate missing changes.
- Task Planner for discrete, abstract triplets: A high-recall detector invokes a high-precision verifier only when a change is suspected, reducing false positives for stable context triplets.
- Task Planner for discrete, abstract triplets: The temporal graph is extracted from human-video transitions and restricts proposals to observed feasible neighbors, suppressing hallucinated transitions.
- Task Planner for discrete, abstract triplets: Guard evaluation triggers replanning from the initial node when validated relations no longer hold.
B. Track Predictor for continuous tracks
The Track Predictor converts discrete, instance-grounded triplet transitions into continuous, entity-aligned tracks for low-level execution. Triplet-transition conditioning and auxiliary objectives provide spatial and relational supervision for generalization and stability.
- Track Predictor for continuous tracks: The predictor conditions continuous track forecasts on explicit triplet transitions rather than inferring motion solely from local cues.This ties predicted tracks to object pairs and relation changes, while supporting pretraining on action-free videos.
- Track Predictor for continuous tracks: Training samples combine video observations, instance masks, tracked trajectories, and an encoded relation change r→r′.Masks localize the subject and object, while CoTracker produces entity-aligned trajectories for the training sequence.
- Track Predictor for continuous tracks: The model predicts future point positions from observations, current points, instance masks, and a relation-change embedding, then decodes continuous tracks.The fused representation provides relation-dependent motion supervision for realizing discrete updates as motion priors.
- Track Predictor for continuous tracks: Two auxiliary objectives improve stability: masked-patch reconstruction and relation-change recognition.The relation-change objective predicts δr from object and subject features with text-decoder tokens, strengthening relation-change modeling.
C. Policy Learning for Action
The Action Policy maps predicted track priors and relational context to robot actions using a Transformer-based track-guided behavior-cloning policy. It executes in a receding-horizon loop with observation checks, subgoal updates, and replanning after guard violations.
- Policy Learning for Action: The policy is trained on a small set of action-labeled robot data to map predicted track priors into the robot action space.This is implemented as track-guided behavior cloning.
- Policy Learning for Action: The low-level policy fuses observation features, triplet-transition priors, and track priors before decoding the current robot action.A Transformer backbone with a learnable action token processes these inputs, followed by an action head.
- Policy Learning for Action: Every 5 steps, the Relation Predictor checks whether the active subgoal has been achieved and triggers the next triplet when it is complete.The policy computes a short-horizon track prior from two camera views before predicting the current action.
- Policy Learning for Action: A guard violation causes the planner to relocalize to the graph’s initial node and replan before continuing.Actions are executed while observations update and the prediction horizon recedes.
V. EXPERIMENTS
The experiments test TTS across performance and generalization, reliability and stability, and the necessity of its architectural components. The supplied passages define these evaluation questions but do not report their results.
- V. EXPERIMENTS: Q1 evaluates real-world long-horizon performance and object-level and compositional generalization against representative baselines.The question emphasizes maintaining high success rates across these evaluation settings.
- V. EXPERIMENTS: Q2 evaluates closed-loop, disturbance-robust execution using observation-feedback replanning and structured, traceable goals.It also asks whether the representation reduces hallucination and preserves execution traceability.
- V. EXPERIMENTS: Q3 evaluates whether the Triplet-to-Track components are necessary and what each contributes.This question provides the basis for component-level design justification.
A. Experimental Setup
The study evaluates language-conditioned manipulation on a physical 7-DoF Franka robot across three task families probing replanning, ordering, and object-level generalization. Training uses robot and human demonstrations, while test episodes span long multi-step horizons.
- A. Experimental Setup: Three task families probe distinct capabilities: Dynamic Scale Balancing tests online replanning, Sequential Stacking tests ordering, and Diverse Pick-and-Place tests object-level generalization.Each family targets a different aspect of the system on the physical robot.
- A. Experimental Setup: Dynamic Scale Balancing replans after each pick-and-place step when the scale remains unbalanced.This evaluates closed-loop planning under evolving observations.
- A. Experimental Setup: Sequential Stacking tests unseen orderings, while Diverse Pick-and-Place substitutes object categories or attributes at test time.These settings evaluate compositional and cross-category generalization, respectively.
- A. Experimental Setup: Evaluated episodes involve up to nine, eleven, and twelve subgoals for DPP, SS, and DSB, respectively.Each task uses 30 robot demonstrations and 50 human demonstrations, with five trajectory annotations for each video type.
- A. Experimental Setup: Table I reports success rates on long-horizon tasks as percentages.The supplied table caption identifies the metric but provides no task-by-task values.
- A. Experimental Setup: The Action Policy runs at 15 Hz, with relation checks every five steps, using third-person and eye-in-hand camera views.The hardware setup uses two 1280 × 720 Intel RealSense D435i cameras and a 7-DoF Franka arm.
B. Baselines and Results
TTS is evaluated against end-to-end and hierarchical baselines across seven settings, achieving higher average success while addressing spatial hallucination, logical misordering, and multi-step tracking failures.
- TTS achieves a 74.8% average success rate across seven settings, compared with 30.4% for the best-performing baseline, SeeDo.
- Baseline failures: Sequential stacking failures chiefly involve logical misordering and drift in multi-step goal tracking, including unnecessary grasps and incorrect execution order.
- Baseline failures: Dynamic scale balancing exposes instability when the correct next action depends on evolving physical state after each pick-and-place step.
- Baseline failures: FT-Pi0.5 often hesitates or misselects visually similar objects under the task-specific low-data adaptation setting.
C. Reliability and Stability
The evaluation examines perturbation robustness, interpretability, hallucination suppression, architectural components, and action-labeled data scale. The results connect explicit triplets with planning reliability and tracks with stable execution.
- TTS is evaluated through controlled perturbation tests and qualitative analyses of interpretability and hallucination suppression.
- Closed-Loop Robustness under Perturbations: Perturbations include adding or removing weights during Dynamic scale balancing and removing a target after grasping during Diverse pick-and-place.
- Triplets and Tracks for Interpretability and Hallucination Suppression: Explicit triplets and tracks make failures traceable to planning or control, while replacing validatable triplets with semantic VLM plans causes performance drops and hallucinated transitions.
- Triplets and Tracks for Interpretability and Hallucination Suppression: Triplets with temporal-graph constraints restrict proposals to observation-consistent, feasible updates, helping suppress hallucinated transitions.
- Effect of Task Planner Components: Removing the temporal graph increases next-triplet errors by increasing hallucinations and dependency-tracking failures, whereas graph constraints limit transitions to feasible updates observed in human data.
- Effect of Hierarchical Architecture: The hierarchy is essential: removing the Task Planner collapses long-horizon behavior, while removing the Track Predictor substantially degrades the discrete-to-continuous control interface.
- Effect of Action-Labeled Data Size: TTS achieves reasonable performance with only a few dozen action-labeled demonstrations, supporting its data efficiency.
VI. CONCLUSIONS
TTS combines discrete triplets for planning with continuous tracks for execution, aiming to improve reliability and generalization under limited data.
- TTS uses a structured, instance-spatially grounded design to reduce hallucinations, improve long-horizon reliability under limited data, and support object-level and compositional generalization.Limitations include out-of-graph failures, perception errors, and non-triplet tasks such as deformable-object manipulation.