PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-to-3D Policy: Fine-Grained Language-Behavior Alignment for Unseen Specification Generalization

Xinhao Yang, Wenhao Wu, Ning Lv, Yanshen Ding, Zhenhong Sun, Daoyi Dong, Chunlin Chen, Zhi Wang

arXiv:2609.39599v1cs.ROcs.AI

TL;DR

Text-to-3D policies struggle to execute unseen fine-grained behavioral specifications, and global language-behavior alignment can blur nearby specifications. T3DP aligns instruction tokens with local behavior representations to condition a 3D diffusion policy. It improves held-out-specification success across simulation benchmarks and real-robot tasks.

  • Problem

    Text-to-3D policies face the open challenge of following behaviorally precise instructions beyond the demonstrations observed during training.

    Source: §1 · p. 2

  • Method

    T3DP learns specification-sensitive language representations through bidirectional token-level alignment between instruction elements and local behavior representations, then conditions a point-cloud-based 3D diffusion policy on them.

    Source: §1 · p. 2

  • Results

    T3DP improves average held-out-specification success over global alignment by 14.2, 13.5, and 11.0 points on Meta-World, ManiSkill, and RoboTwin, with gains on all 15 task families.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Representation and action-probe analyses show that fine-grained alignment better preserves specification geometry and action variation.

    Source: §1 · p. 3

    The approach requires expert trajectories with reward signals, and evaluation is limited to held-out specifications within known task families under templated instructions.

    Source: §6 · p. 10, F LIMITATIONS · p. 24

Abstract

from arXiv · show

3D visuomotor policies provide a strong foundation for spatially precise manipulation, yet current text-to-3D policies struggle to follow unseen fine-grained behavioral specifications beyond those covered by demonstrations. We study this challenge as unseen specification generalization, where language specifies behaviorally significant variations, such as target position, displacement, or articulated state, that are absent from policy training. We find that pretrained language representations and conventional global behavior-language alignment capture coarse task semantics but often blur nearby specifications that require distinct behaviors. We introduce T3DP, a Text-to-3D Policy framework for fine-grained language-behavior alignment. Rather than compressing each instruction and demonstration into a single global embedding, T3DP preserves their local structures and establishes bidirectional token-level correspondence between linguistic elements and behavioral segments. This directly grounds subtle linguistic variations in the behavior components they affect, preventing closely related specifications from collapsing in the representation space. The resulting specification-sensitive language representation conditions a point-cloud-based 3D diffusion policy, enabling more precise control over unseen behavioral specifications without modifying the underlying policy architecture. Across Meta-World, ManiSkill, and RoboTwin, T3DP improves average held-out-specification success over global language-behavior alignment by +11.0-14.2 points, with gains on all 15 task families; on real-robot tasks, it further raises average success from 47.5% to 65.0% (+17.5 points). Representation and action-probe analyses show that fine-grained alignment better preserves specification geometry and action-relevant variation, linking local behavior grounding to downstream control.

1 INTRODUCTION

The paper studies whether text-to-3D policies can execute held-out behavioral specifications within known task families. T3DP aligns instruction tokens with local behavior representations, improving held-out-specification success across simulation and real-robot tasks.

1 INTRODUCTION

  • Unseen specification generalization asks whether a policy trained on some language-specified behaviors can execute held-out specifications without adaptation.Specifications within a task family can vary in target position, displacement, or articulated state, and exhaustive demonstrations are impractical.Source: §1 · p. 2
  • Global alignment can blur nearby specifications because whole-instruction and demonstration embeddings emphasize task- or trajectory-level compatibility over local distinctions.Pretrained language representations also need not preserve how subtle linguistic variations correspond to physical behavior.Source: §1 · p. 2
  • +14.2, 13.5, and 11.0 points in average success over global alignment on Meta-World, ManiSkill, and RoboTwin, respectively, with gains across all 15 task families.On an AgileX PiperX robot, average success rises from 47.5% to 65.0%.Source: §1 · p. 2
  • T3DP establishes bidirectional token-level correspondence between instruction elements and local behavior representations, then uses the resulting language representation to condition a 3D diffusion policy.The framework preserves local structures and does not redesign the underlying policy architecture.Source: §1 · p. 2

2 RELATED WORK

Prior work studies language-conditioned 3D manipulation and several forms of language-behavior alignment. This paper focuses on grounding fine-grained, within-task behavioral variations for downstream control.

Language-Conditioned 3D Visuomotor Policies

  • Language-conditioned 3D policies use linguistic descriptions to condition action prediction across diverse tasks, while this work targets distinct behaviors sharing the same task semantics.Examples of prior policies include PerAct, Act3D, 3DDA, and 3D-LOTUS.Source: §2 · p. 3

Generalization to Language-Specified Behaviors

  • Prior generalization studies cover unseen tasks, objects, scenes, linguistic variations, and continuous goal states; this work also considers spatial specifications such as target positions and relative displacements.The focus is held-out specifications within the same task family.Source: §2 · p. 3

Language-Behavior Alignment

  • Robot language-behavior alignment methods largely operate on tasks, transitions, or aggregated trajectories, leaving local grounding of specification-bearing language less explored.T3DP addresses this gap with local language-behavior correspondence for specification-dependent behavioral variation.Source: §2 · p. 3

3 PRELIMINARY STUDY

The preliminary study tests whether language representations preserve information needed to distinguish held-out behavioral specifications. It finds that explicit visual target cues remain more effective than language alone, while fine-grained alignment better preserves specification geometry than global alignment.

3 PRELIMINARY STUDY

  • Removing the visual target cue drops Meta-World Reach success to 12.7%, while removing the language cue leaves success essentially unchanged at 54.7% versus 55.0%.The study reports increased prediction error when the visual cue is removed.Source: Table 1 · p. 3, §3 · p. 3
  • Fine-grained alignment improves both global ordering and local neighborhood preservation of held-out Reach, Push, and Door Open specifications beyond global alignment.Global alignment improves specification geometry over generic language representations, but still obscures distinctions among nearby specifications.Source: Table 1 · p. 3, §3 · pp. 3–4

4 METHOD

T3DP addresses held-out specifications with reward-predictive behavior representations, fine-grained language-behavior alignment, and a language-conditioned 3D diffusion policy.

4 METHOD

  • T3DP uses three stages: learn behavior representations from demonstrations, align text with them, then train a language-conditioned 3D diffusion policy.The aligned text encoder is frozen during policy training.Source: §4 · p. 4

4.1 PROBLEM FORMULATION

  • At test time, the policy receives only language for a held-out specification from the same task family and must generalize zero-shot.Specifications in a family share state and action spaces and transition dynamics, but differ in reward.Source: §4.1 · p. 4

4.2 BEHAVIOR REPRESENTATION LEARNING FROM DEMONSTRATIONS

  • A reward-predictive objective trains trajectory-segment embeddings to capture specification-dependent behavior by predicting rewards for state-action pairs outside the encoded segment.The target reward is excluded from the encoded segment during training.Source: §4.2 · p. 5

4.3 FINE-GRAINED LANGUAGE-BEHAVIOR ALIGNMENT

  • Because global pooling can obscure decisive specification differences, T3DP retains local representations and bidirectionally matches behavior states and actions with instruction tokens.The two directional scores softly match segments to relevant tokens and tokens to the segments expressing their behavioral meaning.Source: §4.3 · pp. 5–6

4.4 LANGUAGE-GUIDED 3D POLICY LEARNING

  • A point-cloud 3D diffusion policy conditions action-sequence denoising on observations, proprioception, and aligned language to execute held-out specifications without demonstrations or further adaptation.Specification-dependent targets are communicated through language rather than rendered in the point cloud.Source: §4.4 · p. 6

5 EXPERIMENTS

The experiments test held-out specification generalization across simulation benchmarks, ablations, scaling settings, and a physical robot. T3DP consistently outperforms global alignment, while action probes and real-world placement comparisons examine specification-sensitive behavior.

5.1 MAIN RESULTS

  • 14.2 points on Meta-World and 13.5 points on ManiSkill: T3DP improves average success over T2DA, with gains across all families and an advantage across five RoboTwin families.Meta-World average success is 50.3%; ManiSkill average success is 54.2%.Source: §5.1 · p. 8

5.2 ABLATION EXPERIMENTS

  • 50.3% average success: fine-grained alignment improves over global-only alignment by 14.2 points and leads on every Meta-World task family.Raw language conditioning raises average success from 16.2% to 19.1%, while global alignment raises it to 36.1%.Source: §5.2 · p. 8, Figure 3 · p. 8
  • 0.1238 MAE on Reach versus 0.1423 for T2DA: T3DP’s representation better predicts target actions when state is fixed and language varies.The state-only predictor collapses to the average action; Figure 4 contrasts it with T3DP’s specification-following predictions.Source: §5.2 · pp. 8–9, Figure 4 · p. 8

5.3 SCALABILITY ANALYSIS

  • At 10, 20, and 40 training specifications, average success rises from 23.9%, 36.1%, and 69.2% to 32.4%, 50.3%, and 76.4%, respectively.Each training specification contributes one expert demonstration, and evaluation uses the same held-out set.Source: §5.3 · p. 9, Table 3 · p. 9
  • In joint training, T3DP improves average success by 5.2 points over T2DA and 29.4 points over DP3, but T2DA is better on Pick Place and slightly better on Reach.One policy is jointly trained across all five Meta-World task families.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 REAL-WORLD TRANSFER

  • T3DP achieves the highest success rates on both real-robot tasks, with placement more precise than T2DA and improved reliability and precision beyond simulation.The PiPer X evaluation covers two task families and 20 specifications per family; success requires a final position within 3 cm of the target.Source: §5.4 · pp. 9–10, Figure 5 · p. 10

6 CONCLUSION, LIMITATIONS AND FUTURE WORK

The conclusion presents T3DP as a fine-grained alignment approach for held-out specifications, while limiting the evaluation to known task families and templated instructions. The appendices describe the benchmark specifications, split construction, and demonstration processing.

6 CONCLUSION, LIMITATIONS AND FUTURE WORK

  • T3DP aligns instruction tokens with local demonstration behavior representations, but requires expert trajectories with reward signals and is evaluated only on held-out specifications within known task families under templated instructions.Future work could learn behavior representations from state-action sequences and extend evaluation to unseen task families, free-form instructions, and diverse embodiments.Source: §6 · p. 10

A TASK, LANGUAGE SPECIFICATIONS, AND DATASET SPLITS •

  • Meta-World, RoboTwin, and ManiSkill cover spatial targets, displacements, placement, and articulated-object tasks, with benchmark-specific observations, states, and action spaces.The numerical specification is provided through language rather than the observation in Meta-World and ManiSkill.Source: A TASK, LANGUAGE SPECIFICATIONS, AND… · p. 14

A.1 DATASET SPLITS

  • All 15 task domains occur in training and evaluation, so the held-out test measures unseen specifications rather than unseen task categories.Each domain has 20 training and 20 held-out test specifications; test specifications are excluded from alignment and policy training.Source: §A.1 · p. 14

A.2 LANGUAGE INSTRUCTIONS AND TASK SPECIFICATIONS

  • Training and test instructions use the same task-specific language construction but instantiate disjoint behavioral specifications, so evaluation does not test paraphrase generalization.Specifications encode factors such as target location, displacement, placement goal, or articulation extent.Source: §A.2 · p. 14

A.3 DATASET CONSTRUCTION AND SEQUENCE SAMPLING

  • Training data retain only successful expert trajectories, which store point clouds, proprioception, actions, rewards, episode boundaries, and task-specification metadata.Point clouds are converted to 512 XYZ points per frame, and normalization statistics use training demonstrations only.Source: §A.3 · pp. 14–15

A.4 EVALUATION PROTOCOL

Evaluation uses common held-out specifications and closed-loop replanning, with stricter-than-native success thresholds for fine-grained specification following. Baseline comparisons align data and training conditions while retaining baseline architectures and objectives.

A.4 EVALUATION PROTOCOL

  • Policies replan from the latest two observations, execute eight predicted actions, and are evaluated primarily by success rate under fixed, strict specification-specific thresholds.For example, Meta-World Reach requires a final position within 2.5 cm of the target, while Push requires planar displacement error below 1 cm.Source: §A.4 · pp. 15–16

A.5 BASELINE IMPLEMENTATIONS AND FAIR COMPARISON

  • All methods share demonstrations, inputs, splits, evaluation conditions, and aligned training budgets, while baseline architectures and objectives remain unchanged.T2DA and T3DP share the behavior encoder, text initialization, alignment data, policy architecture, and optimization budget; they differ in global versus fine-grained alignment objectives.Source: §A.5 · p. 16
  • The baselines span language-free DP3, instruction-conditioned 3DDA and 3D-LOTUS, and T2DA’s globally pooled behavior–instruction alignment.DP3 receives no language; 3DDA uses global instruction–trajectory supervision, while 3D-LOTUS injects language through cross-attention.Source: §A.5 · p. 16

B IMPLEMENTATION DETAILS

The implementation learns trajectory behavior representations, aligns them with instruction tokens, and uses the resulting language condition in a point-cloud diffusion policy. The three training stages take approximately three hours in total on the reported hardware.

B.1 BEHAVIOR ENCODER

  • The behavior encoder processes trajectory windows with point-cloud, state, action, and reward tokens, then mean-pools Transformer outputs into a global behavior representation.A reward decoder predicts rewards for state–action pairs outside the encoded window to encourage specification-dependent behavior representations.Source: §B.1 · p. 17

B.2 TEXT ENCODER AND LANGUAGE-BEHAVIOR ALIGNMENT

  • Alignment combines bidirectional global contrastive learning with bidirectional token-level local interaction between language tokens and behavior tokens.The behavior encoder is frozen while the language encoder is adapted using LoRA; pooled representations support global alignment.Source: §B.2 · p. 17

B.3 LANGUAGE-CONDITIONED 3D POLICY

  • The frozen aligned language encoder supplies a pooled instruction condition to a point-cloud diffusion policy that predicts action-sequence noise from observations and diffusion timestep.One policy is trained across all training specifications within each task family, and targets are communicated through language rather than rendered in the point cloud.Source: §B.3 · p. 17

B.4 COMPUTATIONAL RESOURCES

  • The three stages use approximately three hours of training on one RTX 3090 GPU; the policy uses a DDIM scheduler and 512-point XYZ observations without RGB.Behavior pretraining, alignment, and policy training each take about one hour, with modest environment- and data-loading-dependent variation.Source: §B.4 · pp. 17–18, Table 6 · p. 18

C TRAINING AND INFERENCE ALGORITHMS

T3DP learns point-cloud behavior representations, aligns language globally and locally to behavior, then uses the frozen aligned encoder to condition a diffusion policy. On RoboTwin, it improves held-out-specification success over global alignment across all five task families.

C.1 BEHAVIOR REPRESENTATION LEARNING

  • The behavior encoder processes interleaved point-cloud state, action, and reward tokens with a bidirectional Transformer, then mean-pools its outputs.Training samples trajectory windows and retains a checkpoint based on validation reward prediction.Source: §C.1 · p. 19, Algorithm 1: Point-cloud behavior… · p. 20

C.2 GLOBAL AND LOCAL LANGUAGE-BEHAVIOR ALIGNMENT

  • Global and local language-behavior similarities are trained with bidirectional contrastive objectives, matching pooled representations and token-level state–action representations.The alignment algorithm combines global and local losses to update the text encoder’s LoRA parameters and text-side projections.Source: §C.2 · p. 19, Algorithm 2: Global–local… · p. 20

C.3 LANGUAGE-CONDITIONED POLICY LEARNING

  • The aligned text encoder is frozen for policy learning, and the policy consumes only its pooled language representation alongside point-cloud observations and robot state.The local alignment objective shapes the pooled condition but token-level features are not passed directly to the policy.Source: §C.3 · p. 19, Algorithm 3: Language-conditioned… · p. 20, §1 · p. 20

C.4 INFERENCE WITHOUT DEMONSTRATIONS

  • At deployment, the policy receives the instruction and current observation, samples an action sequence, executes the configured segment, and replans without demonstrations or the behavior encoder.Held-out specifications therefore do not require corresponding training demonstrations at test time.Source: §C.4 · p. 19, §9 · p. 20

D.1 ADDITIONAL ROBOTWIN RESULTS

  • The RoboTwin evaluation compares DP3, Raw CLIP, T2DA, and T3DP under shared demonstrations, inputs, specification splits, and held-out-specification evaluation.The five task families include Open Microwave, Open Cabinet Drawer, Move Pill Bottle, Move Playing Card, and Move Stapler.Source: §D.1 · p. 21
  • T3DP reaches 41.8% average RoboTwin success versus 30.8% for T2DA, improving over T2DA on all five task families.The comparison covers articulated-object manipulation and object placement; the largest reported gains are on Open Cabinet Drawer (16.7% to 37.3%) and Move Playing Card (32.7% to 52.3%).Source: §D.1 · p. 21, Table 7 · p. 21

D.2 ALIGNMENT REPRESENTATION ANALYSIS

Fine-grained alignment better preserves specification geometry and improves action prediction when language varies at a fixed state. The analysis also describes the real-robot policy’s point-cloud inputs and construction.

D.2 ALIGNMENT REPRESENTATION ANALYSIS

  • Fine-grained alignment improves both global ordering and local neighborhood preservation over global alignment across spatial targets, relative displacements, and articulated states.The representation-geometry evaluation uses Spearman correlation between specification and representation distances and kNN@3 neighborhood preservation.Source: §D.2 · p. 21, Table 8 · p. 22
  • With state fixed and language varied across held-out specifications, fine-grained alignment reduces action-probe MAE over global alignment by 14.7% on Push and 13.0% on Reach.The probe averages four trajectory splits and is presented as diagnostic evidence given the limited number of held-out specifications.Source: §D.2 · pp. 21–22, Figure 6 · p. 22

E.1 ROBOT AND SENSOR SETUP

  • The real-robot policy uses geometric point clouds and proprioceptive state, with specification-dependent target information communicated through language rather than RGB input.Depth images are transformed to the robot frame, workspace-filtered, then subsampled to 512 XYZ points.Source: §E.1 · p. 23, §E.2 · p. 23

E.3 TASKS AND TARGET SPECIFICATIONS

Real-robot evaluation tests placement at held-out tabletop targets using simulation-assisted demonstrations and a 3 cm success threshold. Qualitative results report the strongest placement precision and success for T3DP, while showing residual grasping and placement errors.

E.3 TASKS AND TARGET SPECIFICATIONS

  • Move Pill Bottle and Move Stapler specify tabletop target locations through fixed language templates, with 20 training and 20 disjoint held-out targets per task.Held-out targets remain within the feasible placement region covered by training targets.Source: §E.3 · p. 23, Figure 8 · p. 24

E.4 REAL-ROBOT DEMONSTRATION COLLECTION

  • Demonstrations use simulator-generated expert motions replayed on the real robot, and only successful real-world replays enter the training dataset.The policy training data records real depth observations, proprioception, and executed actions.Source: §E.4 · p. 23

E.5 EXECUTION AND SUCCESS CRITERIA

  • Each method is tested once on each of the 20 held-out targets per task, and success requires final object position within 3 cm of the target.Target locations are used for evaluation and are not explicitly encoded in the policy observation.Source: §E.5 · pp. 23–24

E.6 QUALITATIVE RESULTS AND FAILURE CASES

  • T3DP yields the strongest placement precision and success across both task families, while remaining failures include grasping or placement errors near the 3 cm threshold.DP3 places objects farthest from targets, while T2DA typically brings them closer.Source: §E.6 · p. 24

F LIMITATIONS

T3DP’s reward-predictive behavior encoder assumes expert trajectories annotated with rewards, which may be unavailable or costly to define.

F LIMITATIONS

  • Reward-predictive behavior encoding requires expert trajectories annotated with rewards, often unavailable or costly to define in real-world datasets.Many existing robot datasets contain state–action sequences without a reliable reward signal.Source: F LIMITATIONS · p. 24

Related papers

Loading 2609.39599v1…