PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Policy Learning and State Prediction through Spatial Language Modeling

Minye Wu, Zehao Wang, Tinne Tuytelaars

arXiv:2610.12172v1cs.ROcs.AI

TL;DR

Goal-directed pushing requires actions that account for geometry-dependent contact effects, and subsequent observations offer complementary supervision to expert actions. Spatial Language Modeling trains action generation and action-conditioned state prediction through shared grammar-structured coordinate sequences. On Push-T, it achieves higher real-robot task success and target coverage than the evaluated policy baselines.

  • Problem

    In Push-T, the same pusher displacement can produce different translation and rotation depending on contact location and block orientation, while demonstrations provide both action labels and resulting scene changes.

    Source: §1 · p. 1

  • Method

    Spatial Language Modeling uses grammar-structured coordinate and semantic tokens so one autoregressive model learns action generation and action-conditioned state prediction with a shared next-token objective.

    Source: §1 · p. 2

  • Results

    On the real robot, success rate was 0.80 versus 0.45 for DP-Coord, 0.50 for DP-Tag, and 0.65 for ACT; mean episode-level maximum coverage was 0.95 versus 0.89 for both DP variants and 0.86 for ACT.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The model also predicts successive scene states from supplied action trajectories, and predicted object states retain substantial geometric overlap with simulator references across the evaluated rollout horizon.

    Source: §1 · p. 2

    The real-robot system relies on calibrated AprilTag localization and known object geometry; end-to-end learning from raw sensor observations was not evaluated.

    Source: §5 · p. 8

Abstract

from arXiv · show

Learning how actions change scene geometry can provide complementary supervision for goal-directed manipulation. We introduce Spatial Language Modeling, which represents scene contours, goals, action targets, and future states with a shared vocabulary of discrete coordinates and semantic tokens. A task-specific grammar organizes these elements into spatial sequences, allowing one autoregressive Transformer to learn action generation and action-conditioned state prediction through a common next-token objective. We train the model from scratch using random-play transition pretraining followed by joint action and state training on expert demonstrations. During pretraining, recorded action coordinates condition subsequent state predictions and are excluded from the prediction loss. During control, the model decodes only executable action targets and updates its history with newly observed states. We evaluate the approach on Push-T in simulation and on a real robot. The model achieves competitive simulation performance and higher task success and target coverage than the evaluated real-robot policy baselines. Training ablations show improved control with joint action and state sequences, with further gains from random-play pretraining. Given supplied action trajectories, the same model also predicts successive scene states, capturing the geometric effects of pushing.

1 Introduction

Push-T actions have geometry-dependent effects, and subsequent observations provide a complementary signal to expert action labels. Spatial Language Modeling shares a coordinate vocabulary and autoregressive objective for action generation and state prediction, with training combining random-play transitions and expert demonstrations.

1 Introduction

  • Contact location and block orientation change how the same pusher displacement translates and rotates the block, making subsequent observations a complementary training signal to expert actions.The paper frames this as a motivation for learning both goal-directed actions and the geometric consequences of interaction.Source: §1 · p. 1
  • At control time, the model generates action targets from observed history, while new observations refresh its context.In a separate prediction setting, supplied action trajectories condition successive scene-state predictions.Source: §1 · p. 2
  • Spatial Language Modeling uses a grammar-structured coordinate vocabulary for scene geometry, goals, action targets, and future states, framing action generation and state prediction as sequence completion.A shared next-token objective trains both prediction tasks.Source: §1 · p. 2
  • Random-play transition pretraining and expert-demonstration training jointly supervise action generation and subsequent scene states in the same model.During pretraining, recorded action coordinates condition state predictions but are excluded from the prediction loss.Source: §1 · p. 2
  • The paper demonstrates closed-loop Push-T control in simulation and on a real robot, with reported results and comparisons supporting the formulation.The introduction does not give specific benchmark values.Source: §1 · p. 3

2 Related Work

Prior work models demonstrated actions, predicts future scenes for planning, and applies language models to spatial representations. This work distinguishes itself by using coordinate-level transitions and action targets in one next-token objective for Push-T.

Policy Learning and World Modeling

  • Imitation-learning methods predict actions from demonstrations, including temporally extended sequences, denoised trajectories, and vision-language-action models [2] [3] [8] [4].Decision Transformer and Behavior Transformers also use sequence modeling for control [9] [10].Source: §2 · p. 3
  • World-model approaches connect future predictions to control through trajectory generation, interleaved video and action decoding, compositional plans, or latent-state planning [5, 7, 11, 12, 6].The paper instead uses coordinate-level scene transitions and action targets in one next-token objective, with random-play pretraining and joint action/state training.Source: §2 · p. 3

Language Models in Spatial Computation

  • Spatial-language work represents 3D scenes through point-cloud features, adaptive scene details, scene graphs, or visual coordinate prompts [13] [14] [15] [16] [18] [19].The paper organizes contours, goals, action targets, and future states as coordinate and semantic token sequences.Source: §2 · p. 3

Geometric and Scene Representations

  • Geometric sequence models encode meshes or spatial outputs as discrete tokens, while scene-understanding and generation methods cover point sets, volumetric geometry, scene relations, and 3D content [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44].These approaches provide context for the paper’s coordinate-based spatial interface.Source: §2 · p. 3

Spatial Language Modeling

  • Table 1 specifies that each COORD is one token and that control decoding omits optional state blocks.Figure 2 depicts observed-history completion into actions, followed by new observations closing the control loop.Source: Table 1 · p. 4, Figure 2 · p. 4

3 Method

The method represents Push-T scenes, goals, actions, and optional future states as grammar-structured coordinate sequences. A single autoregressive model completes these sequences for action generation and action-conditioned state prediction, with training and control using task-specific sequence formats.

3 Method

  • Spatial Language Modeling formulates Push-T as autoregressive completion of a grammar-structured sequence containing scene geometry, the goal, pusher actions, and optional future states.An observed-history prefix prompts a completion that is parsed into pusher actions.Source: §3 · p. 4

3.1 Spatial Language Modeling

  • The workspace is discretized into a regular grid whose locations form the coordinate vocabulary; occupied contour cells and action targets are represented with coordinate tokens.Grid resolution trades geometric precision against vocabulary size and contour-token count.Source: §3.1 · p. 4
  • A context-free grammar specifies token ordering, while a compiler serializes observations and actions and a parser converts generated action blocks into target coordinates.For control, a grammar state machine excludes future-state blocks and samples action coordinates autoregressively.Source: §3.1 · p. 4

3.2 Application to Push-T

  • For Push-T, the input describes the pusher, T-block, and goal; the scene representation retains each entity’s outer contour as occupied-cell coordinate tokens.MOVE_TO blocks specify pusher-center targets and may include the subsequent scene state, allowing action and state targets to share the model and objective.Source: §3.2 · pp. 4–5
  • Each rollout begins with a STATE and continues through temporally ordered MOVE_TO steps, each specifying a pusher target and optionally the resulting STATE.Rollout discretization makes consecutive pusher positions adjacent on the grid, yielding local displacements in the serialized trajectories.Source: §3.2 · p. 5
  • Random-play pretraining masks loss on action-coordinate tokens while using subsequent states as transition targets; expert demonstrations supervise actions and retained state blocks.During closed-loop control, the model completes an observed-history prefix with action blocks, excludes future STATE blocks, and refreshes its history with new observations.Source: §3.2 · p. 5

4 Experiments

The experiments evaluate closed-loop pushing, action-conditioned state prediction, and training ablations in simulation and on a real robot. The full method performs competitively in simulation, leads the evaluated robot-policy baselines, and benefits from joint action/state training and random-play pretraining.

4.1 Push-T Setup and Evaluation

  • The study uses 200 expert simulation demonstrations, plus 40 successful robot demonstrations and 20 random-play episodes for transition pretraining.The robot random-play set contains 5,229 frames; robot policy baselines use expert demonstrations only.Source: §4.1 · p. 6
  • Robot episodes terminate successfully at coverage of at least 0.95 or end after 300 steps, and coverage uses filled regions although model inputs use contours.Real-robot evaluation uses 20 initial states per method, manually matched as closely as possible.Source: §4.1 · p. 6

4.2 Push-T Results

  • Simulation Max is 0.95, matching DP-C and DP-T, while Avg is 0.93 versus 0.91 and 0.79; on the robot, SR is 0.80 versus 0.45, 0.50, and 0.65, and Max is 0.95 versus 0.89, 0.89, and 0.86.The robot comparisons are against DP-Coord, DP-Tag, and ACT, respectively.Source: §4.2 · pp. 6–7, Table 2 · p. 7
  • In action-conditioned rollouts, coverage falls from 0.98 at frame one to 0.92 at frame ten, while IoU falls from 0.96 to 0.85.Each predicted state becomes context for the next prediction; simulator reference states use the same manually annotated actions.Source: Figure 5 · p. 7, §4.2 · p. 7
  • Real-robot episode peak coverage concentrates around 90%–100%, while baselines include low-overlap episodes; averaged peak-frame images show a clearer T-block outline with fewer ghosting artifacts.The visual comparison summarizes alignment across episodes.Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 6 · p. 8
  • Adding future-state blocks raises robot success from 0.05 to 0.65 and maximum coverage from 0.67 to 0.92; random-play pretraining further raises them to 0.80 and 0.95.The first comparison is joint action/state training versus action-only; the second adds random-play pretraining to joint training.Source: §4.2 · pp. 7–8
  • On one A100 GPU, generating 100 action targets takes about 0.25 seconds of decoding, while a 20-step execution chunk takes about 0.05 seconds.Each target uses one move-to token and one coordinate token.Source: §4.2 · p. 8

5 Limitations

The evaluation is limited to calibrated, geometry-aware planar Push-T, and several aspects of broader deployment and long-horizon prediction remain unevaluated.

  • Real-robot evaluation relies on calibrated AprilTag localization and known object geometry; end-to-end learning from raw sensor observations has not been evaluated.Source: §5 · p. 8
  • The manually designed grammar and grid-resolution trade-off may require representational changes for new entities, action types, or task structures.Source: §5 · p. 8
  • Evaluation focuses on planar Push-T, leaving generalization to other object geometries, manipulation tasks, and 3D scenes unestablished.Source: §5 · p. 8
  • Autoregressive state-rollout errors accumulate, and longer-horizon prediction requires further evaluation; closed-loop control instead refreshes history with observations.Source: §5 · p. 8

6 Conclusion

Spatial Language Modeling represents geometry, goals, actions, and future states as grammar-structured coordinate sequences for one autoregressive model. Experiments report effective Push-T control and state rollouts that capture scene changes, with training comparisons favoring interleaved action-state sequences and random-play pretraining.

  • One autoregressive model learns action generation and action-conditioned state prediction through a shared next-token objective over grammar-structured coordinate sequences.Source: §6 · p. 8
  • Training combines transition pretraining on random interactions with joint action and state supervision on expert demonstrations.Source: §6 · p. 8
  • Experiments demonstrate effective control in simulated and real-world Push-T, while supplied-action rollouts capture successive geometric scene changes.Source: §6 · p. 8
  • Training comparisons show benefits from interleaved action and state sequences, with additional gains from random-play pretraining.Source: §6 · p. 8

A Model Implementation

The implementation uses a Qwen2 Transformer trained from scratch and represents Push-T coordinates with tokens for a 128×128 grid. Training samples pack multiple spatial sentences within a fixed context limit.

  • The model is a 462M-parameter Qwen2 with 7 transformer layers, hidden dimension 1536, and 24 attention heads, trained from scratch.Source: A Model Implementation · p. 13
  • Push-T uses 16,384 coordinate tokens for its 128×128 grid, alongside semantic tokens that define the sequence format.Source: A Model Implementation · p. 13
  • Training samples pack multiple spatial language sentences with a maximum context length of 13,000, using Rotary Position Embeddings.Source: A Model Implementation · p. 13

B Push-T Representation and Execution Details

Push-T observations and actions are serialized as spatial language sequences over a discretized 2D grid. The grammar structures scene geometry and temporally ordered actions, optionally includes resulting states, and supports action-only closed-loop execution.

C Full Symbol and Token Definitions

Push-T spatial sequences use coordinate and semantic tokens organized by a task grammar. State and action productions define how observations and pusher targets are represented, with optional next-state observations after actions.

C Full Symbol and Token Definitions

C.1 Baseline Implementation Details

The real-robot baselines use state-based inputs and were adapted to emit the same absolute pusher-target action. Their geometry encodings and observation histories differ from one another and from SLS.

C.1 Baseline Implementation Details

  • Real-robot baselines use state-based inputs; DP-Coord and ACT use T-block contour coordinates, while DP-Tag uses 12 AprilTag corners.Contour frames are padded to K=100 with (-1, -1); DP-Tag uses four corners from each of three tags in fixed slot order.Source: §C.1 · pp. 14–15
  • All baselines emit absolute pusher-target coordinates; Diffusion Policy uses two observation frames, ACT one, and SLS three.Source: §C.1 · p. 15
  • DP-Coord and DP-Tag adapt low-dimensional Diffusion Policy to flattened geometry and pusher coordinates, condition on two frames, and execute eight of 16 denoised actions.Coordinates are normalized to [-1, 1], and DP-Coord uses the sentinel-padded contour without an explicit padding mask.Source: §C.1 · p. 15
  • ACT [2] replaces image features with per-coordinate contour tokens, uses one observation, and executes eight of 16 predicted actions per replanning step.Its decoder uses 16 learnable query tokens; inference fixes the latent to its prior mean.Source: §C.1 · p. 15
Loading 2610.12172v1…