PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Latent Planning Survive Point Clouds? Action-Conditioned JEPA World Models for Geometric Observations

Fabio F. Oberweger, Michael Schwingshackl

arXiv:2608.29434v1cs.LGcs.AIcs.CV

TL;DR

The paper asks whether action-conditioned latent planning remains viable for sparse, unordered, self-occluded point clouds, where both slow-feature and geometric shortcuts threaten collapse. It lifts three JEPA designs to point clouds and compares them with image counterparts under controlled re-sensing. All three models plan without collapse, with the distribution-prior model matching its image counterpart and the action-sensitive model strongest where geometry moves most; a target-conditioned goal latent also removes the need for a goal observation.

  • Problem

    Whether latent prediction survives geometric observations is unclear because sparse point clouds combine a slow-feature optimum with a geometric shortcut in 3D self-supervision.

  • Method

    The paper lifts frozen-encoder, distribution-prior, and action-sensitive JEPA designs to point clouds and evaluates them on re-sensed image benchmarks with a target-to-latent goal module.

  • Results

    All three point-cloud models plan without collapse; Point-LeWM is statistically equivalent to its image baseline on every benchmark, while Point-Delta-JEPA is strongest where the most geometry moves.

  • Takeaways & Limitations

    Latent planning can operate over point clouds, with positions linearly decodable, attention focused on moving entities, and commanded 3D targets replacing goal observations without losing success.

  • Takeaways & Limitations

    The study covers four simulated tabletop scenes with an idealized sensor and synthetic test-time degradation, while the target-to-latent module assumes training-time target annotations.

Abstract

from arXiv · show

JEPA world models make latent-space planning a practical route to control, but they are built almost exclusively on images. Whether latent prediction survives geometric observations is unclear: point clouds are sparse, unordered, and self-occluded, and with 0.3-15% of scene points moving, the slow-feature optimum of latent prediction compounds with the geometric shortcut of 3D self-supervision. We lift three canonical JEPA designs to point clouds, frozen-encoder, distribution-prior, and action-sensitive, and re-sense the stable-worldmodel benchmark so that only the observation differs from the image baselines. All three plan without collapse: the distribution-prior model is statistically equivalent to its re-evaluated image counterpart on every benchmark, and the action-sensitive model attains the strongest result in our controlled comparison where the most geometry moves. Probing explains why: object positions are almost perfectly linearly decodable and attention falls on the few moving points. Planning withstands heavy dropout never seen in training, though range noise defeats the thinnest scene. Geometry finally makes a commanded 3D target a natural goal interface: we construct the goal latent from the target and the current latent, at no cost in success rate, without a goal observation.

1 Introduction

This paper asks whether action-conditioned latent planning survives sparse point-cloud observations, where slow-feature and geometric shortcuts create two collapse routes. It lifts canonical JEPA designs to point clouds, evaluates them under controlled re-sensing, and introduces target-conditioned goal latents.

  • Motivation: Point clouds create two collapse routes: temporally constant latent features and geometric shortcuts based on point height or surface normals.Only 1–15% of scene points may move, while coordinates directly enter 3D self-supervised operators.
  • Approach: The study lifts frozen-encoder, distribution-prior, and action-sensitive JEPA designs to point clouds and evaluates them against image counterparts using re-sensed benchmarks.Episodes, actions, goals, and planners are held fixed while observations are replaced by simulated point clouds.
  • Goal specification: A target-conditioned module constructs the goal latent from the current latent and a commanded 3D target, removing the need for an inference-time goal observation without losing success.The interface specifies where an object should end up rather than providing a scan of the completed configuration.
  • Findings: None of the point-cloud models collapses: positions remain linearly readable, attention focuses on moving entities, and planning largely tolerates unseen sensor degradation.The comparison includes Utonia-WM, Point-LeWM, Point-Delta-JEPA, and a geometric Vox-WM ablation.
  • Findings: Point-LeWM matches its image baseline statistically on every benchmark, while Point-Delta-JEPA exceeds it where the most geometry moves.These results come from the controlled modality comparison rather than directly comparable uncontrolled image numbers.

2 Related Work

Related work establishes latent JEPA planning for images and several anti-collapse mechanisms, while point-cloud world models generally reconstruct or generate geometry. The paper positions its contribution as action-conditioned latent planning directly over point-cloud observations.

  • JEPA world models: JEPAs predict action-conditioned future representations rather than reconstructing observations, with frozen-encoder, end-to-end, distributional, and action-sensitive variants.The cited lineage includes DINO-WM, PLDM, LeWorldModel, and later temporal-difference, adaptation, and action-sensitivity methods.
  • Point-cloud world models: Existing point-cloud world models either regress future geometry or generate scans and occupancy, paying planning costs on decoded geometric outputs.These approaches include manipulation models using point displacements or completion and driving models synthesizing future scans or occupancy.
  • Gap: Prior point-cloud work does not establish action-conditioned latent prediction with latent-space planning, while AD-L-JEPA is action-free.The paper identifies this as the missing combination addressed by its point-cloud JEPA models.
  • Goal specification: Goal-conditioned latent planning commonly requires an encoded goal observation, whereas forecasting implicit subgoals does not permit commands to arbitrary targets.The paper’s target interface addresses this limitation by specifying a 3D goal directly.

3 Methods

The paper re-senses existing image benchmarks with simulated LiDAR and instantiates three action-conditioned JEPA world-model designs for point-cloud observations. Planning uses latent-space MPC, with a target-to-latent module providing goal latents from 3D targets and current states.

  • 3.1 Point-cloud sensing and dataset replication: The benchmark replays every recorded image-dataset state with a raycast range sensor, preserving episodes, actions, lengths, and goals while changing observations.The sensor produces fixed (10,000, 3) arrays from a 100 × 100 ray grid; the comparison isolates observation geometry.
  • 3.1 Point-cloud sensing and dataset replication: Only 0.3–15% of returns fall on moving entities, while ground returns dominate the point clouds.Two-Room and Push-T use lightweight MuJoCo scenes only for sensing; their original dynamics, rewards, and termination remain unchanged.
  • 3.2 Point cloud world models and planning toward 3D targets: The offline, reward-free setup represents each observation as an unordered point set in R3 and each action as a continuous control input.At test time, the models plan actions toward a specified goal.
  • 3.2 Point cloud world models and planning toward 3D targets: All models encode observations into latents and predict future latents from recent latent context and actions without reconstructing observations.Point-LeWM and Point-Delta-JEPA train the encoder and predictor end-to-end; Utonia-WM trains only its predictor.
  • 3.2.1 Model architecture: Point-LeWM and Point-Delta-JEPA use a shared point-cloud encoder, while their objectives differ through distributional regularization versus action-discriminative latent displacement.Coordinates use fixed workspace normalization, metric-radius grouping, and LayerNorm; the latent dimension is 192.
  • 3.2.1 Model architecture: Utonia-WM freezes a Point Transformer V3 encoder, pools super-point features into canonical-frame grid cells, and trains a predictor against frozen latent targets.The implementation pairs a frozen ≈137M-parameter backbone with a predictor trained on precomputed features; planning uses frameskip 5 and three past latents.
  • 3.2.2 Planning toward 3D targets: Latent-space MPC minimizes the distance between an autoregressive rollout and a goal latent over action sequences using the cross-entropy method.Planner and CEM hyperparameters are inherited unchanged from the image pipelines.
  • 3.2.2 Planning toward 3D targets: A target-to-latent module maps a task-relevant 3D target and current latent to the latent of a stored goal observation, replacing inference-time goal scans.The module is trained on same-trajectory frame pairs with random temporal offsets; an unconditioned variant omits the current latent.

4 Experiments

Across controlled point-cloud experiments, latent planners remain effective despite sparse geometric observations, while objective choice shapes which tasks benefit. Probing, perturbation tests, and target-encoding ablations explain the observed planning behavior and support goal specification from 3D targets without goal scans.

  • Evaluation: 10 seeds × 50 episodes evaluate all models with CEM on identical fixed start–goal pairs.The evaluation uses the stable-worldmodel solver layer and shared episodes across models.
  • Baselines: Every point-cloud model plans far above Random, while Point-LeWM is statistically equivalent to the re-evaluated image LeWM baseline on every benchmark except a 2.8-point Push-T deficit.Point-LeWM runs at 0.53 s per CEM solve versus 0.48 s for the image baseline.
  • Baselines: 83.4 ± 3.8 is Point-Delta-JEPA’s best controlled result on OGB-Cube, where it outperforms Point-LeWM by 17.4 points; Push-T reverses the ranking.On Reacher, the objectives are statistically indistinguishable, showing task-dependent objective effects.
  • Baselines: 94.0 on Two-Room is Utonia-WM’s strongest result, while it beats Vox-WM by 7.8–16.8 points across benchmarks.Its Push-T result is weaker at 46.6, consistent with the frozen representation’s limited task adaptation.
  • Probing: Positions are linearly decodable at R2 ≥0.95, but angle and joint-velocity organization distinguish planning performance across tasks.Point-LeWM linearly decodes Push-T block angle at R2 = 0.89 versus 0.66 for Point-Delta-JEPA, while only Point-Delta-JEPA decodes OGB-Cube joint velocity with MLP R2 0.63 versus 0.10.
  • Robustness: Under 50% dropout, Point-LeWM retains 99% and 95% of clean performance on Two-Room and Reacher, while Push-T noise reduces both models below 5% relative.OGB-Cube remains near clean performance under noise and dropout, and attention concentrates on moving entities carrying as little as 0.3% of returns.
  • 3D-target planning: Target-to-latent encoders match goal-cloud planning within seed noise, making the goal observation dispensable without sacrificing success.On OGB-Cube, removing the current latent costs Point-Delta-JEPA up to 12 points, whereas unconditioned variants suffice on Push-T and Two-Room.

5 Conclusion

The study finds that latent planning survives point-cloud observations across the evaluated settings, but performance depends on architecture and sensing conditions. It also identifies practical scope limits and a target-conditioned goal interface that avoids requiring a goal observation.

  • Conclusion: None of the three point-cloud JEPA models collapses, even when as little as 0.3% of returns move.All models plan under sparse, self-occluded scans.
  • Conclusion: The distribution-prior model matches its image counterpart statistically, while the action-sensitive model exceeds it where the most geometry moves.
  • Conclusion: Positions are almost perfectly linearly decodable, attention focuses on task-relevant entities, and planning withstands heavy unseen dropout.
  • Limitations: Range noise remains a failure mode for thin structures, and the study covers four simulated tabletop scenes with an idealized sensor.The limitations also include synthetic test-time degradation and uncontrolled dagger baselines.
  • Conclusion: A target-to-latent module lets the planner use a commanded 3D target instead of a goal observation without reducing success rate.

A Implementation details

The implementation uses geometry-only point-cloud encoders, action-conditioned predictors, inherited planning settings, and a target-conditioning module. It also documents compute, licensing, and architectural controls for the model variants.

  • Point-cloud encoder: The shared point-cloud encoder uses sensor-frame coordinates without color or intensity, 256 sampled centres, local ball queries, and 192-dimensional tokens.The tokens feed a 12-layer ViT-Tiny trunk whose CLS output is the latent.
  • Predictor and optimization: Point-LeWM and Point-Delta-JEPA use a six-block causal action-conditioned transformer with three past latents and five-frame action embeddings.
  • Frozen encoder: Utonia-WM uses a frozen approximately 137M-parameter PTv3 backbone after canonical-frame transformation and voxel downsampling.
  • Geometric ablation: Vox-WM replaces frozen features with parameter-free geometric cell statistics while keeping Utonia-WM tokenization and prediction settings fixed.
  • Planning and compute: Planning hyperparameters are inherited unchanged from the image pipelines, while training uses two A100 GPUs and end-to-end runs take roughly two to three days.
  • Target-to-latent module: The target-conditioning module normalizes commanded coordinates with the encoder’s workspace map and lifts them using 32-band Fourier features.
  • Licenses: The released assets span MIT, Apache 2.0, and BSD-3-Clause licenses across the benchmark, environments, pretrained backbone, and software stack.

B The unconditioned target-to-latent objective

The unconditioned target-to-latent objective can produce a goal representation that no observed scene realizes because it averages over unconstrained scene content. Conditioning on the current latent removes fixed-scene variation, but residual ambiguity can still matter when moving entities remain unconstrained.

  • Objective: The unconditioned objective minimizes mean-squared error using the conditional mean of latents associated with a target.
  • Geometric issue: That conditional mean is a barycenter over scenes sharing the target, including unconstrained configurations such as arm poses on OGB-Cube.
  • Geometric issue: Because the encoder’s latent image need not be convex, the barycenter may correspond to no observation in the data distribution.
  • Planning consequence: A planner minimizing distance to such a mean can be steered toward an unattainable goal latent.
  • Conditioning: Conditioning on the current latent averages only over futures of the current scene, leaving residual variation primarily in moving entities the target does not constrain.
  • Empirical result: On Push-T and Two-Room, the unconditioned variant matches the goal-cloud reference, whereas omitting the current latent costs up to 12 points on OGB-Cube.
  • Alternative construction: Synthetic goal-scan construction faces the same ambiguity because the target does not fully specify the scene, including an arm pose on OGB-Cube.

C Point-cloud sensing and dataset replication: details

The sensing setup fixes sensor, viewpoint, sampling, and scene-construction choices to replicate the benchmark with point clouds. Goal markers are excluded from scans so goals must be inferred rather than directly read.

  • Configuration: The released configurations define the exact sensor parameters, mirror geometry, and scene constants used in the experiments.
  • Ray budget: A 100 × 100 ray grid is the coarsest tested sampling density that still returns roughly one hundred points for the smallest task-relevant entities.
  • Viewpoint: Most viewpoints place the camera at 2.5R along a 45° azimuth and elevation bearing, with OGBench retaining its released oblique camera.
  • Mirror scenes: Mirrored-environment entity heights were fixed to separate objects from the floor while limiting self-occlusion, and Reacher3D link lengths were fitted to recorded joint angles.
  • Goal handling: Goal markers are excluded from every scan so the planner cannot read the task directly from returned geometry.

D.1 Physical latent probing

The probing protocol tests whether the latent used for planning preserves privileged physical state, using linear and MLP regressors across environments. Positions are highly recoverable, while orientation reveals task- and objective-specific differences.

  • Protocol: The probed representation is the 192-dimensional latent used by the CEM planning cost.Each model encodes every tenth training-table frame after its projector.
  • Findings: Across all three environments, positions are linearly decodable at R2 ≥0.95 and nearly exact with an MLP.The probes regress privileged simulator state on episode-disjoint splits.
  • Findings: Reacher provides little objective-separation signal because positions and link headings are linearly decodable at R2 ≈1 from both models.The oblique viewpoint fully exposes the arm, making the task saturated for probing.
  • Findings: OGB-Cube yaw has near-zero R2 because cube symmetry makes orientation unidentifiable from geometry, not because the representation fails.The raw yaw label is not folded into a fundamental domain.

D.2 Point cloud perturbations

Evaluation-only point-cloud perturbations test whether clean-trained latent representations remain usable under sensor degradation. Dropout is generally tolerated when returns still cover entities, whereas the thinnest scene is vulnerable under heavy dropout and degraded goals impose an additional cost.

  • Evaluation protocol: Perturbations are applied only at evaluation time, so the test measures clean-trained representation robustness to deployment-time sensor degradation.Noise is along-ray, scaled by each frame’s scan extent, and dropout removes returns through the normal invalid-return filter.
  • Dropout: Dropout is benign wherever surviving returns still sample the entities, but Push-T falls below 50% relative success under 50% dropout.The pusher in Push-T is near the ray-budget floor.
  • Goal degradation: Degrading the goal as well as the observation costs an additional roughly uniform margin, including Two-Room noise 0.25% at 80.2 versus 61.6 relative success for Point-LeWM.The stored target remains clean in the deployment-case condition and is degraded identically in the representation-focused condition.
  • Attention: Both end-to-end encoders concentrate attention on moving entities, while Point-Delta-JEPA also assigns visibly more attention to ground and table.Attention is normalized per cloud and visualized through CLS-token rollout.

E Statistical tests

The statistical analysis uses paired evaluations on identical start–goal episodes to distinguish differences from practical parity. It finds equivalence between Point-LeWM and the image baseline on every benchmark, while objective-family differences depend on the environment.

  • Design: All comparisons use the same 50 start–goal windows and paired seeds, removing seed-to-seed variation in episode difficulty.The tests operate on ten-seed success rates from Table 2.
  • Tests: TOST establishes equivalence at the ±10-point margin when the data rule out a true difference larger than that margin.A non-significant paired t-test alone is not treated as evidence of equality.
  • Modality comparison: Equivalence holds between Point-LeWM and the re-evaluated image baseline on every benchmark at p ≤0.005.Two-Room and Push-T also pass the stricter ±5-point margin, while the widest 95% interval edge is −8.1 points on OGB-Cube.
  • Objective comparison: Point-Delta-JEPA exceeds Point-LeWM by +13.0 points on Two-Room and +17.4 on OGB-Cube, whereas Point-LeWM leads by +12.8 on Push-T.On Reacher, the objectives do not differ significantly: −2.8 points, p = 0.30.
  • Scope: The study is foundational research in simulated tabletop environments rather than a deployment-specific evaluation.The authors describe societal effects and risks as generic to improved robot autonomy.
Loading 2608.29434v1…