PaperlayerView exact revision on arXiv

Source-linked AI summary

Compact Visuotactile World Models for Lifting: Prediction, Reward Alignment, and Force Constraints

Qinzhen Ma, Sida Peng

arXiv:2609.09597v1cs.ROcs.AI

TL;DR

The paper asks whether tactile prediction improvements support reliable, force-constrained manipulation decisions. It combines a compact visuotactile world model, trajectory-level calibration, and behavior-initialized imagined control, finding that sensing and reward revisions improve selected outcomes without resolving force-constrained control. The evidence remains bounded to public sensing records and simulator execution without demonstrated transfer.

  • Problem

    Contact prediction is useful for manipulation only if improved forecasts support effective decisions under trajectory reliability and force constraints.

    Source: §I · p. 1

  • Method

    The study combines a compact randomly initialized visuotactile world model, trajectory-level uncertainty calibration, and behavior-initialized actor-critic learning in imagination.

    Source: §I · p. 1

  • Results

    Matched reward revision improved executed lifting on fresh environments, but force-budgeted success remained below force feedback, while trajectory calibration exposed tail errors and aligned touch could lose to persistence.

    Source: B. Forecasts and the persistence… · p. 4, C. Calibration limits · p. 4, E. Initial control experiment · p. 5

  • Takeaways & Limitations

    The comparisons distinguish improvements in sensing and task reward from improvements in force-constrained control.

    Source: §I · p. 1, §VI · p. 6

    The evidence uses one rigid simulated geometry, simulated contact-force aggregates, public GelSight sensing records, and no demonstrated transfer between sensing data and simulator policy.

    Source: §VI · p. 7

Abstract

from arXiv · show

Accurate contact prediction is useful for robotic manipulation only if it supports effective decisions. We investigate this connection using a compact, randomly initialized visuotactile world model, trajectory-level uncertainty calibration, and behavior-initialized actor-critic learning in imagination. On 160 MuJoCo Lift episodes, adding touch reduces endpoint-force prediction error from 1.058 to 0.228 N and interval-peak error from 2.724 to 0.523 N across three training seeds. However, tactile persistence achieves lower errors of 0.095 and 0.498 N, respectively. Two exploratory control rounds comprise 680 executions on 40 independent test initial conditions. A matched reward revision on fresh test environments increases in-distribution 10 cm lifting success from 20.0% to 93.3%, while success within an 8 N per-finger budget reaches only 33.3%, compared with 70.0% for force feedback. Calibration margins reduce force violations at the cost of task completion. In a separate study of public GelSight recordings, a force regressor achieves 0.04234 N error, but frame-level calibration covers only 15.80% of complete trajectories; trajectory-level calibration raises this to 87.36% at nominal 90% coverage. Together, these findings distinguish improvements in sensing and task reward from improvements in force-constrained control. The evidence is limited to public sensing records and simulator execution, without a demonstrated transfer between them.

I. INTRODUCTION

The paper tests whether tactile information remains useful as manipulation systems move from contact-force prediction to calibrated trajectory reliability and executed control. Its matched comparisons separate sensing gains, reward alignment, and force-constrained decision quality across simulator and public-data branches.

Motivation

  • Contact-rich manipulation couples perception, prediction, and control, but lower prediction error need not produce a better policy.Persistence can predict slowly varying contacts well, while infrequent force peaks may dominate constraints despite small average error.Source: §I · p. 1

Research question

  • The study evaluates grasp closure and lifting through force forecasts, trajectory-level reliability, and explicit force budgets rather than inferring material properties or damage thresholds.Before lift-off, gripper force does not determine object mass, and sticking contact does not uniquely identify friction.Source: §I · p. 1

Experimental design

  • The simulator branch uses contact-force aggregates for action-conditioned forecasting and frozen-world-model control, while the public-data branch tests optical tactile sensing and trajectory calibration.The branches address prediction reliability but are not a sensor-transfer or sim-to-real pipeline.Source: §I · p. 1, Figure 1 · p. 2

Contributions

  • The paper reports three findings: aligned touch can outperform vision-only input yet lose to persistence, trajectory calibration reveals tail errors, and reward revision improves lifting while remaining inferior to force feedback under force budgets.These comparisons identify a correctable reward mismatch alongside remaining dynamics and constraint limitations.Source: §I · p. 1

Positioning

  • Visuotactile world modeling and contact prediction are established areas, so the paper focuses on linking compact contact forecasts, trajectory-level error accounting, and decision usefulness.The paper adapts existing modeling and calibration components rather than claiming modality fusion or contact prediction as novel.Source: §II · p. 1

III. METHOD

The method encodes short histories of vision, touch, and proprioception into a compact latent state, then predicts task-relevant contact and environment quantities over imagined transitions. Training uses matched vision-only and visuotactile models with identical parameter counts and no privileged state inputs.

Inputs

  • The observation history contains three RGB frames, three tactile observations, and three proprioceptive observations, with simulated touch represented by per-finger normal and tangential contact-force components.These tactile inputs are contact-force aggregates rather than optical tactile images.Source: A. Compact action-conditioned model · p. 2

Representation

  • A convolutional visual encoder, separate tactile and proprioceptive MLPs, and a projection layer produce a 128-dimensional SimNorm latent.The visual, tactile, and proprioceptive features have dimensions 256, 64, and 32, respectively.Source: A. Compact action-conditioned model · p. 2

Prediction targets

  • The 652,157-parameter transition model predicts endpoint force, object height, support, contact, reward, an RGB target, and per-finger interval-maximum normal force.Interval-peak heads capture transient forces that endpoint predictions could miss; the model is randomly initialized and receives no privileged object state.Source: A. Compact action-conditioned model · p. 2

Optimization

  • Five-step training unrolls combine Huber regression, classification, image, and representation-consistency losses, with future observations used only as targets.Vision-only and visuotactile variants have identical parameter counts, and normalization uses training episodes exclusively.Source: A. Compact action-conditioned model · p. 2

B. Contact-envelope calibration

The contact-envelope method calibrates trajectory-level residuals for force and height forecasts and converts them into upper bounds for a contact analysis envelope. Its coverage guarantee is marginal under fixed-predictor exchangeability, not robust to distribution shift or adaptive closed-loop policies.

Force representation

  • The public-force experiment represents contact force as F = (Fx, Fy, Fn), with compression-positive normal force and analysis conditions for friction parameters and normal-force ceilings.These parameters are not identified friction coefficients or measured damage thresholds.Source: B. Contact-envelope calibration · p. 2

Calibration

  • Trajectory calibration takes the maximum residual over each complete contact trajectory and applies a corrected finite-sample quantile to obtain simultaneous bounds.The corrected quantile uses rank ⌈(n+1)(1−α)⌉, with negative quantiles clipped to zero.Source: B. Contact-envelope calibration · p. 2

Envelope test

  • The resulting upper bounds are predicted contact quantities plus the calibrated residual quantile, and a frame passes only when both specified limits are satisfied.A coordinate-box baseline instead calibrates maximum absolute coordinate-time error and propagates that box through the contact functions.Source: B. Contact-envelope calibration · p. 2

Scope

  • Under a fixed predictor and exchangeable calibration and test trajectories, the construction provides marginal simultaneous coverage but no guarantee under shape shift, adaptive control distributions, or new closed-loop policies.The method therefore reports acceptance and conditional violations separately.Source: B. Contact-envelope calibration · p. 2

Simulator calibration

  • Simulator calibration maximizes per-finger interval-peak underprediction and height residuals over evaluated windows and five future steps, while preserving the interval peak rather than substituting endpoint force.These bounds concern forecasts along recorded actions, not all counterfactual actions.Source: B. Contact-envelope calibration · pp. 2–3

C. Imagined policy learning

The policy stage freezes the learned world model, initializes an actor from behavior cloning, and optimizes imagined rewards over short rollouts. Force penalties and behavior-prior regularization constrain the actor, but independent simulator evaluation remains necessary because optimization can exploit model errors and shift the state distribution.

Actor initialization

  • A two-output actor is behavior-cloned, then controls vertical motion and gripper opening or closing after a scripted approach.Action mapping stays within training-action ranges, limiting extrapolation without guaranteeing state-distribution support.Source: C. Imagined policy learning · p. 3

Imagined optimization

  • The actor rolls the frozen model forward for five steps and maximizes predicted reward penalized by interval-peak force excess, action magnitude, and deviation from the behavior prior.A value network learns detached imagined-return targets with an EMA target critic for terminal bootstrapping.Source: C. Imagined policy learning · p. 3

Force constraint

  • The imagined stage uses per-finger predicted peaks, an 8 N force budget, and a calibration margin to define force-aware rewards.Ordinary imagined RL uses q = 0, whereas the margin variant uses the fixed calibration allowance.Source: C. Imagined policy learning · p. 3

Training procedure

  • Training uses 3,520 encoded starts, 1,000 behavior-cloning updates, and 1,500 imagined updates without online data collection or world-model retraining.The actor and critic learning rates are 2 × 10^-4 and 3 × 10^-4, respectively.Source: C. Imagined policy learning · p. 3

Evaluation

  • Policies are evaluated independently in simulation because optimization can exploit model errors and alter the state distribution used for recording-based calibration.The actor-critic objective also differs from the official TDMPC2 algorithm.Source: C. Imagined policy learning · p. 3

IV. PUBLIC TACTILE-FORCE STUDY

The public tactile-force study evaluates force prediction on trajectory-separated GelSight recordings, including held-out shapes and batch conditions. It uses a CNN regressor and reports both predictive accuracy and cross-shape generalization.

Data and protocol

  • 102,628 paired frames from 1,513 loading/shear trajectories comprise the public GelSight dataset.GelSight Mini images are paired with ATI Nano17 force measurements.Source: A. Data and protocol · p. 3
  • 453 training, 75 validation, 114 calibration, and 116 testing trajectories are used for sphere batches 1–5, with batch 6 and flat/sharp trajectories held out.Entire trajectories, rather than adjacent frames, define the partitions.Source: A. Data and protocol · p. 3

Model

  • A 361,091-parameter CNN is trained for force regression across three shared data splits and seeds.Training uses AdamW with validation-based model selection, while force statistics use only training labels.Source: A. Data and protocol · p. 3

Results

  • 0.04234 ± 0.00306 N sphere MAE and 0.03915 ± 0.00351 N batch-6 MAE are reported across seeds.A grouped-bootstrap 95% interval for sphere MAE is [0.03823, 0.04675] N.Source: B. Force prediction and distribution… · p. 3
  • The regressor performs worse than the constant baseline on held-out shapes, indicating poor cross-shape generalization in this protocol.Batch identifiers alone do not establish crossday or cross-sensor transfer.Source: B. Force prediction and distribution… · p. 3

C. Frame reliability does not imply trajectory reliability

The calibration analysis compares frame-level and trajectory-level reliability, showing that tighter direct trajectory calibration can trade observed coverage for a smaller allowance. The analysis also distinguishes offline force classification from executed control.

Calibration coverage

Acceptance analysis

  • For µ = 0.8 and Fn ≤2 N, box propagation accepts no frames, whereas direct trajectory calibration accepts 18.84%.Direct calibration has 0.192% conditional proxy-constraint violations in this illustrative envelope.Source: C. Frame reliability does not imply… · p. 3

Forecast evaluation

  • The world-model forecasts use 160 episodes split into training, validation, calibration, and ID/OOD test sets.Evaluation follows recorded actions and compares forecasts with actual simulator states.Source: A. Data and evaluation protocol · p. 4
  • The reported forecast errors are averaged over horizons one through five and summarized as mean ± sample standard deviation across three training seeds.This aggregation defines the comparison reported in Table III.Source: A. Data and evaluation protocol · p. 4, Table III · p. 4

B. Forecasts and the persistence counterexample

Aligned touch improves force prediction over vision, but persistence can outperform learned dynamics on key force metrics. Calibration and force-budget analysis reveal that low average error does not ensure useful or certifiable control.

Forecast accuracy

  • Touch reduces both endpoint and peak-force errors relative to vision across splits, using distinct force targets.Endpoint force averages six signed/contact components, while peak force averages two normal-force maxima.Source: B. Forecasts and the persistence… · p. 4

Contact-restricted evaluation

  • 0.409/1.131 N visuotactile contact-restricted endpoint MAE remains below vision’s 2.087/2.548 N for ID/OOD.All-phase error comparisons can be distorted by different contact mixes, so they do not establish better contact generalization.Source: B. Forecasts and the persistence… · p. 4

Persistence counterexample

  • 0.095 N ID endpoint MAE for tactile persistence outperforms the learned world model, whose ID peak MAE is 0.523 N versus 0.498 N for persistence.Persistence also reaches 0.203 N OOD endpoint MAE, while learned dynamics improve OOD peak MAE to 0.849 N versus 0.966 N.Source: B. Forecasts and the persistence… · p. 4

Calibration limits

  • 80%, 85%, and 90% ID joint episode coverage falls to 65%, 50%, and 45% OOD across three visuotactile calibration runs.Peak allowances are 13.204, 6.945, and 9.438 N, and the small number of independent episodes limits precision.Source: C. Calibration limits · p. 4

Force constraints

  • Two calibrated allowances exceed the entire 8 N per-finger budget before adding any predicted peak, making hard admission reject those forecasts.A soft calibration penalty is not a safety guarantee or evidence that abstention completes the task.Source: C. Calibration limits · p. 4

D. Matched interaction protocol

The matched interaction protocol evaluates fixed controllers on paired fresh simulator environments using strict height and force-budget success criteria. It also exposes feasibility and reward-alignment limits that complicate interpreting initial control results.

Task criteria

  • 0.10 m relative height for ten consecutive 20 Hz observations defines strict lifting success before lowering.Force-budget success additionally requires every finger’s interval peak to remain at or below 8 N throughout the complete 150-action episode.Source: D. Matched interaction protocol · pp. 4–5

OOD feasibility

Evaluation design

  • 20 new environments—ten ID and ten OOD—are shared across paired controller evaluations, with learned methods averaging three fixed policies.The sample is 20 independent environments, not 60, and no controller is selected from test performance.Source: E. Initial control experiment · p. 5, Table IV · p. 5

Reward alignment

  • 55.943 to 61.365 native return for visuotactile RL accompanies a strict lift-success decline from 15.0% to 6.7%.The native shaped reward does not encode the 10 cm, ten-observation criterion, so higher return does not track the strict task metric.Source: E. Initial control experiment · p. 5

Uncertainty

  • −20.0 percentage points is the margin-RL-minus-feedback joint-success difference, with a 95% interval of [−33.3, −6.7].Broad intervals and only ten environments per domain limit inference about training-procedure comparisons.Source: E. Initial control experiment · p. 5

F. Initial executed-distribution diagnostic

The exploratory reward revision was evaluated on fresh matched environments, improving ID strict lifting but not force regulation. Retrospective coverage varied across induced policy state distributions, so these results do not establish online safety.

Matched reward revision

  • ID strict success rose by 73.3 percentage points under height reward, with a fixed-policy environment-bootstrap 95% interval of [66.7, 83.3].Height-reward RL achieved [9] successes out of ten, versus [2, 4, 0] for matched native-reward policies.Source: G. Exploratory matched reward revision · p. 6
  • Joint success increased by 23.3 points, while mean episode peak increased by 0.432 N.The reported intervals were [0.0, 50.0] for joint success and [0.175, 0.711] for mean episode peak.Source: G. Exploratory matched reward revision · p. 6
  • Height-reward joint success remained 36.7 points below force feedback, indicating improved task attainment rather than improved force regulation.All OOD strict and joint successes remained zero in the follow-up.Source: G. Exploratory matched reward revision · p. 6, Table V · p. 6

Calibration diagnostic

  • Retrospective ID joint forecast coverage differed across policy-induced state distributions despite unchanged recording calibration.Coverage was [10%, 80%, 80%] for height RL, [0%, 60%, 50%] for native RL, and [100%, 100%, 100%] for BC.Source: G. Exploratory matched reward revision · p. 6

H. Computation and scope

The study’s executed-control evidence is exploratory and simulator-bound, with limited independent test conditions and unresolved force-constrained-control questions. The authors therefore call for broader evaluation and stronger baselines before claiming reliable force-constrained control.

Computation and evaluation scope

  • 680 simulator executions covered only 40 independent test initial conditions across both control rounds.The follow-up reused frozen world models rather than collecting new training interactions.Source: H. Computation and scope · p. 6

Evidence boundaries

  • Touch improved local force prediction, but persistence outperformed learned dynamics on several force metrics and unfavorable shape transfer remained.These findings prevent attributing the modality gain to useful action-conditioned dynamics alone.Source: §VI · p. 6

Executed control

  • A matched exploratory height-reward revision improved lifting but did not outperform force feedback on force-budgeted success and increased mean peak force.Calibration margins reduced violations while producing poor task completion, and recording-based forecast coverage degraded under learned policies.Source: §VI · p. 6

Future evaluation

  • Fresh confirmatory evaluation, stronger dynamics and reactive baselines, and more diverse feasible episodes remain necessary to test reliable force-constrained control.The conclusion specifically limits the current evidence to the demonstrated exploratory simulator setting.Source: §VI · pp. 6–7

Scope limitations

  • The study uses one rigid object geometry, simulated contact-force aggregates, and a small number of independent environments.The reward follow-up also lacks a vision-only height-reward ablation, and public GelSight sensing results were not transferred to the simulator policy.Source: §VI · p. 7

APPENDIX IMPLEMENTATION AND REPRODUCIBILITY DETAILS

The implementation combines a compact force-predicting world model, actor training, trajectory-level metrics, and simulator-specific control evaluation. Reproducibility depends on explicit preprocessing, calibration, bootstrap, and measurement conventions.

Public-force regressor

  • The public-force regressor processes 3×64×48 tactile images through four strided convolutions and predicts three force components.Inputs are normalized from bytes to [−1, 1], and checkpoint selection uses minimum validation MAE.Source: A. Training settings · p. 7

Training objective

  • The world-model objective combines force, height, reward, support, contact, image, and latent-consistency losses with weighted terms including 0.5Lr + 0.5LI + 20Lz.The losses use normalized Huber, binary cross-entropy, RGB mean-squared error, and latent-consistency terms as specified.Source: A. Training settings · p. 7

Actor training

  • The actor uses two 128-unit hidden layers and 33,282 parameters, with rollouts terminating at action 135 without bootstrapping.Training-data action bounds constrain vertical motion and gripper rate.Source: A. Training settings · p. 7

Control metrics

  • Force-budget violations are measured at every 0.002 s physics substep, whereas strict lifting checks ten consecutive sampled heights at or above 0.10 m.Force-budgeted success requires both strict lift success and no budget violation.Source: B. Measurement and statistical units · p. 7

Statistical units

  • Bootstrap intervals resample paired whole environments, quantifying test-environment variation conditional on fixed trained policies.The control comparisons use stratified resampling and the same environment draw for every fixed policy.Source: B. Measurement and statistical units · p. 7

Reproducibility

  • The public dataset and ancillary materials are distributed under the stated source and licensing conditions, while the compact model adapts TD-MPC2 components at revision e9f5932.The original recordings are not redistributed with the article.Source: DATA, CODE, AND TOOL USE · p. 7
Loading 2609.09597v1…