Compact Visuotactile World Models for Lifting: Prediction, Reward Alignment, and Force Constraints

Qinzhen Ma; Sida Peng

Source-linked AI summary

Compact Visuotactile World Models for Lifting: Prediction, Reward Alignment, and Force Constraints

Qinzhen Ma, Sida Peng

arXiv:2609.09597v1Sep 9, 2026cs.ROcs.AI

TL;DR

The paper asks whether tactile prediction improvements support reliable, force-constrained manipulation decisions. It combines a compact visuotactile world model, trajectory-level calibration, and behavior-initialized imagined control, finding that sensing and reward revisions improve selected outcomes without resolving force-constrained control. The evidence remains bounded to public sensing records and simulator execution without demonstrated transfer.

Problem
Contact prediction is useful for manipulation only if improved forecasts support effective decisions under trajectory reliability and force constraints.
Method
The study combines a compact randomly initialized visuotactile world model, trajectory-level uncertainty calibration, and behavior-initialized actor-critic learning in imagination.
Results
Matched reward revision improved executed lifting on fresh environments, but force-budgeted success remained below force feedback, while trajectory calibration exposed tail errors and aligned touch could lose to persistence.
Takeaways & Limitations
The comparisons distinguish improvements in sensing and task reward from improvements in force-constrained control.
Takeaways & Limitations
The evidence uses one rigid simulated geometry, simulated contact-force aggregates, public GelSight sensing records, and no demonstrated transfer between sensing data and simulator policy.

Abstract

from arXiv · show

Accurate contact prediction is useful for robotic manipulation only if it supports effective decisions. We investigate this connection using a compact, randomly initialized visuotactile world model, trajectory-level uncertainty calibration, and behavior-initialized actor-critic learning in imagination. On 160 MuJoCo Lift episodes, adding touch reduces endpoint-force prediction error from 1.058 to 0.228 N and interval-peak error from 2.724 to 0.523 N across three training seeds. However, tactile persistence achieves lower errors of 0.095 and 0.498 N, respectively. Two exploratory control rounds comprise 680 executions on 40 independent test initial conditions. A matched reward revision on fresh test environments increases in-distribution 10 cm lifting success from 20.0% to 93.3%, while success within an 8 N per-finger budget reaches only 33.3%, compared with 70.0% for force feedback. Calibration margins reduce force violations at the cost of task completion. In a separate study of public GelSight recordings, a force regressor achieves 0.04234 N error, but frame-level calibration covers only 15.80% of complete trajectories; trajectory-level calibration raises this to 87.36% at nominal 90% coverage. Together, these findings distinguish improvements in sensing and task reward from improvements in force-constrained control. The evidence is limited to public sensing records and simulator execution, without a demonstrated transfer between them.

I. INTRODUCTION

The paper tests whether tactile information remains useful as manipulation systems move from contact-force prediction to calibrated trajectory reliability and executed control. Its matched comparisons separate sensing gains, reward alignment, and force-constrained decision quality across simulator and public-data branches.

Motivation: Contact-rich manipulation couples perception, prediction, and control, but lower prediction error need not produce a better policy.Persistence can predict slowly varying contacts well, while infrequent force peaks may dominate constraints despite small average error.
Research question: The study evaluates grasp closure and lifting through force forecasts, trajectory-level reliability, and explicit force budgets rather than inferring material properties or damage thresholds.Before lift-off, gripper force does not determine object mass, and sticking contact does not uniquely identify friction.
Experimental design: The simulator branch uses contact-force aggregates for action-conditioned forecasting and frozen-world-model control, while the public-data branch tests optical tactile sensing and trajectory calibration.The branches address prediction reliability but are not a sensor-transfer or sim-to-real pipeline.
Contributions: The paper reports three findings: aligned touch can outperform vision-only input yet lose to persistence, trajectory calibration reveals tail errors, and reward revision improves lifting while remaining inferior to force feedback under force budgets.These comparisons identify a correctable reward mismatch alongside remaining dynamics and constraint limitations.
Positioning: Visuotactile world modeling and contact prediction are established areas, so the paper focuses on linking compact contact forecasts, trajectory-level error accounting, and decision usefulness.The paper adapts existing modeling and calibration components rather than claiming modality fusion or contact prediction as novel.

III. METHOD

The method encodes short histories of vision, touch, and proprioception into a compact latent state, then predicts task-relevant contact and environment quantities over imagined transitions. Training uses matched vision-only and visuotactile models with identical parameter counts and no privileged state inputs.

Inputs: The observation history contains three RGB frames, three tactile observations, and three proprioceptive observations, with simulated touch represented by per-finger normal and tangential contact-force components.These tactile inputs are contact-force aggregates rather than optical tactile images.
Representation: A convolutional visual encoder, separate tactile and proprioceptive MLPs, and a projection layer produce a 128-dimensional SimNorm latent.The visual, tactile, and proprioceptive features have dimensions 256, 64, and 32, respectively.
Prediction targets: The 652,157-parameter transition model predicts endpoint force, object height, support, contact, reward, an RGB target, and per-finger interval-maximum normal force.Interval-peak heads capture transient forces that endpoint predictions could miss; the model is randomly initialized and receives no privileged object state.
Optimization: Five-step training unrolls combine Huber regression, classification, image, and representation-consistency losses, with future observations used only as targets.Vision-only and visuotactile variants have identical parameter counts, and normalization uses training episodes exclusively.

B. Contact-envelope calibration

The contact-envelope method calibrates trajectory-level residuals for force and height forecasts and converts them into upper bounds for a contact analysis envelope. Its coverage guarantee is marginal under fixed-predictor exchangeability, not robust to distribution shift or adaptive closed-loop policies.

Force representation: The public-force experiment represents contact force as F = (Fx, Fy, Fn), with compression-positive normal force and analysis conditions for friction parameters and normal-force ceilings.These parameters are not identified friction coefficients or measured damage thresholds.
Calibration: Trajectory calibration takes the maximum residual over each complete contact trajectory and applies a corrected finite-sample quantile to obtain simultaneous bounds.The corrected quantile uses rank ⌈(n+1)(1−α)⌉, with negative quantiles clipped to zero.
Envelope test: The resulting upper bounds are predicted contact quantities plus the calibrated residual quantile, and a frame passes only when both specified limits are satisfied.A coordinate-box baseline instead calibrates maximum absolute coordinate-time error and propagates that box through the contact functions.
Scope: Under a fixed predictor and exchangeable calibration and test trajectories, the construction provides marginal simultaneous coverage but no guarantee under shape shift, adaptive control distributions, or new closed-loop policies.The method therefore reports acceptance and conditional violations separately.
Simulator calibration: Simulator calibration maximizes per-finger interval-peak underprediction and height residuals over evaluated windows and five future steps, while preserving the interval peak rather than substituting endpoint force.These bounds concern forecasts along recorded actions, not all counterfactual actions.

C. Imagined policy learning

The policy stage freezes the learned world model, initializes an actor from behavior cloning, and optimizes imagined rewards over short rollouts. Force penalties and behavior-prior regularization constrain the actor, but independent simulator evaluation remains necessary because optimization can exploit model errors and shift the state distribution.

Actor initialization: A two-output actor is behavior-cloned, then controls vertical motion and gripper opening or closing after a scripted approach.Action mapping stays within training-action ranges, limiting extrapolation without guaranteeing state-distribution support.
Imagined optimization: The actor rolls the frozen model forward for five steps and maximizes predicted reward penalized by interval-peak force excess, action magnitude, and deviation from the behavior prior.A value network learns detached imagined-return targets with an EMA target critic for terminal bootstrapping.
Force constraint: The imagined stage uses per-finger predicted peaks, an 8 N force budget, and a calibration margin to define force-aware rewards.Ordinary imagined RL uses q = 0, whereas the margin variant uses the fixed calibration allowance.
Training procedure: Training uses 3,520 encoded starts, 1,000 behavior-cloning updates, and 1,500 imagined updates without online data collection or world-model retraining.The actor and critic learning rates are 2 × 10^-4 and 3 × 10^-4, respectively.
Evaluation: Policies are evaluated independently in simulation because optimization can exploit model errors and alter the state distribution used for recording-based calibration.The actor-critic objective also differs from the official TDMPC2 algorithm.

IV. PUBLIC TACTILE-FORCE STUDY

The public tactile-force study evaluates force prediction on trajectory-separated GelSight recordings, including held-out shapes and batch conditions. It uses a CNN regressor and reports both predictive accuracy and cross-shape generalization.

Data and protocol: 102,628 paired frames from 1,513 loading/shear trajectories comprise the public GelSight dataset.GelSight Mini images are paired with ATI Nano17 force measurements.
Data and protocol: 453 training, 75 validation, 114 calibration, and 116 testing trajectories are used for sphere batches 1–5, with batch 6 and flat/sharp trajectories held out.Entire trajectories, rather than adjacent frames, define the partitions.
Model: A 361,091-parameter CNN is trained for force regression across three shared data splits and seeds.Training uses AdamW with validation-based model selection, while force statistics use only training labels.
Results: 0.04234 ± 0.00306 N sphere MAE and 0.03915 ± 0.00351 N batch-6 MAE are reported across seeds.A grouped-bootstrap 95% interval for sphere MAE is [0.03823, 0.04675] N.
Results: The regressor performs worse than the constant baseline on held-out shapes, indicating poor cross-shape generalization in this protocol.Batch identifiers alone do not establish crossday or cross-sensor transfer.

C. Frame reliability does not imply trajectory reliability

The calibration analysis compares frame-level and trajectory-level reliability, showing that tighter direct trajectory calibration can trade observed coverage for a smaller allowance. The analysis also distinguishes offline force classification from executed control.

Calibration coverage: α = 0.10 is the nominal level for joint contact-function coverage.Table II reports coverage under this nominal calibration setting.
Calibration coverage: Direct trajectory calibration narrows the allowance relative to box propagation, but its observed trajectory coverage is lower and does not reach nominal 90%.The grouped-bootstrap 95% interval is [81.90%, 92.24%].
Acceptance analysis: For µ = 0.8 and Fn ≤2 N, box propagation accepts no frames, whereas direct trajectory calibration accepts 18.84%.Direct calibration has 0.192% conditional proxy-constraint violations in this illustrative envelope.
Forecast evaluation: The world-model forecasts use 160 episodes split into training, validation, calibration, and ID/OOD test sets.Evaluation follows recorded actions and compares forecasts with actual simulator states.
Forecast evaluation: The reported forecast errors are averaged over horizons one through five and summarized as mean ± sample standard deviation across three training seeds.This aggregation defines the comparison reported in Table III.

B. Forecasts and the persistence counterexample

Aligned touch improves force prediction over vision, but persistence can outperform learned dynamics on key force metrics. Calibration and force-budget analysis reveal that low average error does not ensure useful or certifiable control.

Forecast accuracy: Touch reduces both endpoint and peak-force errors relative to vision across splits, using distinct force targets.Endpoint force averages six signed/contact components, while peak force averages two normal-force maxima.
Contact-restricted evaluation: 0.409/1.131 N visuotactile contact-restricted endpoint MAE remains below vision’s 2.087/2.548 N for ID/OOD.All-phase error comparisons can be distorted by different contact mixes, so they do not establish better contact generalization.
Persistence counterexample: 0.095 N ID endpoint MAE for tactile persistence outperforms the learned world model, whose ID peak MAE is 0.523 N versus 0.498 N for persistence.Persistence also reaches 0.203 N OOD endpoint MAE, while learned dynamics improve OOD peak MAE to 0.849 N versus 0.966 N.
Calibration limits: 80%, 85%, and 90% ID joint episode coverage falls to 65%, 50%, and 45% OOD across three visuotactile calibration runs.Peak allowances are 13.204, 6.945, and 9.438 N, and the small number of independent episodes limits precision.
Force constraints: Two calibrated allowances exceed the entire 8 N per-finger budget before adding any predicted peak, making hard admission reject those forecasts.A soft calibration penalty is not a safety guarantee or evidence that abstention completes the task.

D. Matched interaction protocol

The matched interaction protocol evaluates fixed controllers on paired fresh simulator environments using strict height and force-budget success criteria. It also exposes feasibility and reward-alignment limits that complicate interpreting initial control results.

Task criteria: 0.10 m relative height for ten consecutive 20 Hz observations defines strict lifting success before lowering.Force-budget success additionally requires every finger’s interval peak to remain at or below 8 N throughout the complete 150-action episode.
OOD feasibility: All OOD strict and joint successes are zero in the initial-round table.This OOD group is a feasibility stress test: under the stated ideal pinch model, the 8 N budget cannot support the lightest OOD object.
Evaluation design: 20 new environments—ten ID and ten OOD—are shared across paired controller evaluations, with learned methods averaging three fixed policies.The sample is 20 independent environments, not 60, and no controller is selected from test performance.
Reward alignment: 55.943 to 61.365 native return for visuotactile RL accompanies a strict lift-success decline from 15.0% to 6.7%.The native shaped reward does not encode the 10 cm, ten-observation criterion, so higher return does not track the strict task metric.
Uncertainty: −20.0 percentage points is the margin-RL-minus-feedback joint-success difference, with a 95% interval of [−33.3, −6.7].Broad intervals and only ten environments per domain limit inference about training-procedure comparisons.

F. Initial executed-distribution diagnostic

The exploratory reward revision was evaluated on fresh matched environments, improving ID strict lifting but not force regulation. Retrospective coverage varied across induced policy state distributions, so these results do not establish online safety.

Matched reward revision: ID strict success rose by 73.3 percentage points under height reward, with a fixed-policy environment-bootstrap 95% interval of [66.7, 83.3].Height-reward RL achieved [9] successes out of ten, versus [2, 4, 0] for matched native-reward policies.
Matched reward revision: Joint success increased by 23.3 points, while mean episode peak increased by 0.432 N.The reported intervals were [0.0, 50.0] for joint success and [0.175, 0.711] for mean episode peak.
Matched reward revision: Height-reward joint success remained 36.7 points below force feedback, indicating improved task attainment rather than improved force regulation.All OOD strict and joint successes remained zero in the follow-up.
Calibration diagnostic: Retrospective ID joint forecast coverage differed across policy-induced state distributions despite unchanged recording calibration.Coverage was [10%, 80%, 80%] for height RL, [0%, 60%, 50%] for native RL, and [100%, 100%, 100%] for BC.

H. Computation and scope

The study’s executed-control evidence is exploratory and simulator-bound, with limited independent test conditions and unresolved force-constrained-control questions. The authors therefore call for broader evaluation and stronger baselines before claiming reliable force-constrained control.

Computation and evaluation scope: 680 simulator executions covered only 40 independent test initial conditions across both control rounds.The follow-up reused frozen world models rather than collecting new training interactions.
Evidence boundaries: Touch improved local force prediction, but persistence outperformed learned dynamics on several force metrics and unfavorable shape transfer remained.These findings prevent attributing the modality gain to useful action-conditioned dynamics alone.
Executed control: A matched exploratory height-reward revision improved lifting but did not outperform force feedback on force-budgeted success and increased mean peak force.Calibration margins reduced violations while producing poor task completion, and recording-based forecast coverage degraded under learned policies.
Future evaluation: Fresh confirmatory evaluation, stronger dynamics and reactive baselines, and more diverse feasible episodes remain necessary to test reliable force-constrained control.The conclusion specifically limits the current evidence to the demonstrated exploratory simulator setting.
Scope limitations: The study uses one rigid object geometry, simulated contact-force aggregates, and a small number of independent environments.The reward follow-up also lacks a vision-only height-reward ablation, and public GelSight sensing results were not transferred to the simulator policy.

APPENDIX IMPLEMENTATION AND REPRODUCIBILITY DETAILS

The implementation combines a compact force-predicting world model, actor training, trajectory-level metrics, and simulator-specific control evaluation. Reproducibility depends on explicit preprocessing, calibration, bootstrap, and measurement conventions.

Public-force regressor: The public-force regressor processes 3×64×48 tactile images through four strided convolutions and predicts three force components.Inputs are normalized from bytes to [−1, 1], and checkpoint selection uses minimum validation MAE.
Training objective: The world-model objective combines force, height, reward, support, contact, image, and latent-consistency losses with weighted terms including 0.5Lr + 0.5LI + 20Lz.The losses use normalized Huber, binary cross-entropy, RGB mean-squared error, and latent-consistency terms as specified.
Actor training: The actor uses two 128-unit hidden layers and 33,282 parameters, with rollouts terminating at action 135 without bootstrapping.Training-data action bounds constrain vertical motion and gripper rate.
Control metrics: Force-budget violations are measured at every 0.002 s physics substep, whereas strict lifting checks ten consecutive sampled heights at or above 0.10 m.Force-budgeted success requires both strict lift success and no budget violation.
Statistical units: Bootstrap intervals resample paired whole environments, quantifying test-environment variation conditional on fixed trained policies.The control comparisons use stratified resampling and the same environment draw for every fixed policy.
Reproducibility: The public dataset and ancillary materials are distributed under the stated source and licensing conditions, while the compact model adapts TD-MPC2 components at revision e9f5932.The original recordings are not redistributed with the article.