PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Modeling Closes the Regression and Generative Policy Gap in Robot Learning

Yuchen Zhou, Jiacheng You, Weikang Wan, Weijun Dong, Yang Gao, Jiayuan Mao

arXiv:2610.12231v1cs.RO

TL;DR

The gap between direct MSE regression and generative robot policies is commonly attributed to multimodal demonstrations, but this paper investigates residual modeling and its optimization effects. It introduces heteroscedastic Student-t action regression, which learns input-dependent scales and reduces the influence of large residuals. HT-Policies achieve success rates competitive with generative baselines across simulation and real-robot evaluations, with improved training and inference efficiency.

  • Problem

    Generative policies often outperform MSE regression, and this gap is commonly attributed to multimodal demonstrations; the paper examines how residual modeling also relates to the gap.

    Source: §1 · p. 1

  • Method

    HT-Policies use a Student-t likelihood with input-dependent residual scales to model heavy-tailed action residuals and downweight gradients from large residuals.

    Source: §4 · p. 5

  • Results

    HT-Policies achieve success rates competitive with generative baselines across simulation and real-robot evaluations while improving training and inference efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HT-Policies offer an efficient direct-regression alternative across architectures and tasks, particularly for tasks with limited multimodality.

    Source: §1 · p. 2, §6 · p. 9

    The approach is not a universal solution: HT-Policies cannot explicitly represent multiple action modes, and the analysis does not directly connect prediction errors to control failures.

    Source: §6 · p. 9

Abstract

from arXiv · show

Learning from demonstration has enabled impressive robot behaviors. A common choice for policy learning is to use diffusion or flow matching (Flow-Policies), which often outperforms direct action regression trained with mean squared error (MSE-Policies). This gap is commonly attributed to multimodal demonstrations. We revisit this gap from the perspective of statistical modeling: how action-prediction residuals shape policy optimization. Our analysis of real-world robot demonstration data reveals substantial state-dependent variation in residual scales and heavier-than-Gaussian tails. While both MSE-Policies and Flow-Policies exhibit heavy-tailed action residuals, their training gradients behave differently: MSE allocates more gradient magnitude to observations with large action residuals, which hurts optimization. Motivated by these findings, we introduce heteroscedastic Student-t action regression (HT-Policies), which learns input-dependent residual scales and reduces the influence of heavy tails. HT-Policies predict action chunks with a single feed-forward pass and can reuse pretrained flow-matching-based policy networks as the backbone. Across four simulation benchmarks and real-robot evaluations, HT-Policies achieves success rates competitive with generative policy baselines, both when trained from scratch and from pretrained vision-language-action and world-action models, despite being faster in training and inference. Together, these findings shed light on the practical advantages of generative objectives in robot learning from demonstrations and offer an efficient direct-regression alternative for a range of architectures and tasks. Project page: https://the-labone.github.io/regression-policy-project/

1 INTRODUCTION

Generative policies often outperform MSE regression, a gap commonly attributed to multimodal demonstrations. This paper instead examines how residual modeling affects optimization and introduces HT-Policies as an efficient direct-regression alternative.

1 INTRODUCTION

  • The performance gap is commonly attributed to multimodality: generative policies can represent multiple action modes, whereas MSE policies may average them.A prior deterministic two-pass method, MIP, closes the gap through iterative refinement, but its explanation remains largely empirical with limited formal characterization and real-data verification.Source: §1 · p. 1
  • The paper analyzes prediction residuals and their effects on policy optimization as an alternative perspective on the gap between direct regression and generative policies.It uses this perspective to motivate direct action regression that can resemble benefits of denoising supervision without iterative inference.Source: §1 · p. 1
  • MSE and Flow policies both have heavy-tailed residuals, but large-residual observations dominate MSE gradients while flow matching spreads gradients more evenly, especially at low noise.The paper links the flow behavior to its denoising training objective.Source: §1 · p. 2
  • HT-Policies achieve success rates competitive with generative baselines across simulation and real-robot evaluations while improving training and inference efficiency.They use a single feed-forward pass and can reuse pretrained flow-matching policy networks as backbones.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses variation in robot demonstrations by modeling it as multimodality or suppressing it as noise, with less emphasis on its statistical structure and effects on learning. Related work also covers iterative refinement and robust or heteroscedastic regression.

2 RELATED WORK

  • Prior approaches model demonstration variation with mixtures, diffusion, or conditional VAEs, or seek to suppress it as undesirable noise.These lines largely study how to model or filter variation empirically rather than its statistical structure and effects on learning.Source: §2 · p. 2
  • Figure 2 presents example tasks from RoboCasa-GR1, Bridge, Fractal, and Tool-Hang.Source: Figure 2 · p. 3
  • MIP attributes its performance to iterative refinement toward the demonstrated-action manifold, but offers limited formal characterization and does not distinguish manifold adherence from generalization.The paper revisits the policy gap through statistical residual modeling.Source: §2 · p. 3
  • Robust losses and Student-t likelihoods address large residuals, while heteroscedastic regression models input-dependent scales; this paper combines these ideas for action learning.Prior work also notes that variance-based reweighting under Gaussian likelihoods can adversely affect mean estimation.Source: §2 · p. 3

3 RESIDUAL STRUCTURE AND OPTIMIZATION

Across the evaluated settings, MSE- and Flow-Policies have heavier-than-Gaussian action residuals, while residual scales vary across observations. MSE gradients concentrate on large-residual observations; flow matching distributes gradients more evenly at low noise through denoising supervision.

3 RESIDUAL STRUCTURE AND OPTIMIZATION

  • Both MSE- and Flow-Policies exhibit heavier-than-Gaussian action residual tails across evaluated settings, including after per-observation normalization.Figure 3 reports scalar residuals exceeding three times their coordinate RMS; Figure 6 compares normalized HG-regression and flow-matching residuals.Source: §3 · p. 3, Figure 3 · p. 3, Figure 6 · p. 4
  • MSE assumes fixed-scale Gaussian residuals, but large-residual observations receive disproportionate optimization weight and may leave other action samples underfit.Source: §3 · p. 3
  • MSE and high-noise flow matching concentrate gradient magnitude on large-residual observations, whereas low-noise flow matching distributes it more evenly across residual deciles.The analysis ranks observations by residual RMS and compares flow gradients across high- and low-noise regimes.Source: §3 · p. 4, Figure 4 · p. 4
  • Residual RMS varies with demonstrated action magnitude and episode progress, so observation-dependent scales can yield heavy-tailed mixtures even when per-observation actions are Gaussian.The scale variation is illustrated on three Bridge tasks.Source: §3 · p. 4, Figure 5 · p. 4
  • Flow matching conditions on a noisy demonstrated action, allowing light-tailed velocity residuals despite heavy-tailed action-prediction residuals.For a baseline predictor, the velocity residual is Gaussian with scale determined by noise and timestep, regardless of the overall action distribution.Source: §3 · pp. 4–5
  • At low-noise timesteps, the learned flow predictor has tails no heavier than the reference bound, and the decreasing reference scale helps explain more balanced gradients.The bound is stated for sufficiently large δ; the paper reports empirical support across evaluated benchmarks and training stages.Source: §3 · p. 5

4 HETEROSCEDASTIC STUDENT-t ACTION REGRESSION

HT-Policies replace fixed-scale Gaussian MSE with a heteroscedastic Student-t objective that predicts both action chunks and input-dependent residual scales. The objective reduces the influence of large residuals and can be used to fine-tune pretrained flow-policy networks for single-pass action prediction.

Heteroscedastic Student-t loss

  • HT-Policies jointly predict action chunks and input-dependent Student-t residual scales, modeling residual tails and scale variation in the regression objective.The predicted scale is the Student-t scale, not the conditional standard deviation used in MSE.Source: §4 · p. 5

Gradient weighting

  • The HT loss gradient depends on predicted scale and error size, with larger scales and larger residuals reducing the influence of a sample.Gradients propagate through the action prediction layer to the shared backbone.Source: §4 · p. 5

Selecting the degrees of freedom

  • Setting ν = cd makes the tail-weighting hyperparameter c independent of action dimension, while smaller c makes weights drop faster as error grows.As c →∞, the objective approaches heteroscedastic Gaussian regression and no longer downweights large-residual samples.Source: §4 · p. 5

Fine-tuning pretrained Flow-Policies

  • A pretrained flow network can serve as the action predictor by reading out its pure-noise endpoint, adding a scale head, and fine-tuning with the HT loss.For the stated flow path, the endpoint predictor under squared error is the conditional mean action minus the noise input; the reversed path uses the corresponding negative-velocity readout.Source: §4 · pp. 5–6

5 EXPERIMENTS AND ANALYSIS

Across scratch-trained simulation and real-robot tasks, HT-Policies perform on par with generative policies, and they improve on MSE-Policies across pretrained fine-tuning settings. The experiments also find more balanced gradient allocation, strong likelihood-performance association, and faster training and inference than flow policies.

5.1 CLOSING THE PERFORMANCE GAP WHEN TRAINING FROM SCRATCH

  • Across RoboMimic and RoboCasa-GR1, HT-Policies are on par with generative policies or outperform them, and they largely close the MSE gap on three real-robot tasks.On RoboMimic they are on par with generative policies and substantially outperform MSE; on RoboCasa-GR1 they outperform both MSE and Flow-Policies.Source: §5.1 · pp. 6–7, Figure 7 · p. 7

5.2 FINE-TUNING PRETRAINED FLOW-POLICIES WITH HT REGRESSION

  • HT-Policies outperform MSE-Policies in all five pretrained fine-tuning settings and Flow-Policies in three, using recipes adopted directly from Flow-Policies.Only the learning rate of the newly added scale prediction head was tuned.Source: §5.2 · p. 7, Table 2 · p. 7

5.3 RESIDUAL MODELING, GRADIENT ALLOCATION, AND POLICY PERFORMANCE

  • HT loss gradients are more balanced across observations than MSE gradients and resemble low-noise flow matching for t > 0.5; across eight objectives, calibrated mean log-likelihood correlates with success rate (Spearman ρ = 0.86; Pearson r = 0.83).HT achieves the highest success rate; Student-t tails or heteroscedastic Gaussian scales alone improve over MSE but neither matches HT.Source: §5.3 · p. 8, Figure 8 · p. 8, Figure 9 · p. 8

5.4 TRAINING AND INFERENCE EFFICIENCY

  • In all four training settings, HT-Policies reach strong task success in fewer steps than Flow-Policies; Cosmos3-Nano has a 17.83× whole-model speedup over Flow-Policies.HT inference uses one feed-forward pass, replacing Flow-Policies' 30-step sampling procedure.Source: §5.4 · pp. 8–9, Figure 10 · p. 8, Table 3 · p. 9

5.5 ABLATION STUDIES

  • Among the compared robust-regression objectives, HT-Policies achieve the best performance when compared by calibrated likelihood and policy performance.Figure 9 summarizes the comparison; the passage points to Table 27 for detailed results and Table 28 for objective definitions.Source: Figure 9 · p. 8, §5.5 · p. 9

6 CONCLUSION

The paper explains the regression–generative policy gap through residual modeling and its effects on optimization, then introduces HT-Policies as an efficient alternative. Its scope is limited: HT-Policies cannot explicitly represent multiple action modes, and the analysis does not connect prediction errors directly to control failures.

6 CONCLUSION

  • The paper argues that the performance gap is better explained by how objectives handle prediction residuals than by generative policies modeling multimodal actions.The analysis links MSE's fixed-scale residual model to gradient imbalance and denoising objectives to better mitigation.Source: §6 · p. 9
  • HT-Policies offer a policy alternative without iterative computation, with improved training and inference efficiency and support for scratch training or pretrained-model fine-tuning.The paper presents them as an efficient alternative rather than a universal solution to policy learning.Source: §6 · p. 9
  • HT-Policies cannot explicitly represent multiple action modes, so the paper frames them as an alternative for tasks with limited multimodality, not a universal solution.The analysis also does not explain why small errors at a few states can catastrophically reduce overall success rate; connecting prediction errors to control failures remains future work.Source: §6 · p. 9

A.1.1 RESIDUAL DEFINITIONS AND SAMPLING

This appendix defines the residual and sampling conventions used for tail analyses, along with their observation and coordinate-scale calibration. The diagnostics analyze training demonstrations, while calibration splits hold out observations for scale estimation.

A.1.1 RESIDUAL DEFINITIONS AND SAMPLING

  • Residuals are computed in normalized action space without augmentation or dropout, and the analysis uses training demonstrations.Calibration splits hold out observations only for scale estimation.Source: §A.1.1 · p. 13
  • The sampled data use complete, evenly spaced action chunks, with the same selected observations and demonstrations across methods within each dataset.GR00T uses ten episodes per task or instruction group and four eight-step chunks per episode; Tool-Hang uses eight 16-step chunks from each of 200 demonstrations.Source: §A.1.1 · p. 13
  • Residuals subtract demonstrated actions from predicted means for MSE and HG policies, but from each sampled action for Flow-Policies.The Flow-Policy residuals are not computed after averaging generated actions; channel inclusion also varies by dataset.Source: §A.1.1 · p. 13
  • Fixed coordinate RMS values are estimated from the other four folds for each evaluated fold, without mean subtraction, trimming, or rescaling on the evaluated fold.The tail probability pools valid scalar coordinates, and Figure 3 reports the fraction exceeding three times coordinate RMS, compared with a two-sided Gaussian reference of approximately 0.27%.Source: §A.1.1 · p. 13
  • For HG-Policies, observation-dependent normalization uses the predicted scale; Flow-Policy spread estimates use separate sample banks to avoid an evaluated draw increasing its own scale.For the own-spread curve, residuals are divided by the estimated local spread before applying Eq. (4).Source: §A.1.1 · pp. 13–14

A.1.2 PER-OBSERVATION PARAMETER GRADIENTS

The appendix measures per-observation parameter-gradient contributions by residual size and, for flow matching, by noise and timestep group. It specifies the parameter scopes and gradient aggregation conventions used in those diagnostics.

A.1.2 PER-OBSERVATION PARAMETER GRADIENTS

  • Gradients are computed from individual observation losses before optimization, clipping, or weight decay, with parameter scopes tailored to each architecture.For HT, the mean and scale gradient paths are combined before computing the norm, retaining their cross term.Source: §A.1.2 · p. 14
  • Observations are ranked by each policy’s action-chunk residual RMS and split into ten nearly equal groups to compare gradient contributions across residual sizes.MSE and HT use predicted means, while Flow-Policies use the complete native sampler starting from zero initial noise.Source: §A.1.2 · p. 14
  • The reported contribution is not the norm of an accumulated batch gradient: norms are computed before averaging draws, and observations are summed afterward.Figure 12 plots relative gradient contribution by action-residual decile, separating high- and low-noise flow-matching gradient RMS.Source: §A.1.2 · p. 14, Figure 12 · p. 14
  • Flow-matching gradients use native velocity MSE at 16 equally weighted midpoint quantiles, with independent Gaussian action noise for each observation and draw.High- and low-noise curves retain representative timesteps below or above 0.5; each quantile interval represents probability mass 1/16.Source: §A.1.2 · p. 14, Table 6 · p. 15

A.1.3 GAUSSIAN TAIL BOUNDS FOR FLOW-MATCHING VELOCITY RESIDUALS

The appendix tests a Gaussian tail-bound hypothesis for flow-matching velocity residuals at fixed low-noise timesteps. Across the reported datasets and RoboCasa-GR1 training stages, the largest empirical tail probability stays below the reference over the plotted range.

A.1.3 GAUSSIAN TAIL BOUNDS FOR FLOW-MATCHING VELOCITY RESIDUALS

  • The test compares empirical tails of t∥rθ∥2/√d at six fixed low-noise timesteps, where t > 0.5, against a reference.The residual is the flow-matching velocity residual defined in the appendix.Source: §A.1.3 · p. 15
  • Across datasets and RoboCasa-GR1 training stages, the largest empirical tail probability remains below the Gaussian reference over the plotted range.The comparisons cover GR00T on RoboCasa-GR1, Bridge, and Fractal, and LIBERO results using π0.5; training stages span 5k to 60k updates.Source: §A.1.3 · p. 15, Figure 13 · p. 15, Figure 14 · p. 15
  • For RoboCasa-GR1, the training-stage comparison uses d = 232 and reports models after 5k, 10k, 20k, 40k, and 60k updates.The curves use the same convention as the dataset comparison: individual timesteps, their pointwise maximum, and a Gaussian norm reference.Source: Figure 14 · p. 15

A.1.4 GRADIENT ALLOCATION ACROSS TRAINING STAGES

This section examines residual-tail gradient allocation across training stages and reports supplementary evaluations of objectives, degrees of freedom, calibration, training progress, inference timing, and policy behavior.

A.1.4 GRADIENT ALLOCATION ACROSS TRAINING STAGES

  • MSE concentrates disproportionate gradient share on the largest residuals at every evaluated stage, while flow and HT distribute gradients more evenly.HT already reduces residual-tail gradient concentration at 5k updates, before its strongest performance at 40k and 60k, and the effect persists across stages.Source: §A.1.4 · pp. 15–16, Figure 15 · p. 16

A.2 ACTION AVERAGING IN FLOW-POLICIES

  • Action averaging caused no decrease in benchmark-average success across the three tested evaluations, though individual tasks could improve or decline.The single-run comparisons do not establish a consistent improvement across evaluation seeds.Source: §A.2 · p. 16, Table 7 · p. 17

A.3 EXPLORATION WITH CONTROLLABLE SAMPLING NOISE

  • Both stochastic HT-Policy variants reached 100% pass@8 across tested noise strengths, and modest noise gave pass@k comparable to Flow-Policies.The evaluation measured the estimated probability of at least one successful rollout within k attempts on state-based Tool-Hang.Source: §A.3 · p. 18, Table 11 · p. 18

A.4.1 REAL-ROBOT EXPERIMENTS

  • The real-robot Push-T, Peel-Note, and Insert-T experiments used 100 demonstrations per task, U-Net policies trained from scratch, and 50 evaluation rollouts per task.All methods used the same demonstrations and gripper; each policy call predicted 16 actions, of which the first eight were executed before replanning.Source: §A.4.1 · p. 19

A.4.2 SCALE PREDICTION

  • Each HT-Policy jointly trains a positive scale σθ(o) shared across valid action coordinates in an action chunk, but uses only predicted actions at inference.The scale head is trained jointly with the action predictor through the HT loss.Source: §A.4.2 · p. 19

A.4.5 FROM-SCRATCH POLICIES ON ROBOCASA-GR1

  • GR00T fine-tuning freezes its pretrained vision-language backbone, while from-scratch training learns the action head from fixed VLM features.The from-scratch experiments otherwise use the same GR00T N1.7 architecture, data, optimization, and evaluation protocols.Source: §A.4.4 · p. 20, §A.4.5 · p. 21

A.4.6 COSMOS3-NANO ON LIBERO-10

  • Cosmos3-Nano retains joint video and action training, with video supervision affecting action prediction through shared generation components.At inference, one joint forward pass predicts actions without future ground-truth video or video generation; video noise can still make predictions stochastic.Source: §A.4.6 · p. 21

A.5.2 COMPARISON WITH MIP

  • HT-Policies score 91.3%/85.5% versus MIP’s 91.5%/86.1% best/last5 averages across ten tasks and both observation types.HT scores slightly lower on state observations and slightly higher on image observations.Source: Table 17 · p. 23, §A.5.2 · p. 23, Table 18 · p. 24

A.6 TRAINING CURVES AND INFERENCE TIMING

  • Figure 10 tracks success against optimizer updates, using single-seed GR1 and WidowX evaluations and multi-seed averages for Transport; Table 26 reports A800 inference-efficiency measures.A800 batch-one latency measurements use BF16 mixed precision; whole-model latency includes observation encoding and action prediction from preprocessed GPU inputs.Source: §A.6 · p. 28, Table 26 · p. 29

A.7.1 REGRESSION OBJECTIVES

  • Learning observation-dependent scales improved every tested objective; robust losses beat MSE except fixed-scale L1, while RMSE improved on L1 under both scale settings.L1 penalizes coordinates independently, whereas RMSE uses a joint residual norm, making each coordinate’s gradient depend on the full residual vector.Source: §A.7.1 · pp. 29–30
  • 51.50 for heteroscedastic Student-t (HT) exceeded 40.70 for heteroscedastic Huber, and HT achieved the highest success among tested objectives.The comparison used GR00T N1.7 on RoboCasa-GR1 after 60,000 updates; each model had a single evaluation seed.Source: Table 27 · p. 29, §A.7.1 · p. 29

A.7.2 LIKELIHOOD CALIBRATION

  • Likelihood calibration fits one global scale multiplier per frozen policy while preserving relative scales, and it does not use or change policy success rates.Scoring uses joint log-likelihood with log-scale terms and normalization constants, with radial densities for RMSE and Huber; folds group all passes from each episode together.Source: §A.7.2 · p. 30

A.7.3 DEGREES-OF-FREEDOM SWEEP SETTINGS

  • ν = 4d gave the highest tested success in each degrees-of-freedom sweep setting; all variants learned observation-dependent scales.The points use single-seed evaluations, so their success rates may differ from Table 2’s 10-seed means.Source: Table 29 · p. 30, §A.7.3 · p. 30

Related papers

Loading 2610.12231v1…