Source-linked AI summary
Conservative Offline Robot Policy Learning via Posterior-Transition Reweighting
Wanpeng Zhang, Hao Luo, Sipeng Zheng, Yicheng Feng, Haiweng Xu, Ziheng Xi, Chaoyi Xu, Haoqi Yuan, Zongqing Lu
TL;DR
Heterogeneous robot datasets contain conflicting and unevenly informative demonstrations, making uniform offline post-training unreliable. PTR reweights samples using a conservative posterior over identifiable post-action consequences, and empirical validation spans simulation and 12 real-robot tasks across three embodiments. The method is limited when future observations or sufficiently discriminative representations are unavailable, and it curates training data rather than directly optimizing task success.
Problem
Heterogeneous robot datasets mix embodiments and demonstrations of varying quality, so uniform post-training can average over conflicting or weakly informative supervision.
Method
PTR encodes post-action consequences as latent targets, scores their identification among mismatched candidates, and applies clipped, mixed, self-normalized weights to supervised action regression.
Results
PTR demonstrates general effectiveness on simulation benchmarks and 12 real-robot tasks across three embodiments.
Takeaways & Limitations
PTR reallocates credit to suppress ambiguous supervision while amplifying useful cross-embodiment transfer, without reward labels or a tractable policy likelihood.
Takeaways & Limitations
PTR falls back toward uniform weighting without usable future observations or a sufficiently discriminative learned representation, and it does not directly optimize task success.
Abstract
from arXiv · showhide
Offline post-training adapts a pretrained robot policy to a target dataset by supervised regression on recorded actions. In practice, robot datasets are heterogeneous: they mix embodiments, camera setups, and demonstrations of varying quality, so many trajectories reflect recovery behavior, inconsistent operator skill, or weakly informative supervision. Uniform post-training gives equal credit to all samples and can therefore average over conflicting or low-attribution data. We propose Posterior-Transition Reweighting (PTR), a reward-free and conservative post-training method that decides how much each training sample should influence the supervised update. For each sample, PTR encodes the observed post-action consequence as a latent target, inserts it into a candidate pool of mismatched targets, and uses a separate transition scorer to estimate a softmax identification posterior over target indices. The posterior-to-uniform ratio defines the PTR score, which is converted into a clipped-and-mixed weight and applied to the original action objective through self-normalized weighted regression. This construction requires no tractable policy likelihood and is compatible with both diffusion and flow-matching action heads. Rather than uniformly trusting all recorded supervision, PTR reallocates credit according to how attributable each sample's post-action consequence is under the current representation, improving conservative offline adaptation to heterogeneous robot data.
1 Introduction
PTR addresses heterogeneous offline robot data by reallocating supervised credit according to how attributable each chunk’s post-action consequence is, while conservatively limiting distribution shift. It targets both noisy supervision and selective cross-embodiment transfer without requiring rewards or policy likelihoods.
- Heterogeneous datasets combine embodiments, viewpoints, delays, operator skills, recovery behaviors, and hesitations, producing multimodal action chunks with uneven supervision quality.
- Cross-embodiment data can provide positive transfer when different robots demonstrate the same high-level skill, but embodiment-specific artifacts can cause negative transfer.
- PTR uses observed post-action consequences as a reward-free identification signal for deciding which recorded chunks deserve more training credit.
- Concentrated identification posteriors increase sample weights, whereas diffuse posteriors down-weight ambiguous or suboptimal samples.
- Conservative clipping and mixture constraints bound induced distribution shift while preserving the supervised action objective.
- PTR improves the training distribution’s floor by suppressing poor supervision and its ceiling by leveraging aligned cross-embodiment coverage.
2 Related Work
Related work spans VLA pretraining, offline policy improvement, and conservative data reweighting. PTR differs by applying identification-based, reward-free reweighting to supervised robot post-training.
- VLA systems unify vision, language, and action components into end-to-end robot policies, with open-source models broadening access to generalist pretraining.
- Standard behavioral cloning weights demonstrations equally, while prior data-curation and policy-improvement methods use quality estimators, representation modulation, mutual information, or reward-derived weights.
- Figure 1 depicts PTR augmenting the policy stack with a lightweight scorer and BeliefTokenizer that converts target identification into conservative loss weights.
- PTR uses no reward, value function, or policy gradient, instead adapting identification-based scoring and conservative reweighting principles to reward-free supervised post-training.
3 Preliminaries and Notation
The preliminaries define offline robot samples, the shared representation used by PTR, and the action-head objectives it reweights. Future observations serve as training-time identification targets rather than inference inputs.
- Each offline sample is a five-tuple containing visual observation, state, instruction, action chunk, and future observation.
- Only the current observation, state, and instruction are used at inference, while the future observation is exclusively a training-time identification target.
- The VLA backbone maps observations and instructions to hidden states and pooled context, while a shared action space aligns heterogeneous robots.
- The action head predicts an action chunk from pooled context and state, with flow-matching and diffusion heads supporting corresponding supervised losses.
4 Posterior-Transition Reweighting
PTR reweights offline post-training samples using a reward-free identification posterior over observed post-action consequences. Its conservative scoring and weight controls suppress ambiguous supervision while selectively amplifying attributable cross-embodiment data without requiring policy likelihoods.
- Scoring mechanism: PTR uses a consequence encoder and transition scorer to identify each observed post-action target among mismatched candidates, producing per-sample quality weights without rewards or policy likelihoods.The future consequence is represented in a latent target space and used only for the training-time identification test.
- Score behavior: Uniform posteriors produce zero scores and uniform supervision, whereas concentrated matched-target posteriors yield positive scores; less-supported targets can receive weights below one after clipping.Ambiguous samples remain near the uniform baseline, while clearly attributable samples receive more credit.
- Theoretical foundations: Bayes-optimal identification logits recover a log density ratio between action-conditioned and baseline target distributions, grounding the PTR score statistically.The theoretical chain connects candidate identification to density ratios and KL divergence.
- Compatibility: PTR is compatible with diffusion and flow-matching action heads because density-ratio and KL interpretations apply to the scorer rather than requiring a policy action likelihood.The original supervised action objective is preserved while sample influence is reallocated.
- Selective transfer: Exponential tilting selectively transfers effective weight toward sources whose samples consistently have high PTR scores, including aligned cross-embodiment data.Sources dominated by ambiguous or mismatched trajectories are suppressed or remain closer to their original proportions.
- Conservative control: With default clipping wmin = 0.25, wmax = 4.0, and α = 1, the induced distribution satisfies KL(q∥pD) ≤ log 16 ≈ 2.77 nats.The bound limits how far the reweighted training distribution can move from the original data distribution.
- Adaptive scale control: Adaptive control lowers τscore to approximately 0.03 and increases the hard-negative ratio from 0.0 to 0.50 as the contrastive scorer strengthens.The controller changes sharpness and sampling parameters while preserving the matched-target designation for a fixed candidate set.
5 Experiments
PTR improves robustness and transfer across simulation and real-robot settings by selectively emphasizing informative post-action consequences. Its benefits are strongest under corrupted data and cross-embodiment generalist training, while clipping and EMA targets are key stabilizing components.
- Standard simulation results: PTR outperforms SFT on RoboCasa by 1.4 pp, reaching 55.6 versus 54.2, with gains across Pick&Place, Door/Drawer, and Others.SFT+Belief reaches 54.4, leaving a 1.2 pp gap to PTR.
- Robustness under corrupted training data: Under combined corruption, PTR limits performance losses to 6.8 pp on LIBERO and 9.8 pp on RoboCasa, versus 12.9 pp and 17.8 pp for SFT.The corresponding absolute gains are +5.6 pp and +9.4 pp.
- Real-robot evaluation: Across real-robot suites, specialist PTR improves over SFT on Bimanual and Robust by +11.7 pp each and on Spatial by +3.3 pp, while SFT retains a slight Long-Horizon edge.Across all 12 tasks, PTR averages 67.1% versus 60.8% for SFT.
- Cross-embodiment transfer: In generalist training, PTR narrows the real-robot degradation from 10.8 pp under SFT to 3.3 pp and expands its specialist-to-SFT advantage to +13.8 pp.PTR-Generalist reaches 65.0 on Long-Horizon, surpassing SFT-Specialist at 63.3.
- Cross-embodiment transfer: PTR improves over SFT on 8 of 12 specialist tasks and all 12 generalist tasks, with the generalist checkpoint exceeding specialist SFT on 7 tasks and matching it on 4.The results support broad rather than single-task improvements and selective use of cross-embodiment data.
- Ablations: Removing weight clipping drops LIBERO and RoboCasa by 2.3 pp and 6.5 pp, while removing the EMA target encoder drops them by 2.5 pp and 5.4 pp.These ablations identify bounded weights and a stable target space as the most important stabilizing components.
6 Conclusion and Limitations
PTR conservatively reallocates credit across heterogeneous robot demonstrations using post-action identification, without reward labels or tractable policy likelihood. Clipping, exponential mapping, and self-normalization keep the induced training distribution close to the original data, while limitations concern observations, representation quality, and direct task optimization.
- PTR reallocates credit across heterogeneous demonstrations through a candidate-set identification posterior over post-action consequences.
- Exponential mapping, clipping, and self-normalization make PTR conservative by keeping the induced training distribution close to the original data.
- PTR requires usable post-action observations, depends on representation quality, and curates data rather than directly optimizing task success.
A Proofs
The appendix contains formal proofs for the three propositions stated in the main text.
- The appendix collects formal proofs for three propositions from the main text.
- The proof material is presented as an appendix section supporting the paper’s stated propositions.
- This section serves as the formal-proof repository for the paper’s three propositions.
A.1 Proof of Proposition 1 (Density-ratio form)
The proof derives the Bayes-optimal identification score under the symmetric candidate-set model. Pairwise posterior odds imply that shared classifier logits recover a density-ratio form up to a candidate-independent constant.
- Under the symmetric candidate-set model, the proof defines positive and negative target distributions conditioned on context and consequence.
- Posterior odds between candidate positions are obtained by comparing the positive and negative target distributions across the candidate set.
- A softmax classifier represents these pairwise log-odds, so optimal shared logits equal the log density ratio up to a candidate-independent constant.
A.2 Proof of Proposition 2 (KL lens)
The proof establishes the KL lens for PTR by analyzing empirical negative density-ratio averages. Under bounded density ratios, the empirical quantity converges and remains uniformly bounded, enabling dominated convergence.
- The proof starts from Bayes-optimal logits expressed through the density ratio and substitutes them into the PTR score.
- Negative density ratios have baseline mean one, and their empirical average converges almost surely by the strong law of large numbers.
- The matched candidate contribution vanishes asymptotically, yielding the pointwise limit used in the KL analysis.
- When density ratios are bounded between positive constants, the score is uniformly bounded, so dominated convergence gives the stated result.
A.3 Proof of Proposition 3 (Source reweighting)
The proof models the dataset as a mixture of sources and derives the source marginal of a score-tilted induced distribution. Rewriting the normalizer as a mixture expectation yields the closed-form expression corresponding to Eq. (20).
- Source mixture: The dataset is modeled as a mixture of source distributions, with each source weighted by its prior proportion.Sources may represent embodiments, operators, or domains.
- Assumptions: The proof assumes each source has a finite exponential score moment, ensuring the tilted source marginals are well-defined.
- Source reweighting: Tilting the joint source distribution by exp(J(x)/β) produces an induced distribution whose source marginal depends on source priors and expected tilt factors.
- Derivation: Pulling out the source prior and rewriting the integral as an expectation provides the normalizer used in the closed-form derivation.
- Conclusion: Substituting the normalizing constant yields the closed-form source-marginal expression identified as Eq. (20).
B Training Configuration
Table 6 provides the complete hyperparameter configuration for PTR post-training. These defaults apply to all experiments unless a parameter is varied in the Section 5.6 sensitivity sweep.
- Training Configuration: Table 6 lists the complete hyperparameter configuration used for all PTR experiments.
- Training Configuration: The listed settings are experimental defaults unless a parameter is explicitly varied in the Section 5.6 sensitivity sweep.