PaperlayerView exact revision on arXiv

Source-linked AI summary

RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

Dongchi Huang, Hongyin Zhang, Bohan Hou, Siteng Huang, Zhian Su, Hang Guo, Tong Lu, Zhaofeng Xu, Jiahao Tang, Jianfei Yang, Donglin Wang, Peixi Peng, Mingxiu Chen, Deli Zhao, Xin Li

arXiv:2608.09853v1cs.ROcs.CVcs.LG

TL;DR

Scaling general-purpose robot reward models across heterogeneous data remains difficult because existing supervision depends on task-specific anchors. RynnValue instead learns goal-conditioned temporal distance from timestamps, outperforming preference-supervised methods and improving real-world policy learning online and offline.

  • Problem

    Existing robot reward models rely on task-specific preferences, demonstrations, comparisons, or normalized progress, limiting reusable value supervision across heterogeneous data.

  • Method

    RynnValue learns goal-conditioned temporal distance from timestamp-derived labels across heterogeneous robot data, using sampling, order shuffling, and value-isolation attention for robust value learning.

  • Results

    0.675 average Kendall’s tau_a on RBM-EVAL-OOD surpasses the preference-supervised state of the art (0.655), while potential-based shaping improves online and offline real-world policy learning.

  • Takeaways & Limitations

    Temporal distance provides a scalable, preference-free supervision target and practical dense-reward interface for generalist robot policies.

  • Takeaways & Limitations

    RynnValue currently estimates temporal distance from short windows, assumes an approximately minimum-time objective, and has limited demonstrated scope across embodiments and streaming use.

Abstract

from arXiv · show

General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.

1 Introduction

RynnValue reframes general-purpose robotic reward modeling as learning language-conditioned temporal cost-to-go, a scalable target derived from timestamps across heterogeneous data. Its shortcut-suppression designs and potential-based reward interface enable strong out-of-distribution ranking and real-world policy-learning gains without preference labels.

  • Problem and framing: Temporal distance reframes reward modeling as predicting directed, language-conditioned cost-to-go rather than trajectory-internal progress or preference anchors.Under a minimum-time objective, temporal distance corresponds to hitting-time cost-to-go and provides a reusable value interface.
  • Scalable supervision: 7,000+ hours and roughly 3M instruction-conditioned clips are labeled from timestamps across real-world, simulated, and egocentric trajectories.The corpus spans diverse embodiments, viewpoints, and task families, with subtask segmentation and cutoff relabeling supporting the data recipe.
  • Shortcut suppression: Temporal-order shuffling and value-isolation attention suppress shortcuts that could make predictions insensitive to failures, regressions, and non-monotonic events.These designs ground temporal-value predictions in visual evidence and are paired with dual distributional heads for absolute and relative temporal values.
  • Evaluation: 0.675 average Kendall’s τa on RBM-EVAL-OOD exceeds the preference-supervised state of the art (0.655) and a progress-only counterpart (0.292) without preference labels.RynnValue generalizes zero-shot to unseen tasks, embodiments, and viewpoints.
  • Downstream reward interface: 20.0 percentage points online and 18.7 percentage points offline are added to real-world policy success through potential-based shaping, reaching 72.5% and 82.5%, respectively.The corresponding baselines are 52.5% online and 63.8% offline.

2 Model Architecture

RynnValue is a language-conditioned visual value model that jointly estimates absolute temporal distance and relative temporal displacement from sequences of robot observations. Its architecture uses grouped temporal queries, continuous distributional readouts, value-isolation attention, and auxiliary language-based video verification.

  • Temporal predictions: RynnValue jointly predicts absolute temporal distance and relative temporal displacement for each observation, using K = 8 sampled observations per video.The absolute target measures remaining time to task completion, while the relative target measures signed displacement between adjacent presented observations.
  • Grouped temporal queries: Grouped temporal queries replace a single bottleneck token with N repeated tokens that capture complementary visual cues for each temporal prediction.The model uses N = 8 repeated tokens for both absolute- and relative-value query groups.
  • Continuous temporal readouts: Two specialized distributional heads produce continuous absolute and relative temporal estimates, while the original LM head remains dedicated to language analysis and task verification.Absolute values cover [0,512] seconds and relative values cover [-256,256] seconds, each discretized into 256 bins before continuous decoding.
  • Value-isolation attention: Value-isolation attention forces temporal estimates to rely on the instruction and corresponding visual context rather than previously exposed value tokens.The constraint prevents indirect propagation through later language or visual representations, reducing shortcuts that ignore regressions and failures.
  • Natural-language video analysis and task verification: After processing the visual sequence, the model generates a video description followed by task-matching and task-success judgments for interpretable verification.These auxiliary language outputs are generated from the complete visual context and are not fed back into the temporal heads.

3 Learning Temporal Distance as a Reward Interface

RynnValue converts heterogeneous robot trajectories into unified temporal-distance supervision and trains value predictions with sampling and attention mechanisms that suppress shortcut learning. It then transforms predicted temporal distance into a dense potential for downstream policy learning while preserving a task-independent temporal scale.

  • 3.1 Data and temporal-distance relabeling: Over 7,000 hours of heterogeneous robot, simulation, and egocentric data are converted into over 3M instruction-conditioned trajectory segments with temporal-distance supervision.The corpus spans diverse embodiments, viewpoints, task boundaries, and execution speeds, beginning from 1.67M original episodes.
  • 3.1 Data and temporal-distance relabeling: Temporal-distance labels use timestamps: observations before the completion cutoff receive remaining time, while observations at or after it receive zero.This yields unified dense supervision without dataset-specific progress normalization.
  • 3.2 Distributional training recipe: Random temporal sampling and temporal-order shuffling suppress shortcuts based on sampling intervals and sequence position, grounding predictions in visual evidence and task semantics.Each clip samples K = 8 observations at irregular timestamps, while shuffled sequences can include backward transitions and negative relative temporal targets.
  • 3.2 Distributional training recipe: The absolute objective anchors observations to task completion, whereas the relative objective captures forward and backward temporal displacement between consecutively presented observations.Both temporal targets are trained jointly with a causal language-modeling objective using distributional temporal bins.
  • 3.3 Reward interface: RynnValue converts non-negative temporal distance into an observation potential whose negative values approach zero at task completion, preserving the temporal scale instead of task-specific [0, 1] normalization.A large temporal-distance estimate indicates that more time is required, while vt = 0 corresponds to the predicted completion boundary.

4 Experiments

Experiments show that RynnValue achieves strong out-of-distribution value ranking, benefits from diverse data and shortcut-suppression objectives, tracks regressions and completion more reliably, and improves real-world reinforcement-learning success.

  • Intrinsic value quality: RynnValue-8B achieves the highest average Kendall’s τa of 0.675, exceeding the 0.655 obtained by fully preference-supervised Robometer.RynnValue-4B reaches 0.670 and ranks first on UTD SO101, while RynnValue-8B ranks first on USC Franka, USC Koch, and USC Trossen.
  • Ablations: Removing temporal-order shuffling reduces average Kendall’s τa from 0.675 to 0.189, while uniform sampling reduces it to 0.379.Removing value-isolation attention reduces performance to 0.482, supporting complementary shortcut suppression.
  • Scaling: Increasing training-task diversity reduces error monotonically, whereas adding more within-task episodes produces an early error plateau.The result indicates that heterogeneous task coverage contributes more than simply increasing episode volume within existing tasks.
  • Temporal-value behavior: RynnValue decreases its potential more strongly during regressions, rises more steadily after recovery, and remains responsive to late disturbances near completion than Robometer.Joint absolute and relative temporal-value learning supports global remaining-cost estimation and local temporal-change detection.
  • Real-world reinforcement learning: RynnValue reaches a 72.5% average online success rate, outperforming Robometer at 52.5% and sparse rewards at 48.8% across four real-world tasks.In offline RL, RynnValue also consistently achieves the highest success rate across all four tasks.
  • Real-world reinforcement learning: On Bread Basket Placement, RynnValue achieves 100% offline success using 16.8 action chunks, compared with Robometer’s 80% success and 18.9 chunks.RynnValue also reaches 90% success with 14.9 chunks on both Steak Serving with a Spatula and Box-in-Drawer Placement, and solves tasks with no successful SFT executions.

5 Conclusion and Future Works · Appendix

RynnValue uses timestamp-derived temporal distance as a preference-free, cross-dataset value target for robotic manipulation. The paper reports robust zero-shot generalization and improved real-world reward-based policy learning, while identifying longer horizons, richer value semantics, and broader embodiments as future directions.

  • 5 Conclusion and Future Works: RynnValue replaces trajectory-internal progress supervision with directed, goal-conditioned temporal distance.This reframes value learning around a state’s cost-to-go to the language-specified goal.
  • 5 Conclusion and Future Works: Timestamp-derived labels create a single preference-free value interface across heterogeneous data, embodiments, viewpoints, and task durations.The approach avoids dataset-specific progress normalization.
  • 5 Conclusion and Future Works: Random temporal sampling and temporal-order shuffling make temporal-value learning more robust at scale.The supplied passage identifies these components as part of the scaling recipe.
  • 5 Conclusion and Future Works: Without preference labels, RynnValue surpasses the fully preference-supervised state of the art on RBM-EVAL-OOD and nearly doubles a progress-only counterpart.The comparison is reported alongside zero-shot generalization across unseen tasks, embodiments, and viewpoints.
  • 5 Conclusion and Future Works: Potential-based shaping converts RynnValue into dense rewards that improve online and offline real-world policy learning over strong reward-model and sparse-reward baselines.The reported gains apply to both online and offline settings.
  • 5 Conclusion and Future Works: Future work targets broader temporal horizons, richer value semantics, and more diverse embodiments.The current model estimates temporal distance from a short window of sampled observations.
  • 5 Conclusion and Future Works: Longer-horizon modeling and streaming inference could broaden RynnValue’s use as an online reward source.This direction addresses the limitation of short-window temporal-distance estimation.
  • 5 Conclusion and Future Works: Incorporating energy, safety, or precision costs could extend RynnValue beyond its approximately minimum-time objective.These task-specific costs are proposed as richer value semantics.

A Data Curation for Heterogeneous Robot Corpora

RynnValue uses source-aware curation to retain actionable, manipulation-relevant language goals while removing malformed annotations and task-irrelevant motion. The complete pipeline preserves broad task coverage and most unique instructions despite filtering trajectory units.

  • Instruction validation: The pipeline retains well-formed, actionable robot goals while removing placeholders, metadata, malformed strings, ambiguous noun-only labels, and unsuitable non-English annotations.This instruction-validation stage uses source-specific usable-instruction sets when canonical vocabularies are available.
  • Action-relevance filtering: Mobile-manipulation segments containing locomotion or approach predicates without manipulation predicates are removed from temporal-distance training.This filtering is especially prominent in Galaxea and targets segments lacking an interaction objective.
  • Quantitative curation effect: 1,436,150 of 1,722,966 trajectory units (83.35%) and 192,989 of 194,967 unique instructions (98.99%) are retained after curation.The disparity indicates that removals concentrate in invalid annotations rather than rare valid tasks.
  • Skill distribution: The miscellaneous other category decreases from 4.54% to 1.37%, while the dominant pick-place category decreases from 43.38% to 33.61% after filtering.The three largest skill groups still account for 68.21% of the curated data, indicating broad coverage remains.
  • Atomic-skill diagnostic: 640 atomic verb categories in raw annotations decrease to 549 after curation, primarily through removal of empty tokens, termination markers, malformed strings, and artifacts.This analysis diagnoses curation effects only; it does not define temporal-distance targets or alter the training sampler.

B Real-World Experiment Details

This section presents the implementation details for reproducing the robotic policy-learning experiments in Section 4.5, covering interfaces, experimental settings, procedures, and compute resources.

  • Implementation Overview: The section defines interfaces between the VLA policy and auxiliary RL networks.These interfaces support reproduction of the robotic policy-learning experiments in Section 4.5.
  • Experimental Setup: It describes the tasks, datasets, robotic platform, evaluation protocol, and initialization.
  • Training and Resources: It details the offline and online RL procedures and the compute resources used.

B.1 Implementation Inputs and Outputs

The policy-learning pipeline uses camera observations and task instructions rather than proprioceptive or robot-state vectors. Its VLA, offline IQL, and online DSRL interfaces produce action chunks or value estimates for policy refinement.

  • Implementation inputs and outputs: All three interfaces use multi-camera visual observations, and no component receives a proprioceptive or robot-state vector.At policy-decision step t, o_t denotes the observation and a_t the corresponding executable action chunk.
  • Implementation inputs and outputs: The VLA consumes 224 × 224 images, validity masks, the task instruction, and a flow-matching timestep, outputting an H = 16 action chunk with d_a = 32.The action dimension is padded to 32.
  • Implementation inputs and outputs: The IQL critic and value networks consume current and next observations plus demonstrated action chunks to estimate Q(o, a) and V(o).Their estimates produce the advantage weight used to refine the VLA.
  • Implementation inputs and outputs: The DSRL actor–critic uses only 64 × 64 multi-camera observations; its actor predicts z ∈[−1, 1]16×32, while critics estimate Q-values and tune α automatically.Automatic entropy tuning optimizes the temperature α.

B.2 Tasks and Datasets

The evaluation covers four robotic manipulation tasks using mixed-expertise datasets that combine approximately 100 successful trajectories per task with unsuccessful attempts from the same collection process.

  • 4 robotic manipulation tasks are used to evaluate all methods.
  • Approximately 100 successful trajectories are collected for each task.
  • Unsuccessful attempts encountered during collection are retained, yielding mixed-expertise datasets.
  • Success rate is computed over all collected trajectories for each task.

B.3 Robotic Platform and Evaluation Protocol · B.4 Policy and RL Initialization

The evaluation uses a dual-arm Franka platform with specified camera, control, reset, and reward-scoring protocols. SFT, offline IQL, and online DSRL share the π0.5 backbone but differ in auxiliary networks and initialization procedures.

  • B.3 Robotic Platform and Evaluation Protocol: The dual-arm Franka platform uses four Intel RealSense cameras: two D435 third-person cameras and two D405 wrist-mounted cameras.Images are padded to 224 × 224 for the VLA and IQL critic, and downsampled to 64 × 64 for the DSRL visual encoder.
  • B.3 Robotic Platform and Evaluation Protocol: Table 8 specifies the exact camera streams supplied to the policy and RL encoders.
  • B.3 Robotic Platform and Evaluation Protocol: The low-level controller runs at 10 Hz with absolute joint-position and relative gripper commands, padding task-specific actions to d_a = 32.VLA predictions use action chunks with horizon H = 16.
  • B.3 Robotic Platform and Evaluation Protocol: Offline evaluations use fixed initial configurations, while online evaluations randomize initial objects for three tasks and use a fixed reset for Box-in-Drawer Placement.The randomized tasks are Bread Basket Placement, Steak Serving with a Spatula, and Bimanual Box Transfer.
  • B.3 Robotic Platform and Evaluation Protocol: RynnValue and Robometer receive the task instruction and right_side RGB sequence through the same causal, history-conditioned scoring interface.RynnValue uses only its absolute temporal-distance head during reward relabeling and online RL.
  • B.3 Robotic Platform and Evaluation Protocol: RynnValue converts predicted remaining time v_t into the higher-is-better potential Φ_t = −v_t and constructs shaping rewards at policy-decision boundaries.Each complete trajectory is scored once after collection, rather than at every low-level control step.
  • B.4 Policy and RL Initialization: SFT, offline IQL, and online DSRL all use the same flow-matching vision-language-action policy, π0.5, isolating learning-algorithm and reward-specification effects.
  • B.4 Policy and RL Initialization: Offline IQL uses a ResNet-18 visual encoder and two Q-functions plus a separate value function, whereas online DSRL uses a four-layer convolutional encoder.Both auxiliary networks use GroupNorm, spatial softmax, and a 50-dimensional bottleneck; IQL heads use hidden dimensions (256, 256).

B.5 Offline RL with IQL · B.6 Online RL with DSRL

Offline IQL trains on mixed-expertise trajectories with sparse task rewards and potential-based shaping from RynnValue or baseline potentials. Online DSRL instead optimizes a SAC policy over latent noise while using the frozen VLA to decode executable action chunks, with the same reward variants and shaping coefficients.

  • B.5 Offline RL with IQL: Mixed-expertise offline datasets combine successful and unsuccessful trajectories for each task.Trajectories are segmented at policy-decision boundaries into transitions with action chunks and bootstrap masks, setting the mask to 0 only for terminal transitions.
  • B.5 Offline RL with IQL: Offline reward variants share transitions, outcome labels, policy initialization, and optimization settings, differing only in potential source and shaping coefficient.The common dataset is relabeled using RynnValue or a corresponding baseline reward model.
  • B.5 Offline RL with IQL: RynnValue defines the offline potential as Φ_h = −v_h, while sparse rewards assign −1 before completion and 0 to the completing transition.All transitions in unsuccessful trajectories receive −1; the final IQL reward combines sparse reward with potential-based shaping.
  • B.5 Offline RL with IQL: Shaping coefficients are fixed across tasks at κ = 0.1 for RynnValue, κ = 1.0 for Robometer, and κ = 0 for the sparse-reward baseline.IQL uses expectile value regression, one-step Q targets, minimum aggregation across 2 target Q-functions, and Polyak averaging; VLA loss weighting increases for higher estimated advantages after warmup.
  • B.6 Online RL with DSRL: Online DSRL freezes the task-specific VLA as a noise-conditioned decoder and trains a SAC policy over latent noise rather than executable robot actions.The latent action space uses an action horizon H = 16 and per-step latent dimension d_z = 32.
  • B.6 Online RL with DSRL: Latent-space SAC uses entropy regularization, ten target-Q functions averaged for critic targets, automatic temperature tuning toward target entropy H = −dim(z), and Polyak-updated critics.The actor balances ensemble-mean Q-value against policy entropy.
  • B.6 Online RL with DSRL: Online DSRL assigns −1 until task completion and 0 at completion, converts per-environment-step discounting across q low-level steps, and applies potential-based shaping.The shaping coefficients remain κ = 0.1 for RynnValue, κ = 1.0 for Robometer, and κ = 0 for the sparse-reward baseline, fixed across tasks.

B.7 Compute Resources

The compute setup uses two 80GB GPUs for offline IQL and SFT, while online DSRL runs on a large single-server system. Training one task takes approximately 16 hours with offline IQL and 6 hours with SFT.

  • Hardware and training configuration: Offline IQL and SFT each use 2 GPUs with 80GB of memory each.IQL assigns one FSDP device to each replica, whereas SFT uses two-way model sharding.
  • Training time: Training one task requires approximately 16 hours for offline IQL and 6 hours for SFT.
  • Online DSRL infrastructure: Online DSRL runs on a single x86 server with two Intel Xeon Platinum 8575C processors, 1.5 TiB of system memory, and eight NVIDIA GeForce RTX 5090 GPUs.The processors provide 48 physical cores per socket with hyper-threading enabled and 192 logical cores in total; each GPU has 32 GB of memory.
Loading 2608.09853v1…