PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-Then-Act: Reward-Free Control from Frame-Order Progress

Yuriy Maksyuta, George Bredis, Ruslan Rakhimov, Daniil Gavrilov

arXiv:2607.01897v1cs.LGcs.AI

TL;DR

Learning control from pixels without extrinsic rewards remains difficult when reward design is brittle or unavailable. RTA learns ordinal progress from shuffled expert videos and uses windowed Spearman correlation as the sole reward, matching or outperforming strong baselines across discrete and continuous tasks while transferring one scorer across tasks and environments.

  • Problem

    Learning control from pixels without extrinsic rewards remains challenging when reward design is brittle, unavailable, or vulnerable to trivial shortcuts.

  • Method

    RTA fine-tunes a VLM on shuffled expert clips, then freezes it and uses windowed Spearman correlation between progress ranks and timestamps as the sole policy-learning reward.

  • Results

    RTA matches or outperforms strong video-based baselines across discrete and continuous benchmarks, while one pretrained progress scorer transfers across tasks and environments.

  • Takeaways & Limitations

    Correlation-structured ordinal supervision from expert video can support reward-free policy learning across discrete and continuous control settings.

  • Takeaways & Limitations

    The windowed correlation reward is scale-dependent, with small windows overemphasizing local consistency and large windows diluting meaningful structure.

Abstract

from arXiv · show

We introduce Rank-Then-Act (RTA), a framework for learning control policies from expert video demonstrations without environment rewards. RTA trains a Vision-Language Model (VLM) offline as a progress-based ordinal scorer, using a Group Relative Policy Optimization (GRPO) objective over shuffled frame sequences, which forces the model to recover temporal ordering from visual semantics rather than trivial time cues. Importantly, instead of using the scorer directly as a scalar reward model, we propose a correlation-based reward function for reinforcement learning: at each interaction window, we compute the Spearman rank correlation between predicted progress rankings and true temporal indices, yielding a bounded, scale-invariant learning signal. This design decouples reward learning from absolute calibration and enables stable transfer across tasks and environments. We evaluate RTA on discrete control benchmarks (PyBoy: Catrap, Kirby) and continuous control tasks (PointMaze, MetaWorld). RTA consistently matches or outperforms prior video-based reward learning methods and rank-based baselines, while demonstrating strong cross-task reuse of a single pretrained progress scorer. Our results suggest that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design.

1 Introduction

RTA addresses reward-free pixel control by learning ordinal progress from shuffled expert videos and converting rank agreement into a bounded, scale-invariant reinforcement-learning signal. The framework supports control across discrete and continuous benchmarks and transfers one pretrained scorer across tasks and environments.

  • Motivation: Reward-free pixel control is difficult where rewards are brittle, unavailable, or exploitable, including retro games, weak-API simulators, and real-world robotics.Proxy rewards can cause specification gaming.
  • Method: Stage 1 trains a frozen VLM progress scorer with GRPO on shuffled expert video clips, forcing temporal ordering recovery from visual semantics without absolute frame positions.The scorer assigns progress ranks to shuffled clips and remains frozen during policy learning.
  • Method: Stage 2 uses Spearman correlation between predicted progress ranks and temporal indices in sliding interaction windows instead of treating the VLM as a scalar reward model.This correlation-only signal provides dense, shaped feedback for online control.
  • Contributions and results: RTA enables effective control across discrete and continuous benchmarks without environment rewards, outperforms strong video-based baselines, and transfers one pretrained scorer across tasks and environments.The paper positions ordinal video supervision as a general alternative to engineered goals, action labels, and environment rewards.

2 Related Work

Related work uses temporal ordering, pretrained VLMs, intrinsic motivation, and video-only imitation or inverse reinforcement learning to obtain task-agnostic control signals. RTA instead freezes an offline listwise VLM progress scorer and uses Spearman correlation between predicted progress ranks and timestamps for online control.

  • Ordinal progress and ordering-based supervision from video: Rank2Reward learns shaped rewards from passive video via temporal ranking and serves as RTA’s non-VLM ranking baseline.The comparison uses the same game video data and evaluates learned rewards and downstream control performance.
  • Vision–language models as reward/value estimators: Pretrained VLMs estimate reward or value by scoring observations against language goals or ordering shuffled frames, leveraging world knowledge to avoid task-specific reward engineering.VLAC trains a critic using the logic provided by GVL.
  • Reward-free RL and intrinsic motivation: Reward-free RL separates reward-free exploration from downstream reward queries, while intrinsic motivation supplies task-agnostic signals through curiosity or prediction-error bonuses.These approaches emphasize state-space coverage or generic exploration rather than learning task-specific progress.
  • Imitation from observation and IRL from video: Imitation-from-observation and video-based IRL learn behaviors from state- or video-only data, including adversarial distribution matching without explicit reward engineering.GAIL and GAIfO are examples of adversarial formulations in this line of work.
  • Imitation from observation and IRL from video: RTA trains and freezes a listwise VLM progress scorer offline, then rewards online control with Spearman correlation between progress ranks and true timestamps.This avoids adversarial training and scalar reward regression; T-REX/D-REX and XIRL/ORCA use a different data regime requiring multiple demonstrations of the same task or decomposition.

3 Method

RTA converts frame-order understanding from expert videos into a bounded, reward-free control signal through offline scorer training and online policy optimization. It uses Spearman rank correlation throughout to reward temporally correct progress ordering without requiring absolute score calibration.

  • Two-stage framework: RTA has two stages: GRPO trains a VLM to rank shuffled video frames by progress, then the frozen scorer supplies online rewards for control.The resulting signal is bounded in [−1, 1].
  • Expert clip batching: Training and inference anchor the first frame while randomly shuffling the remaining frames, preventing temporal shortcuts and encouraging task-relevant progress assessment.Only non-anchor frames are scored, and the same procedure is applied during inference.
  • Progress scoring: The scorer predicts progress ranks for shuffled non-anchor frames, and Spearman correlation with their original temporal indices rewards correct monotone ordering independently of rank scale.The VLM generates a reasoning trace and predicted rank for each scored frame; unparsable outputs receive Rmin = −1.
  • Inference reward: At query steps, the frozen scorer evaluates an anchored interaction window under multiple non-anchor permutations, averaging the resulting Spearman correlations as the reward.Rewards are computed every N steps, while non-query steps receive rt = 0.
  • Policy optimization: Policy-gradient training uses rt as its sole reward signal, with bounded sparse rewards and Generalized Advantage Estimation for optimization.No additional environment reward signals are used.

4 Experiments

RTA is evaluated without extrinsic rewards, environment APIs, or task annotations across discrete game and continuous-control tasks. Experiments show that ordinal progress scoring and correlation-based rewards support transfer and policy learning, while revealing limitations under difficult exploration and extreme window scales.

  • Experimental setup: RTA evaluates reward-free vision–language policy learning using only expert video demonstrations, without extrinsic rewards, environment APIs, or task annotations.The evaluation trains a progress scorer and then optimizes policies using progress–time correlation as reward.
  • Learning dynamics: Almost all videos reached Spearman ρ > 0.9 within at most 300 training steps, showing rapid convergence of the VLM progress scorer.The scorer is trained on shuffled expert clips across levels and complete game playthroughs.
  • Cross-level generalization: Pooled training across all levels produced the strongest overall transfer, while simpler Level 1 generalized especially well from other source levels.A diverse pool of 70 GameBoy playthroughs also provided solid zero-shot performance, supporting reuse of a shared scorer.
  • Continuous control: RTA consistently outperformed R2R on PointMaze-UMaze and surpassed R2R on MetaWorld under ranking-only rewards, though classifier-based GAIL rewards remained stronger there.The MetaWorld gap is attributed to extensive exploration and occasional high VOC-scores for unsuccessful but dynamically plausible trajectories.
  • GameBoy results: RTA achieved the strongest success rates on all tested GameBoy levels except Kirby and was the only method with non-zero Kirby success without environment rewards.The GameBoy evaluation included Catrap Levels 2, 4, and 6, plus the long-horizon Kirby game.
  • Scalability and reward analysis: RTA remains competitive under matched compute, reuses one pretrained scorer across tasks, and performs best with intermediate correlation windows that preserve final-goal sensitivity.Very small windows overreact to local consistency, whereas large windows dilute the signal through temporal averaging.

5 Discussion

RTA suggests scalable reward-free learning from expert videos by using progress–time correlation, reducing reliance on manually designed rewards. Its broader applicability is balanced by data, temporal-window, scorer, and efficiency limitations.

  • Contributions: Progress–time correlation enables policy learning directly from expert videos without manually designed reward functions, potentially reducing reward hacking and unintended behaviors.The approach frames ordinal video-based rewards as a promising direction for scalable generalist control.
  • Limitations: Undertrained Stage 1 models can assign incorrect rewards, including high rewards to out-of-domain or non-progressing trajectories.The paper shows correct and incorrect scoring examples and proposes complementary signals such as a GAIL-style reward.
  • Contributions: Sampling frames and using VLM reasoning to infer temporal order allows RTA to score long trajectories beyond strictly monotonic tasks.
  • Limitations: Sufficiently diverse expert video data is required; limited coverage or dataset bias can cause level-specific overfitting and weaker cross-level transfer.This limitation is reflected in asymmetric Stage 1 generalization.
  • Limitations: Window size affects rank-correlation rewards: small windows overemphasize local consistency, large windows dilute structure, and intermediate windows are most stable across environments.Cyclic analysis and ablations support the intermediate-window regime; multi-scale or hierarchical temporal aggregation is suggested for highly non-monotonic settings.
  • Limitations: Current VLM backbones introduce latency despite caching and striding, motivating more efficient or distilled models.

6 Conclusion

RTA trains VLM-based agents from expert video and progress ranks via Spearman correlation, without extrinsic rewards. It achieves high success rates across discrete games and long-horizon continuous environments while offering a scalable path toward robust multimodal agency.

  • Method: RTA uses expert video and progress ranks via Spearman correlation to train VLM agents without extrinsic rewards.The method is two-stage and uses a purely reward-free learning signal.
  • Results: RTA achieves high success rates and outperforms strong baselines in challenging discrete-action games and long-horizon continuous environments.These results are obtained using only the reward-free signal.
  • Motivation: Learning from expert video alone offers a scalable path to multimodal agency when real-world rewards are sparse or difficult to design, while reducing reward-hacking risks.The motivation is to avoid reliance on manually specified extrinsic rewards.
  • Future Direction: RTA represents a step toward robust, generalist agents that understand and act in open, dynamic, and underspecified environments.This frames the broader direction suggested by the method.

A Appendix · A.1 Experiments with LOOP

The LOOP experiments apply end-of-episode rewards with trajectory refreshing to extend progress-based learning beyond limited episode windows. Refreshing can improve task completion, although oversized rollouts may temporarily reduce reward after the task finishes.

  • A.1 Experiments with LOOP: LOOP sets episode length to T = N, computes reward only at episode end, and assigns the same advantage to all output tokens.This multi-turn GRPO extension uses a shared end-of-episode advantage across tokens.
  • A.1 Experiments with LOOP: When average episode reward rT exceeds threshold τ, LOOP restarts from a new starting point and repeats using the highest-reward trace.The selected trace determines the next rollout’s starting state.
  • A.1 Experiments with LOOP: Figure 6 compares per-query progress–time Spearman reward curves with and without starting-point refreshing during LOOP training.The experiment reports separate reward curves for the two refreshing conditions.
  • A.1 Experiments with LOOP: Without refreshing, rewards increase steadily, but success rate does not rise because the default window length N cannot contain the full task.The fixed episode window limits long-horizon interaction and task completion.
  • A.1 Experiments with LOOP: Starting-point refreshing addresses the insufficient window by selecting the highest-reward trajectory after threshold τ and continuing from its final state.This mechanism links successive rollouts after the threshold is reached.
  • A.1 Experiments with LOOP: Refreshing can temporarily decrease reward when rollout length N exceeds the steps needed to finish the level, leaving extra steps with little scoring signal.The decline appears in post-refresh segments because the agent completes the level quickly.

A.2 Stage 1: Progress-Scorer Learning Dynamics

Stage 1 training usually converges rapidly to high progress–time rank correlation, although some runs diverge and less clearly defined levels are more sensitive to preparation and duration. Pooling full-game videos smooths training and enables one scorer across levels, with a slightly lower final plateau than most single-level runs.

  • Per-level training curves: Most Catrap levels converge within 200 steps, reaching asymptotic progress–time Spearman ρ near 0.9.The scorer was trained on individual levels, and all training videos were extracted under human supervision.
  • Per-level training curves: Some runs diverge after high performance, consistent with known GRPO instabilities.These instabilities were also observed during extended GRPO training.
  • Per-level training curves: Level 3 alternates between convergence and divergence, indicating greater sensitivity to data preparation and training duration when task goals are less clear.The result suggests progress remains learnable even in levels with less clearly defined goals.
  • Full-playthrough training: Pooling Catrap levels 1–8 increases data diversity and smooths training dynamics, but slightly lowers the final performance plateau relative to most single-level runs.Static or non-informative frames, including menus, may explain the lower plateau.
  • Full-playthrough training: A single scorer can be trained end-to-end from full-game videos, eliminating level-specific tuning.The pooled experiment used 300 training steps on three different games and evaluated mixed-level and held-out sets.

A.3 Setup

The setup uses expert gameplay videos and agent trajectories to train across discrete PyBoy games and continuous-control tasks. Policy learning uses standardized visual inputs, discrete directional actions, and a frozen scorer for progress ranks.

  • Expert videos: Stage 1 uses successful human Catrap playthroughs, one full-game run, 70 additional GameBoy videos, and 100 expert-agent trajectories for MetaWorld and PointMaze.Videos are split into fixed-length clips of K consecutive frames and converted to the anchor+shuffle interface.
  • Tasks and levels: Evaluation covers Catrap levels L2, L4, and L6, plus Kirby level-0 in PyBoy, with both games requiring at least 15 steps for success.Catrap provides logically complex spatial-planning puzzles, while Kirby is a standard VideoGameBench VLM benchmark.
  • Observations and actions: PyBoy observations are rendered at 160 × 144 pixels, centrally cropped and resized to 160 × 120, with actions corresponding to four D-pad directions.Frames are resized back to the original training resolution before querying the frozen scorer fϕ for progress ranks.
  • Technical details: VL-DAC experiments train only LoRA adapters, using rank 16 and alpha 32; scorer training uses 4 H100 GPUs, while policy learning uses 1 H100 GPU.Two hundred scorer-training steps take approximately 2 hours, and Stage 2 training takes up to 24 hours.

A.4 Experimental details

The experiments use a two-stage reward-free training setup: Stage 1 ranks shuffled frame sequences with correlation rewards, while Stage 2 trains policies using windowed progress–time correlations.

  • Stage 1: Stage 1 applies GRPO to a tuned VLM with the first frame anchored and remaining K−1 frames shuffled, using R = spr(p, q).A moving-average baseline is used; p denotes progress ranks and q temporal indices.
  • Stage 2: Stage 2 computes the windowed progress–time correlation ρ ∈[−1, 1] every N = 15 steps, with L = 2 shuffled-window evaluations per query.The window length is m as specified in the experimental setup.
  • Stage 2: LOOP optionally refreshes starting points when reward exceeds threshold τ to support long-horizon completion.This mechanism is specific to the multi-step Stage 2 training variant.
  • Stage 2: Stage 2 trains policy and value networks with VL-DAC and LOOP, a multi-step GRPO variant using the same correlation-only reward.VL-DAC uses GAE parameters γg and λg, while learning rates follow cosine decay.
  • Reward-free training: No environment rewards are used at any stage, and per-episode standardization is restricted to advantage estimation.Stage 1 additionally uses KL control with target KL = 0.1.

A.5 Ablations … A.8 Cyclic trajectory construction.

Ablations show that RTA remains robust across policy architectures and implementation choices, while cyclic-trajectory tests identify intermediate windows as balancing local sensitivity with globally consistent progress. Examples and matched-budget comparisons further show that the bounded correlation reward can support reward-free control and remain competitive across tasks.

  • A.5 Ablations: RTA’s reward design remains effective with an MLP policy backbone, indicating performance is not dependent on VLM-specific biases.The optimal feedback schedule differs: initialized MLPs benefit from termination-step feedback, whereas VLM policies often benefit from shorter-window shaping.
  • A.5 Ablations: Window lengths m = 15 and higher yield stable performance, whereas very short windows degrade results and larger windows slow inference.The ablation fixes reward frequency to 15 and L to 2.
  • A.5 Ablations: Using only one shuffle degrades reward stability and can slow convergence, while reward frequency does not qualitatively affect stability.The number of shuffles is therefore important for learning a robust ordinal signal, whereas convergence-speed effects from reward frequency depend more on optimization dynamics.
  • A.6 Examples: 100% success is attained at several points during Level 2 training when smoothing is disabled, while mean bounded reward steadily tracks success.The reward is computed every N = 15 steps and remains bounded in [−1, 1].
  • A.6 Examples: The video-derived progress–time correlation is informative and stable enough to drive online reinforcement learning without extrinsic rewards.Level 3 nevertheless failed to converge within 200 steps, motivating longer training experiments.
  • A.7 Training efficiency: Under matched wall-clock budgets, RTA is competitive on every task, stronger on a majority, and outperforms Rank2Reward on PointMaze.RTA is stronger on door-open, door-close, and reach; near-parity on drawer-open; and weaker on button-press-topdown and hammer.
  • A.8 Cyclic trajectory construction.: For cyclic trajectories, 1-second windows continue increasing reward during cycles, 3–5-second windows decrease or stabilize, and windows ≥5 seconds approach zero.The trajectory combines forward and reversed expert-demonstration segments, with rank correlation computed over sampled frames in non-overlapping windows.
  • A.8 Cyclic trajectory construction.: After cyclic regions end, every window size sharply increases reward during renewed forward progress, correctly identifying goal completion.This supports fixing one intermediate window size across tasks to avoid local artifacts from very small windows and signal dilution from very large ones.
Loading 2607.01897v1…