PaperlayerView exact revision on arXiv

Source-linked AI summary

From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation

Yibin Liu, Yaxing Lyu, Daqi Gao, Zhixuan Liang, Weiliang Tang, Shilong Mu, Xiaokang Yang, Yao Mu

arXiv:2603.15600v2cs.ROcs.AIcs.CLcs.CV

TL;DR

Long-horizon robotic manipulation needs reliable process supervision, but video MLLMs often describe events without quantitatively evaluating progress toward task goals. PRIMO R1 addresses this gap with outcome-based reinforcement learning for explicit Chain-of-Thought reasoning and temporal anchoring between initial and current states. Across domains, the 7B framework achieves state-of-the-art progress estimation and zero-shot failure detection, including 67.0% accuracy on RoboFail.

  • Problem

    Video MLLMs trained primarily with supervised fine-tuning act as passive Observers, recognizing ongoing events rather than measuring progress relative to final task goals.

  • Method

    PRIMO R1 uses outcome-based reinforcement learning to elicit Chain-of-Thought progress reasoning and anchors video sequences between initial and current state images.

  • Results

    PRIMO R1 achieves state-of-the-art performance across domains, including 67.0% accuracy on RoboFail and a 50% reduction in mean absolute error versus specialized baselines.

  • Takeaways & Limitations

    Optimizing continuous progress reasoning supports zero-shot generalization to discrete failure detection across unseen tasks and environments.

Abstract

from arXiv · show

Accurate process supervision remains a critical challenge for long-horizon robotic manipulation. A primary bottleneck is that current video MLLMs, trained primarily under a Supervised Fine-Tuning (SFT) paradigm, function as passive "Observers" that recognize ongoing events rather than evaluating the current state relative to the final task goal. In this paper, we introduce PRIMO R1 (Process Reasoning Induced Monitoring), a 7B framework that transforms video MLLMs into active "Critics". We leverage outcome-based Reinforcement Learning to incentivize explicit Chain-of-Thought generation for progress estimation. Furthermore, our architecture constructs a structured temporal input by explicitly anchoring the video sequence between initial and current state images. Supported by the proposed PRIMO Dataset and Benchmark, extensive experiments across diverse in-domain environments and out-of-domain real-world humanoid scenarios demonstrate that PRIMO R1 achieves state-of-the-art performance. Quantitatively, our 7B model achieves a 50% reduction in the mean absolute error of specialized reasoning baselines, demonstrating significant relative accuracy improvements over 72B-scale general MLLMs. Furthermore, PRIMO R1 exhibits strong zero-shot generalization on difficult failure detection tasks. We establish state-of-the-art performance on RoboFail benchmark with 67.0% accuracy, surpassing closed-source models like OpenAI o1 by 6.0%.

1 Introduction

PRIMO R1 reframes video MLLMs from passive observers into active critics by combining outcome-based reinforcement learning for explicit process reasoning with temporal anchoring between initial and current states. Its dataset, benchmark, and experiments support strong progress-estimation and failure-detection performance across domains.

  • Motivation: Existing video MLLMs recognize ongoing events but struggle to quantitatively judge progress toward final task goals.Supervised fine-tuning encourages event description rather than measuring the distance between current and desired states.
  • Approach: PRIMO R1 uses outcome-based reinforcement learning to elicit Chain-of-Thought reasoning for progress estimation instead of supervising only a scalar label.The framework is a 7B model designed to transform video MLLMs from passive Observers into active Critics.
  • Approach: Explicitly anchoring video sequences between initial and current state images provides structured boundary conditions for state-alignment reasoning.The architecture connects temporal perception with verification against task goals.
  • Resources: The PRIMO Dataset combines SFT and RL post-training data with Chain-of-Thought annotations, while PRIMO Benchmark evaluates cross-task and cross-environment generalization.The benchmark includes out-of-domain evaluation settings.
  • Results: 82.90 MRA and 15.52 MAE show that the 7B model outperforms 72B-scale general MLLMs by +9.10 absolute MRA points.These are reported as aggregate experimental results across multiple domains.
  • Results: 67.0% accuracy on RoboFail establishes state-of-the-art failure detection and surpasses closed-source models such as OpenAI o1 by 6.0%.The result demonstrates zero-shot generalization in execution anomaly verification.

2 Related Work

Prior video MLLMs primarily act as passive observers, while progress-estimation methods often depend on demonstrations or engineered temporal mechanisms. PRIMO R1 extends this line of work by using outcome-based reinforcement learning to elicit verifiable reasoning for robotic process supervision.

  • Video MLLMs: Early video MLLMs used temporal aggregation, context compression, and hierarchical structures but remained focused on perceptual question answering.These architectures lacked quantitative temporal reasoning for rigorous process judgment.
  • Video MLLMs: Recent designs add evidence searching and timestamp encoding to support the transition from passive observation toward active temporal judgment.PRIMO R1 positions progress reasoning as the next step in this transition.
  • Reward and Progress Modeling: Semantic reward models use representation distances or frozen embeddings, while progress estimators employ frame ordering, multimodal integration, and synthetic trajectory augmentation.These approaches target universal value functions or explicit progress estimation through different visual and temporal mechanisms.
  • Reward and Progress Modeling: VLAC, Robo-Dopamine, and PROGRESSLM are structurally limited by dependence on explicit reference demonstrations.The related-work discussion identifies this dependency as a primary limitation of those methods.
  • Reasoning and RL: Outcome-based reinforcement learning can induce Chain-of-Thought without dense annotations, and PRIMO R1 maps this capability to robotic process supervision through outcome rewards.The approach formulates task-completion metrics as rewards for verifiable, self-correcting reasoning paths.

3 Method

PRIMO R1 formulates robotic process supervision as state estimation and transforms video MLLMs from passive observers into active critics through reasoning-based reinforcement learning. The method combines boundary-anchored temporal inputs, structured post-training data, and GRPO-based rewards to evaluate progress and generalize across task and environment shifts.

  • 3.1 Problem Formulation: Robotic process supervision maps initial and current images, a process video, and a language instruction to a scalar progress value from 0 to 100.The boundary images represent the initial and latest states, while the instruction provides the semantic task reference.
  • Dataset and Benchmark: The framework explicitly anchors the temporal video between initial and current states, while PRIMO Dataset and PRIMO Bench support post-training and distribution-shift evaluation.The benchmark separates in-domain same-task evaluation from out-of-domain cross-task and cross-environment testing, including teleoperated humanoid trajectories.
  • 3.1 Problem Formulation: Existing video MLLMs treat progress estimation as regression or classification, describing ongoing events rather than measuring distance to the final task goal.This passive Observer formulation is trained through supervised fine-tuning.
  • 3.1 Problem Formulation: PRIMO R1 replaces direct scalar regression with Chain-of-Thought generation followed by a progress estimate, using reinforcement learning rewards based on final-prediction accuracy.The policy is conditioned on diverse natural-language goals so reasoning aligns visual execution with task objectives.
  • 3.3 Process Reasoning RL with Group Relative Policy Optimization: GRPO estimates baselines from groups of sampled outputs and uses format and accuracy rewards to encourage structured reasoning and precise continuous progress predictions.The optimization also constrains the policy relative to a reference policy through a KL-divergence penalty to limit reward hacking or language degeneration.

4 Experiments

PRIMO R1 is evaluated for continuous progress estimation across diverse environments and for zero-shot failure detection, with analyses of metrics, generalization, temporal inputs, and trajectory tracking. Across these evaluations, the RL-finetuned model achieves strong average performance, improves out-of-domain transfer, and benefits from explicit temporal state references.

  • Overall Performance: PRIMO R1 achieves the highest average MRA (82.90) and lowest average MAE (15.52) across AgiBot, Behavior, RoboTwin, and Real Humanoid.It consistently outperforms evaluated open-source baselines across the four environments.
  • Overall Performance: 9.10 MRA points separate PRIMO R1 from Qwen2.5-VL-72B, while average MAE falls from approximately 27–29 to 15.52 against Video R1 7B and Robobrain 7B.These comparisons show lower estimation error despite PRIMO using a 7B backbone.
  • Overall Performance: PRIMO improves average MRA from 78.32 to 82.90 and reduces average MAE from 20.87 to 15.52 against specialized progress estimation models.It leads on AgiBot, Behavior, and RoboTwin, while ProgressLM performs better in the Real Humanoid setting under its dedicated input configuration.
  • Fine-Grained Error Analysis: PRIMO maintains lower MAE through five completion intervals, particularly mitigating baseline error spikes and hallucinations during the final 80−100% execution stage.In the unseen Real Humanoid environment, PRIMO reaches 72.32 MRA versus 56.46 for Qwen2.5-VL-7B.
  • Generalization and Ablations: RL improves transfer beyond SFT: the complete SFT+RL pipeline lifts cross-environment accuracy to 72.32, whereas RL-only reaches 76.72 average MRA.The reported results attribute the transfer gains to self-correction and rigorous causal reasoning learned during RL.
  • Temporal Input Ablation: 59.50 Average MAE using only the current image falls to 36.61 with the video sequence, while combining temporal input with state anchors further improves task-specific performance.Vseq + Icurr is best on RoboTwin at 40.24, and Iinit + Vseq is best on AgiBot at 24.94; PRIMO uses all three modalities.

5 Conclusion

PRIMO R1 combines diverse simulation and real-world humanoid data with reinforcement learning to support process supervision across long-horizon robotic tasks.

  • PRIMO R1 is a 7B framework for robotic process supervision using outcome-based reinforcement learning and explicit Chain-of-Thought generation.
  • The dataset aggregates high-fidelity simulation and real-world humanoid manipulation data spanning atomic actions to long-horizon composite tasks.
  • BEHAVIOR-1k and RoboTwin provide the primary simulation sources, with annotations segmented into fine-grained subtasks.
  • AgiBot supplies training and in-domain real-world data, while Real Humanoid data supports cross-environment generalization evaluation.

A.2 Dataset Statistics and Distribution

The constructed dataset statistics summarize task coverage, video demonstrations, subtask complexity, and processed samples across training and evaluation splits.

  • Table 5 reports task distribution, video counts, and processed trajectory-segment scale across different domains.
  • Real Humanoid processed samples refer specifically to the validation set count used for evaluation.

A.3 Other Benchmark and Datasets Details

The paper supplements progress-estimation evaluation with failure detection, reasoning, planning, perception, and temporal-understanding resources from robotics and multimodal video research.

  • RoboFail contains 100 difficult evaluation examples targeting action affordance and task-completion verification.Its scenarios require observant perception or comprehensive temporal-context processing.
  • Additional multimodal datasets provide supervision for task planning, temporal reasoning, and scene understanding during training.
  • ShareRobot contributes task-planning annotations for abstract reasoning and goal decomposition, while EgoPlan-Bench evaluates egocentric real-world planning.
  • RoboVQA, Perception Test, and STAR support robotics video question answering, grounded perception and reasoning, and geospatial scene-graph understanding.
  • The GRPO modality study compares pure temporal video input with the proposed triad of initial image, video sequence, and current image.
  • NExT-QA is included as a video question-answering benchmark, while RoboFail specifically targets causal and temporal action reasoning.

B Training Dynamics of GRPO Across Input Modalities

Adding initial and current boundary images to the temporal sequence accelerates GRPO convergence, reduces reward variance, and supports more structured reasoning about task progress.

  • Training dynamics: The triad configuration Iinit + Vseq + Icurr converges faster and reaches a higher final reward plateau than the Vseq-only baseline.
  • Training dynamics: The Vseq-only model shows higher reward variance, whereas explicit initial-to-current tracking yields a more stable fine-tuning process.
  • Reasoning processes: In the Fold shorts case study, RoboBrain-7B reports 0% progress and Qwen2.5-VL-7B reports 25% progress while lacking structured evaluation.
  • Reasoning processes: A Real Humanoid example demonstrates PRIMO R1 reasoning in an out-of-domain physical environment.
  • Reasoning processes: PRIMO R1 decomposes the goal into five steps, verifies visual execution, and aligns observed states with the planned topology.

C.2 Quantitative Analysis: Chain Length and Inference Latency

The analysis compares reasoning efficiency with progress-estimation accuracy, showing that PRIMO R1 achieves a stronger balance than evaluated baselines. Its concise reasoning and low latency accompany substantially higher average MRA.

  • Baseline comparisons: 66.52 average MRA is achieved by Cosmos-Reasoning 7B despite 1109.75-token reasoning chains and 1.30s latency.It has the lowest average MRA among the evaluated reasoning MLLMs, indicating that longer generation does not ensure effective structural reasoning.
  • Baseline comparisons: Robobrain 7B is fastest at 0.50s with 242.3 tokens but trails PRIMO R1 by nearly 15% in average MRA.Its lower computational cost is accompanied by a substantial accuracy sacrifice.
  • PRIMO R1 efficiency: PRIMO R1 combines 0.62s latency and 359.18 tokens with an average MRA of 82.90.The reported balance is competitive in computational overhead while delivering higher accuracy than fast baselines such as Video R1 7B.
  • Efficiency–accuracy trade-off: Inference efficiency is evaluated jointly with MRA because latency or token count alone cannot characterize real-time robotic suitability.The comparison considers both reasoning-chain length and inference latency alongside predictive accuracy.

D Cross-Benchmark Generalization on ProgressLM

PRIMO R1 is evaluated zero-shot on ProgressLM, a benchmark with reference-frame supervision and a different progress-matching formulation. It achieves the strongest reported cross-benchmark outcomes among comparable models and remains effective without reference frames.

  • Evaluation setting: PRIMO R1 is tested on ProgressLM without additional fine-tuning, despite the benchmark using dense reference sequences and models trained directly on that benchmark.ProgressLM provides demonstrations labeled at 0%, 50%, and 100% progress, whereas PRIMO R1 receives no reference frames.
  • Cross-benchmark results: PRIMO R1 attains the lowest NSE and highest PRC among comparable-scale models on the external benchmark.The benchmark reports Normalized Score Error, Progress Rank Correlation, and Answerable False Rejection Rate.
  • Cross-benchmark results: PRIMO R1 maintains AFRR at most 1.0 across vision-based and text-based settings.Several baselines reject much larger fractions of answerable cases, including Intern3.5-VL-38B at 38.3 on the micro setting.

G Question Variations for Task Progress Estimation

The paper uses diverse natural-language prompts to query task progress rather than relying on a single wording. The curated variations span direct percentage questions, procedure-focused formulations, and numerical completion estimates.

  • Prompt diversity: 100 distinct question variations are curated for training and evaluation to improve robustness across linguistic phrasings.The variations are intended to prevent over-fitting to one instruction template and support generalization.
  • Direct progress queries: The prompt set includes direct completion questions such as asking how much of the task has been completed.These formulations request estimates of overall task completion from the visual evidence.
  • Procedure-focused queries: Other prompts ask how complete the overall procedure is or request the current progress percentage.These variants frame progress in terms of the procedure or its present completion level.
  • Numerical estimates: Several variations require numerical estimates, including completion scores between 0 and 100.The set also includes predicted completion rates and quantified progress toward completion.

H Experimental Setup and Config

The experiments use separate configurations for training and inference, with training constrained for computational efficiency and temporal modeling, while inference uses denser, higher-resolution video inputs. Training details are summarized for both SFT and RL stages.

  • Training configuration: Training uses up to 16 video frames at 128 × 28 × 28 pixels on a node with 8 NVIDIA A100 80GB GPUs.The configuration balances computational efficiency with temporal modeling capabilities.
  • Inference configuration: Inference standardizes evaluated models at 32 frames and 256 × 28 × 28 pixels to capture finer visual details.The denser temporal input is used for precise progress estimation and fair comparison across models.
  • Training configuration: Table 9 summarizes the Supervised Fine-Tuning training configuration.
  • Training configuration: Table 10 summarizes the Reinforcement Learning training configuration.
Loading 2603.15600v2…