PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboJev: An Open Vision-Language Judge for Robot Success Detection

Josef Liyanjun Chen

arXiv:2610.33718v1cs.ROcs.LG

TL;DR

Robot manipulation videos often lack machine-readable success labels, limiting straightforward evaluation of whether tasks were completed. RoboJev is an open-weight vision-language episode judge that predicts success from instructions and video frames. It reaches 95.2% balanced accuracy on held-out SO-101 data, while the same-input base model has a slight AUROC edge on unseen domains.

  • Problem

    Success in robot manipulation videos is often not machine-readable and cannot be readily derived from low-level heuristics.

    Source: §1 · p. 1

  • Method

    RoboJev fine-tunes an open-weight vision-language model to predict episode success from a task instruction and video frames, returning a calibrated probability.

    Source: §1 · pp. 1–2

  • Results

    95.2% balanced accuracy on held-out SO-101 rollouts; on unseen domains RoboJev reaches 0.890 AUROC versus 0.900 for the same-input base model.

    Source: §1 · p. 2, §5.2 · p. 5

  • Takeaways & Limitations

    The released checkpoint, predictions, and inference and evaluation code support per-episode inspection of an open-weight tool for policy evaluation.

    Source: §8 · p. 7

    Evaluation covers eight testing domains, and validation-set temperature calibration does not establish calibration in other domains.

    Source: §7 · p. 7

Abstract

from arXiv · show

Many robot manipulation videos do not indicate whether the task was successfully completed. RoboJev-27B is an open-weight vision-language model that takes a task instruction, 10 frames of a manipulation attempt, and (if available) final frames from auxiliary cameras and predicts whether the robot accomplished the task. RoboJev-27B is a variant of JEV-27B-VL, trained with a rank-32 LoRA adapter on the language model only (keeping the vision encoder frozen) using 52,598 episode records. It also uses a source-specific KL penalty that applies only to research data, anchoring predictions closer to the original model on those sources. It uses a calibrated two-token readout that outputs success/failure probabilities in a single forward pass. On 268 held-out episodes of the same robot model (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) to infer on a single H100, which is about \$0.19 per 1,000 success predictions at \$2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also compare against strong API-based frontier VLMs, which take in fewer input frames (4 images from the primary camera only); note that this comparison is not strictly like-for-like because RoboJev-27B is given more frames at inference.

1 Introduction

Robot manipulation videos often lack machine-readable success labels, which are useful for review queues, learned terminal rewards, and aggregate evaluation. The paper fine-tunes an open-weight episode judge and evaluates it on held-out and out-of-domain episodes.

Motivation

  • Success labels support review queues, learned terminal rewards, and aggregate evaluation, while low-level heuristics may not readily determine success.Source: §1 · p. 1

Approach

  • RoboJev fine-tunes JEV-27B-VL as an open-weight detector using a task instruction, ten primary-camera frames, and available final auxiliary-camera frames.Source: §1 · p. 1
  • The judge returns a calibrated success probability by scoring true/false tokens, enabling reuse across tasks and environments without generating explanations.Source: §1 · p. 2

Evaluation

  • 95.2% balanced accuracy on held-out SO-101 rollouts; on unseen domains RoboJev reaches 0.890 AUROC versus 0.900 for the same-input base model.The authors describe this as evidence of an inspectable evaluation tool, not a generalizable improvement from fine-tuning.Source: §1 · p. 2

2 Related work

Prior work studies visual outcome questions, trajectory rewards, and transferable reward signals. RoboJev instead uses supervised success/failure labels for a terminal prediction and does not evaluate downstream policy learning.

Related methods

  • SuccessVQA, RoboCLIP [4], and Video-Language Critic [5] study visual outcome questions or trajectory rewards, whereas RoboJev reports a detector without evaluating downstream policy learning.Source: §2 · p. 2
  • RoboJev uses supervised success/failure labels for terminal two-token prediction, unlike the objectives and methods of RoboReward and Robometer [7].The paper uses converted RoboReward data for training but does not evaluate its model as a direct baseline.Source: §2 · p. 2

Data sources

  • Open X-Embodiment, DROID [10], and LeRobot community datasets provide demonstrations across embodiments and tasks, with differing viewpoints, labels, and sources.Source: §2 · p. 2

Positioning

  • The system uses a pretrained vision-language model with low-rank adaptation of its language-model component and a frozen vision encoder, rather than proposing a novel architecture.Its contributions include an open-weight judge, a training and inference recipe, and an evaluation artifact describing trade-offs.Source: §2 · p. 2

3 A task-conditioned episode judge

RoboJev judges task completion from a task instruction and sampled video views, then adapts a pretrained vision-language model using mixed episode data and source-specific anchoring. Held-out validation data is used to select adapter scaling and calibrate probabilities.

3.1 Input and decision readout

  • The readout uses logits for false and true tokens to return a calibrated probability of task completion at the end of the episode, not progress or a natural-language explanation.The input includes ten evenly spaced primary-camera frames and the last image from each available auxiliary camera.Source: §3.1 · p. 2

3.2 Training data

  • 52,598 episodes from robot failures, demonstrations, and simulation interaction data form the fitting set, with validation datasets and evaluated community users held out.The source counts are post-split records and are not necessarily independently de-duplicated.Source: §3.2 · p. 2
  • Community annotations are filtered for teacher agreement and confidence, producing 10,216 training examples after validation and held-out-robot exclusions.Training examples from 106 owners in the human-labeled community check are also discarded to reduce label leakage.Source: §3.2 · p. 3
  • Augmentation uses truncated positive episodes and mismatched instructions as heuristic negatives, which may introduce label noise when truncation still satisfies a shorter instruction.Augmentation and repetition yield 102,172 training examples in an epoch.Source: §3.2 · p. 3

3.3 Adaptation and calibration

  • Rank-32 LoRA adapts the language model while the vision encoder and visual merger stay frozen; KL anchoring is applied to RBM/research sources but not community or ArmnetBench records.Synthetic negatives receive weight 0.5, while original examples use source-balanced weights.Source: §3.3 · p. 3
  • Adapter scale is selected using held-out community AUROC subject to a validation-source constraint, and temperature scaling is fitted by minimizing validation negative log-likelihood.The published test sets were inspected during model creation, so they are not fully one-shot blind tests.Source: §3.3 · p. 3

4 Evaluation protocol

The evaluation covers held-out SO-101 rollouts, unseen lab/robot domains, and clearly labeled community videos. Comparisons span different model and input protocols, while metrics and uncertainty intervals are computed from episode-level predictions.

4.1 Three complementary test settings

  • The evaluation includes SO-101 rollouts, eight unseen lab/robot domains with 459 rollouts, and community videos, of which 135 have definite binary labels.The SO-101 task-held-out subset contains 80 of 268 runs; 65 community episodes remain ambiguous.Source: §4.1 · p. 4, Table 1 · p. 4

4.2 Model and protocol comparisons

  • RoboJev and the dense base use ten primary-camera frames plus available final auxiliary views, while recorded frontier API runs use four primary-camera frames.The 29 model/protocol entries contain 20,628 predictions, but differing protocols do not disentangle model capability from visual input.Source: §4.2 · p. 4

4.3 Metrics and uncertainty

  • The study reports AUROC and balanced accuracy, uses a 0.5 binary threshold, and reports accuracy for community videos.New intervals use stratified episode bootstrap with 4,000 replicates; they do not capture uncertainty from generalization to new labs or within-domain correlation.Source: §4.3 · pp. 4–5

5 Results

RoboJev-27B performs strongly on held-out SO-101 episodes, but adaptation does not improve transfer to unseen domains, where calibration also worsens. Community accuracy applies only to unambiguous clips, while the temporal-context and anchoring analyses point to better unseen-domain AUROC than final-frame-only or unanchored variants.

5.1 Strong SO-101 discrimination

  • 95.2% balanced accuracy and AUROC 0.993 on 268 held-out SO-101 runs, versus 88.1% and AUROC 0.979 for the base model with the same ten inputs.The paired balanced-accuracy difference was 7.08 percentage points (95% interval [3.04, 11.45]).Source: §5.1 · p. 5

5.2 Transfer reveals the limits of specialization

  • 78.2% balanced accuracy (AUROC 0.890) on 459 unseen-domain episodes was slightly below the base model’s 79.4% (AUROC 0.900); the AUROC difference was −0.0092 (95% interval [−0.0218, 0.0026]).The interval includes zero, and RoboJev’s unseen-domain ECE of 0.148 was much higher than on SO-101.Source: §5.2 · p. 5
  • In a different setting, RoboJev had an AUROC 0.0282 higher than the recorded four-frame Astra run; the 95% difference interval was [0.0049, 0.0514], and the comparison was not controlled for input images.Source: §5.2 · p. 5

5.3 Community accuracy and ambiguous outcomes

  • RoboJev correctly labeled 126 of 135 unambiguously labeled community episodes (93.3% accuracy; AUROC 0.984); competing methods’ accuracies differed by only one or two episodes.RoboJev and Astra differed by 0.74 percentage points (95% interval [−4.44, 5.93]).Source: §5.3 · p. 5
  • 65 ambiguous clips (32.5% of community episodes) were not used to extrapolate accuracy to all community episodes and remain in the released data artifact.The authors identify these clips as cases where a detector should not confidently label an unobservable event as ground truth.Source: §5.3 · p. 5

5.4 Temporal evidence and anchoring

  • The released model had higher unseen-domain AUROC than the unanchored variant (0.890 vs. 0.847), while using only the final frame yielded AUROC 0.776 on unseen domains.These results come from a single run per model; the authors note temporal context is useful when a task label cannot be inferred from one image.Source: §5.4 · p. 5

5.5 Calibration changes across domains

  • Calibration worsened across domains: ECE rose from 0.032 on SO-101 to 0.148 on unseen domains, with Brier scores of 0.033 and 0.162, respectively.The unseen-domain reliability plot shows frequent underestimation of the true success rate; the authors suggest local labeled validation when choosing a deployment threshold.Source: §5.5 · p. 5

6 Serving and intended uses

RoboJev supports post-hoc episode review and offline evaluation, with tools for scoring attempts and integrating terminal rewards. Its measured inference cost is low at the episode level, but catalogue-scale and policy-learning claims remain bounded by compute-only estimates and untested downstream impact.

6 Serving and intended uses

  • The released overlay scores attempt prefixes, while a LeRobot plugin, processor, and grader support episode rewards, thresholding, and stored probabilities.Scoring requires extracting log probabilities for the two answer tokens; a standard vision-language inference endpoint can serve the merged checkpoint.Source: §6 · pp. 5–6
  • Figure 4 compares matched-input adapted-minus-base results with deployment costs and median response times, but the infrastructure and input budgets differ.Its unseen-domain interval includes zero.Source: Figure 4 · p. 6
  • Figure 5 presents post-hoc calibration diagnostics, with ten equal-width bins, Wilson intervals, outcome-separated histograms, ECE, and Brier score.It is an evaluation-set diagnostic, not another calibration fit.Source: Figure 5 · p. 6
  • 0.257 seconds is the median serial H100 response time per episode; the corresponding compute estimate excludes media processing, queueing, and serving costs.The measurement is operational, not a controlled hardware comparison.Source: §6 · pp. 6–7
  • About 4.9 GPU-hours is the estimated serial compute needed to score the sampled catalogue; this scenario excludes media and service costs.It is an estimate from measured latency, not a report that the released checkpoint has already regraded the catalogue.Source: §6 · p. 7
  • Terminal-reward integration was implemented and tested, but its effect on policy learning was not measured; online use would require closed-loop evaluation and an operating threshold.A controller could exploit reward errors or terminate attempts too early.Source: §6 · p. 7

7 Limitations and data responsibility

The paper identifies limits in what the videos and evaluation can establish, and cautions against treating confidence or low average error as evidence of safety. Data-label provenance, domain coverage, calibration transfer, and licensing also bound interpretation and reuse.

7 Limitations and data responsibility

  • Sparse frames or insufficient visibility can prevent certification of success or failure, and brief events such as contact, spillage, or recovery may be missed.Wrist-camera views and videos ending too soon are cited as visibility constraints.Source: §7 · p. 7
  • Community training labels are model-generated, test labels come from one human annotator, and pretraining overlap cannot be ruled out; related benchmarks were also explored previously.The authors therefore do not regard the intervals as a completely prospective preregistered evaluation.Source: §7 · p. 7
  • Eight testing domains provide limited evidence about practical success, and calibration on a validation set does not establish calibration in other domains.Wrist-camera and long-horizon tasks remain difficult, and sensitivity to instruction phrasing was not checked.Source: §7 · p. 7
  • Neither low average error nor high confidence makes the detector a safety monitor.Source: §7 · p. 7
  • The training collection has heterogeneous licensing and provenance; some research datasets lack specific licenses and some community sources are no longer publicly accessible.The paper reports these gaps rather than assuming public availability permits redistribution of all underlying assets.Source: §7 · p. 7

8 Conclusion

RoboJev is an open-weight, task-conditioned success detector that returns a probability in one forward pass. The paper releases the model and evaluation records and code for per-episode inspection.

8 Conclusion

  • RoboJev returns a success probability from a task description and episode in a single forward pass.The conclusion describes it as an open-weight, task-conditioned detector.Source: §8 · p. 7
  • The release includes evaluation predictions and inference and evaluation code for browsing results per episode.The release also lists weights and inference tools, evaluation records, a data explorer, and an interactive demo.Source: §8 · p. 7
  • The paper identifies matched-input frontier comparisons, more human-labelled domains, and downstream-policy reward use as future work.It does not claim better policies learned from the judge.Source: §8 · p. 7

A Reproducibility and protocol ledger

The reproducibility ledger records the released model settings and names the inference protocols used for model comparisons and diagnostics.

A Reproducibility and protocol ledger

  • The artifact includes a calibration file, inference code, metric and plot-generation code, and benchmark-record snapshots.The paper states that model weights and private training videos are not included in the artifact.Source: A Reproducibility and protocol ledger · p. 10

A.1 Named protocols

  • The release protocol uses ten evenly spaced primary-camera frames and final frames from other cameras when available.This is the clip-v1 protocol used for the current release and matched-input base-model row.Source: §A.1 · p. 10
  • The clip-v1 protocol uses four primary-camera frames at the beginning, one-third, two-thirds, and end of the rollout.It was used for recorded decision-API and frontier baselines and earlier model variants.Source: §A.1 · p. 10
  • The final-frame diagnostic uses the final frame from every available camera, while the video protocol uses a baseline's full-clip interface.The video protocol is identified as a distinct input budget.Source: §A.1 · p. 10

B Complete metric inventory

The metric inventory collects available SO-101, unseen-domain, and clearly labelled community results across models and protocols. It also records the scope of the withheld-task subset and model-source references.

B Complete metric inventory

  • The inventory tables include all models and protocols, including deprecated entries, and evaluate each model on episodes with binary labels.A dash means predictions were not recorded, not zero performance.Source: B Complete metric inventory · p. 10
  • Table 3 covers available SO-101 and unseen-domain results, reporting balanced accuracy, AUROC, and available binary-labelled prediction counts.Its caption identifies dense-v1, clip-v1, and whole-video protocols.Source: Table 3 · p. 11
  • Table 4 covers 135 clearly labelled community episodes; the other 65 community clips are marked unclear.The caption specifies that the reported community metric is ordinary accuracy, not balanced accuracy.Source: Table 4 · p. 11

B.1 The two withheld tasks

  • The cable_unclip and ring_insert subset contains 80 episodes, split evenly between 40 successes and 40 failures, and differs from the full 268-run SO-101 evaluation.Frontier rows in this subset still use fewer images.Source: §B.1 · p. 12

B.2 Model identities and primary links

C.1 Training records and validation

The training and validation ledger documents record counts, label provenance, and validation-set discrepancies, then describes source provenance and the scope of release verification.

C.1 Training records and validation

  • The fit set totals 52,598 records, with 102,172 examples after repetition and augmentation.The records span selected RBM-1M sources, ArmnetBench, teacher-labelled community data, demonstrations, ViFailback, and converted DROID episodes.Source: §C.1 · p. 13
  • A 28-record discrepancy between community-loader records and validation IDs has an unknown cause, and does not by itself establish train/test contamination.Repeated keys are assigned to the same split by a deterministic owner/robot split.Source: §C.1 · p. 13
  • Training labels require high-confidence Sol and Astra labels on four primary-camera frames, with Opus used when available for agreement and veto.The saved target count matches the final consensus file used by the executed run.Source: §C.1 · p. 13

C.2 Originating corpora and processed subsets

  • The RBM-1M harvest was not pinned to a complete historical repository revision, limiting byte-for-byte reconstruction of all training inputs.The ledger identifies 41 RBM source keys, 38 of which contribute fit records; these are not 41 independent originating datasets.Source: §C.2 · p. 13

C.3 Licences and figure credits

  • Training-data provenance and licensing remain incomplete, and the source bundle does not include the training corpus.The paper notes that some accessed dataset copies lack explicit licences and that the complete licence chain is not fully documented.Source: §C.3 · p. 14

D Verification and scope of the release

  • The release's metric scripts verify arithmetic and coverage over released prediction records, not new large-scale inference or completeness of historical API calls.The paper also disclaims claims of state-of-the-art performance under matched inputs, general probability calibration, safety certification, or improved downstream policies.Source: D Verification and scope of the release · pp. 14–15

Related papers

Loading 2610.33718v1…