Source-linked AI summary
Vision-Language Models as Success Detectors
Yuqing Du, Ksenia Konyushkova, Misha Denil, Akhil Raju, Jessica Landon, Felix Hill, Nando de Freitas, Serkan Cabi
TL;DR
The paper addresses how to build success detectors that remain useful across changing tasks and perceptual conditions, a prerequisite for training and evaluating intelligent agents. It reformulates success detection as SuccessVQA, fine-tunes pretrained Flamingo models with human annotations across three domains, and finds stronger robustness than bespoke models under tested language and visual changes, while real-world video detection remains challenging.
Problem
Success detection is needed for agent rewards and evaluation, but real-world success criteria are difficult to define and standard detectors often do not generalize beyond fixed tasks and conditions.
Method
The paper fine-tunes pretrained Flamingo vision-language models on human-annotated SuccessVQA examples across simulated household agents, robotic manipulation, and egocentric human videos.
Results
The pretrained VLM has comparable in-distribution performance and greater robustness to language and visual changes than task-specific reward models, while Ego4D success detection remains challenging.
Takeaways & Limitations
SuccessVQA provides a universal success-detection formulation that can be applied across substantially different domains using pretrained VLMs.
Takeaways & Limitations
The Ego4D benchmark remains very challenging, and further work is needed before deploying the approach as a reward model, including studying false positives and false negatives.
Abstract
from arXiv · showhide
Detecting successful behaviour is crucial for training intelligent agents. As such, generalisable reward models are a prerequisite for agents that can learn to generalise their behaviour. In this work we focus on developing robust success detectors that leverage large, pretrained vision-language models (Flamingo, Alayrac et al. (2022)) and human reward annotations. Concretely, we treat success detection as a visual question answering (VQA) problem, denoted SuccessVQA. We study success detection across three vastly different domains: (i) interactive language-conditioned agents in a simulated household, (ii) real world robotic manipulation, and (iii) "in-the-wild" human egocentric videos. We investigate the generalisation properties of a Flamingo-based success detection model across unseen language and visual changes in the first two domains, and find that the proposed method is able to outperform bespoke reward models in out-of-distribution test scenarios with either variation. In the last domain of "in-the-wild" human videos, we show that success detection on unseen real videos presents an even more challenging generalisation task warranting future work. We hope our initial results encourage further work in real world success detection and reward modelling.
1. Introduction
The paper frames success detection as a prerequisite for training and evaluating intelligent agents, especially where real-world success criteria are difficult to define. It proposes using pretrained vision-language models and the SuccessVQA formulation to generalize across language and perceptual changes.
- Motivation: Real-world success detectors are difficult to define because they must infer environment state and resolve ambiguous goals.The paper contrasts these challenges with engineered detectors for specific games or control tasks.
- Problem: Robotic success detectors should generalize across both language variations in task instructions and visual variations in scene conditions.Examples include paraphrased instructions, camera movement, and additional objects.
- Problem: Standard reward models typically target fixed tasks and conditions, so adapting them to new conditions requires new annotations and retraining.
- Approach: The paper reframes success detection as SuccessVQA, using pretrained Flamingo models and human annotations to learn generalisable detectors from videos and task text.The same approach is intended to support reward modelling or agent evaluation across domains.
- Scope: The method is evaluated across simulated household agents, real-world robotic manipulation, and in-the-wild human videos, with zero-shot generalisation to unseen language and visual conditions.The paper reports that bespoke reward models fail in some such conditions.
2. Related Work
Related work covers multimodal vision-language models, learning reward functions from demonstrations or preferences, and foundation-model-based reward modelling. The paper distinguishes SuccessVQA by applying a generative pretrained VLM across three substantially different domains, including real-world settings.
- Vision-Language Models: Vision-language models provide a foundation for tasks combining language and visual inputs, and this work uses Flamingo as its pretrained model.
- Reward Modelling: Reward modelling is used when rewards are difficult to simulate or hand-engineer, including approaches trained from demonstrations or human preferences.
- Large-Scale Pretraining: Prior foundation-model approaches include language-model reward models and vision-language methods that derive rewards from task descriptions and observations.
- Novelty: This work differs by using a generative VLM built on a frozen language model and applying it across three domains, including real-world domains with human-derived reward labels.
3. SuccessVQA: Success Detection as a VQA Task
SuccessVQA converts human-annotated trajectories into visual question-answering examples and trains a shared Flamingo-based detector across domains. Evaluation predicts success on clips and consolidates those predictions at the episode level to compare with bespoke detectors.
- Formulation: SuccessVQA gives a VLM visual state input and a question asking whether the specified task was successfully completed.
- Formulation: The formulation unifies success detection across simulated agents, robotic manipulation, and in-the-wild human videos using one architecture and training scheme.
- Formulation: The task formulation supports both singular successful states and target behaviours requiring reasoning across multiple frames.
- Dataset Creation: Human-annotated trajectories are split into subsequences, which become SuccessVQA datapoints with task questions and success or failure answers.Known tasks use templates, while narrated actions can be converted into questions with a frozen Flamingo model.
- Evaluation: Evaluation predicts success for each clip and consolidates clip predictions into episode-level accuracy for comparison with bespoke success detectors.
- Experiments: The experiments test generalisation to unseen language tasks and compare Flamingo-based models with domain-specific bespoke models.
4. Language Robustness with Interactive Agents (IA Playroom)
The IA Playroom evaluates success detectors under varied language, behaviours, and unseen tasks in a diverse multi-task environment. Finetuned Flamingo matches bespoke performance on some tests and outperforms it on unseen tasks, though substantial room for improvement remains.
- Environment and challenge: The IA Playroom contains diverse language-specified movement tasks generated from human interactions, making automated success evaluation challenging.The environment includes varied task phrasing and behaviours across people and trained agents.
- Methodology: The bespoke baseline evaluates whole episodes using 32 downsampled frames, learned text embeddings, a transformer, and an instruction-matching contrastive loss.The whole-episode model consistently outperformed the autoregressive baseline.
- Evaluation: The evaluation uses unseen episodes, unseen behaviours from new agents, and unseen tasks across three test sets.The selected model is chosen using balanced accuracy on a held-out validation split.
- Results: Finetuned Flamingo matches the bespoke detector on unseen episodes and unseen behaviour, while the unf finetuned model is near random chance.The comparison uses episode-level balanced accuracy, with human performance around 88% because of inter-rater disagreement.
- Results: On unseen tasks, the bespoke model falls to random chance while FT Flamingo 3B exceeds it by 10%.The authors interpret this as evidence of some semantic understanding, while noting a large margin for improvement.
5. Visual Robustness with Robotic Manipulation
The robotics experiments test whether one success detector can remain accurate across gear-manipulation tasks and natural visual changes. Flamingo is more robust than the bespoke detector to viewpoint and distractor-object shifts, and its predictions support preliminary policy training.
- Tasks and data: The robotics benchmark contains six Panda-arm tasks: inserting or removing small, medium, and large gears in a basket.The experiments use images from a basket camera and target robustness to changes such as viewpoint, lighting, backgrounds, and added objects.
- Methodology: Human operators provide 101,789 demonstrations, with frame-level rewards marking successful states and reverting to zero if a task is undone.A single FT Flamingo 3B model is trained across all six tasks.
- Methodology: The bespoke baseline uses separate ImageNet-pretrained ResNet-18 binary classifiers for each gear task, whereas Flamingo uses one multi-task model.Episode success is determined from the first and last classified clips or frames.
- In-Domain Performance: In-domain, the bespoke detector consistently outperforms FT Flamingo 3B, although Flamingo remains comparable for insertion tasks.Lower Flamingo performance on removal tasks is hypothesized to reflect five times more insertion training data and the opposing temporal structure of insertion and removal.
- Policy training: Filtered behaviour cloning with FT Flamingo 3B yields policy success rates between 50% and 75% under manual resets.For insert-large under harder automated resets, success is 30% with Flamingo and 33% with the bespoke detector.
- Visual Robustness: Under viewpoint changes, bespoke accuracy drops by 10–50 percentage points, while Flamingo decreases by less than 10% and becomes best on 5 of 6 tasks.Under distractor objects, bespoke accuracy drops by 4–30 percentage points, while Flamingo drops by at most 4.5%.
6. Real World Success Detection with Ego4D
The Ego4D SuccessVQA dataset uses human-annotated temporal state changes in egocentric videos to define successful and unsuccessful action clips. Finetuned Flamingo performs substantially better than zero- and few-shot baselines, but generalisation to unseen videos and language tasks remains difficult.
- Limitations: Success detection on completely unseen videos and language tasks remains difficult, making this domain an open avenue for future work.The benchmark is described as substantially more challenging than the prior domains because of its visual and linguistic diversity.
- Dataset construction: Ego4D provides diverse in-the-wild egocentric videos of people performing common tasks, with narrations and annotated object state changes.Each narration centers on an 8-second clip containing action, object, state-change, and critical-frame annotations.
- SuccessVQA construction: PNR frames are treated as the point of success, while frames before PRE provide negative examples that show the action beginning without its successful completion.Flamingo rephrases narrations into SuccessVQA questions for these examples.
- Results: 99% training balanced accuracy and 62% test balanced accuracy were achieved by FT Flamingo 3B on held-out Ego4D clips.The evaluation uses clips from held-out videos because one Ego4D episode can contain multiple narrations and actions.
- Results: Zero-shot and 4-shot Flamingo reached 50% and 52% balanced accuracy, respectively, showing that finetuning is needed for strong in-domain success detection.Finetuning improves performance over random chance, but a substantial train-test gap remains.
7. Conclusion
The paper introduces SuccessVQA as a shared formulation for training pretrained VLM-based success detectors across simulated, robotic, and egocentric domains. These models improve robustness to language and visual changes, while remaining limited by accuracy gaps, expensive inference, and the need for in-domain finetuning.
- Conclusion: SuccessVQA reformulates success detection as a task for pretrained VLMs and applies it across simulated agents, robotic manipulation, and in-the-wild human videos.The formulation is intended to support multi-task success detectors across diverse domains.
- Conclusion: The pretrained VLM has comparable in-distribution performance and greater robustness to language and visual changes than task-specific reward models.The conclusion emphasizes SuccessVQA as a universal formulation that can be applied across substantially different domains.
- Conclusion: VLM-based reward models focus on what to do rather than how to do it, which may support transfer when tasks admit many solutions and fine visual details are unnecessary.The paper contrasts this reward-model role with using VLMs as policies.
- Limitations: The approach still has performance gaps versus bespoke reward models, especially on some robotics tasks, while larger-VLM inference is expensive for online detection.The paper also reports that robust success detection currently requires sufficient in-domain finetuning because zero- and few-shot performance is inadequate.
- Future work: Improving inference speed, distilling to smaller models, and evaluating false-positive and false-negative effects are identified as priorities before deployment for policy learning.The authors also suggest studying larger models, few-shot generalisation, and shared finetuning across datasets.
A. Simulated household domain
The simulated-household evaluation uses human annotations of agent continuations to determine success or failure and rank policies. Human annotators also mark when success or failure occurs.
- Evaluation protocol: Each agent receives context up to a continuation point, after which its policy completes the trajectory for evaluation.The continuation is assessed offline rather than during policy execution.
- Evaluation protocol: Human annotators label each continuation as successful or failed and identify the point at which success or failure occurs.These annotations provide the ground-truth signal for evaluating agent capabilities.
- Policy ranking: Agent policies are ranked using the proportion of successful human annotations they receive.The ranking aggregates judgments across evaluated continuations.
A.1. Baseline Evaluation Models
The STS baseline evaluation models automate human assessment of agent trajectories by predicting success from episodes or individual frames. Their architectures combine visual and language inputs, with episode-level success determined directly or from any successful frame.
- Motivation: Human annotation costs grow with evaluations for each new task and agent, motivating automated models that classify success from behavior and task utterances.The desired evaluator conditions on an episode of agent behavior and the input task instruction.
- Baseline models: The STS uses whole-episode and autoregressive evaluation models, with reward annotations for each episode aggregated by majority voting.These are the two baseline evaluation-model families described for the standardized scenarios.
- Whole episode evaluation models: Whole-episode models downsample episodes to 32 frames, embed images and text, and use a transformer with success and instruction-matching prediction heads.The auxiliary matching head predicts whether an instruction belongs to the episode or has been shuffled.
- Autoregressive evaluation models: Autoregressive models process images and language per frame through a multimodal transformer and LSTM, predicting success or no-success at each frame.An episode is classified as successful if any single frame is predicted successful.
- Reference performance: Human balanced accuracy on the IA Playroom STS evaluation is around 88%.This value is provided as the reference level for the zero-shot episode-level evaluation table.
B.1. Ground truth in robotics domain
Robotics episodes receive ground-truth success labels based on their initial and terminal states. An episode is successful only if it starts in failure and ends in success; all others are negative.
- Ground-truth labels are assigned to complete robotics episodes.
- An episode is positive only when it starts in a failure state and terminates in a success state.
- All episodes that do not meet both state-transition conditions are labeled negative.
B.2. Data Efficiency in robotics domain
The study evaluates Flamingo-based success detection in a lower-data regime using 100–200 episodes per task and the same in-domain test set. For five of six tasks, Flamingo is less affected by reduced training data than the ResNet-based model.
- The lower-data experiments train each task on only 100–200 episodes.This is 100x fewer episodes than the tens of thousands used in the preceding experiments.
- For five of six tasks, the Flamingo-based model is less affected by the smaller dataset than the ResNet-based model.