PaperlayerView exact revision on arXiv

Source-linked AI summary

PROGRESSLM: Towards Progress Reasoning in Vision-Language Models

Jianshu Zhang, Chengxuan Qian, Haosen Sun, Haoran Lu, Dingcheng Wang, Letian Xue, Han Liu

arXiv:2601.15224v2cs.CVcs.CL

TL;DR

The paper asks whether VLMs can infer task progress from partial observations rather than merely describe static content. It introduces PROGRESS-BENCH and a two-stage retrieval-plus-simulation approach, finding that existing models struggle while explicitly trained PROGRESSLM-3B reaches performance comparable to or surpassing GPT-5 on the benchmark.

  • Problem

    It remains unclear whether VLMs can estimate how far a task has progressed from a partial observation, which requires reasoning over long-horizon dynamics beyond static perception.

  • Method

    The paper introduces PROGRESS-BENCH and studies human-inspired episodic retrieval followed by mental simulation through prompting and training on PROGRESSLM-45K.

  • Results

    Existing VLMs show limited progress reasoning, while training-based PROGRESSLM-3B achieves performance comparable to or surpassing GPT-5 on PROGRESS-BENCH.

  • Takeaways & Limitations

    Robust progress estimation emerges when coarse anchor retrieval and fine-grained reasoning are explicitly learned.

  • Takeaways & Limitations

    The benchmark and training data focus on robotic manipulation with relatively clear, monotonic progress, limiting direct generalization to open-ended or non-monotonic tasks.

Abstract

from arXiv · show

Estimating task progress requires reasoning over long-horizon dynamics rather than recognizing static visual content. While modern Vision-Language Models (VLMs) excel at describing what is visible, it remains unclear whether they can infer how far a task has progressed from partial observations. To this end, we introduce Progress-Bench, a benchmark for systematically evaluating progress reasoning in VLMs. Beyond benchmarking, we further explore a human-inspired two-stage progress reasoning paradigm through both training-free prompting and training-based approach based on curated dataset ProgressLM-45K. Experiments on 14 VLMs show that most models are not yet ready for task progress estimation, exhibiting sensitivity to demonstration modality and viewpoint changes, as well as poor handling of unanswerable cases. While training-free prompting that enforces structured progress reasoning yields limited and model-dependent gains, the training-based ProgressLM-3B achieves consistent improvements even at a small model scale, despite being trained on a task set fully disjoint from the evaluation tasks. Further analyses reveal characteristic error patterns and clarify when and why progress reasoning succeeds or fails. Website: https://progresslm.github.io/ProgressLM/

1 Introduction

The paper frames task-progress estimation as long-horizon reasoning from partial observations, then introduces PROGRESS-BENCH and a human-inspired two-stage approach to study and improve it.

  • VLMs can describe visible content but must reason over long-horizon task dynamics to estimate completion from an intermediate observation.
  • PROGRESS-BENCH evaluates numerical progress estimation from a single observation across demonstration modality, viewpoint correspondence, and answerability.
  • The proposed approach combines episodic retrieval of a coarse task anchor with mental simulation of state evolution toward the observation.
  • Experiments on 14 VLMs find limited and unstable progress reasoning, including sensitivity to modality and viewpoint changes and poor handling of unanswerable cases.
  • Training-free prompting provides only conditional gains, whereas training produces PROGRESSLM-3B with performance comparable to or surpassing GPT-5 on PROGRESS-BENCH.

2 PROGRESS-BENCH

PROGRESS-BENCH tests whether models can place a single observation within a demonstrated task, while varying modality, viewpoint, sampling, and answerability to probe progress reasoning.

  • Each benchmark instance pairs a full-task demonstration with an intermediate observation and requires a normalized progress score or N/A when the instance is unanswerable.
  • The benchmark varies vision versus text demonstrations, same-view versus cross-view observations, and answerable versus mismatched cases.
  • Vision demonstrations use key frames, whereas text demonstrations provide stepwise action descriptions requiring implicit state-change integration.
  • Intermediate observations are sampled within segments between key steps, with progress assigned by linear interpolation and additional boundary sampling near transitions.
  • Answerability augmentation creates N/A cases by modifying either the demonstration or observation while keeping the other fixed.
  • The benchmark is built from temporally coherent robotic-manipulation trajectories using discrete step-level labels and within-step interpolation for fine-grained supervision.

3 Towards Progress Reasoning

The paper operationalizes progress reasoning as coarse anchor retrieval followed by fine-grained mental simulation, exploring both structured prompting and explicit training.

  • The two-stage formulation treats progress estimation as reasoning over a latent task trajectory rather than matching an observation to a fixed timestamp.
  • Stage one retrieves a representative reference step, and stage two simulates how the task state evolves from that anchor to the current observation.
  • Training-free prompting exposes retrieval and simulation through structured reference and score fields without parameter updates.
  • The training-based approach constructs PROGRESSLM-45K from manipulation tasks nonoverlapping with PROGRESS-BENCH to encourage generalizable reasoning.
  • Supervised fine-tuning first internalizes the two-stage reasoning pattern using demonstrations, observations, and reasoning sequences containing reference and score targets.
  • A second GRPO-based reinforcement-learning stage rewards structured reasoning, accurate reference retrieval, and precise progress estimation.

4 Evaluation on PROGRESS-BENCH

PROGRESS-BENCH evaluates VLM progress estimation across answerability, modality, and viewpoint conditions. Existing models are unstable, training-free reasoning offers conditional benefits, while targeted training improves performance and cross-view robustness.

  • Experimental Setup: The evaluation covers 14 VLMs from 2B to 72B using direct prediction, training-free reasoning, and PROGRESSLM training on disjoint tasks.Metrics capture pointwise accuracy, temporal consistency, and answerability awareness.
  • Performance on Answerable Scenarios: Current VLMs show limited and unstable progress estimation, with vision-based demonstrations outperforming text-based demonstrations.Several models produce collapsed or distorted progress rankings, reflected in abnormally low, negative, or undefined PRC values.
  • Training-free Progress Reasoning: Training-free reasoning benefits larger models conditionally but can marginally worsen NSE or AFRR for smaller models.Large models gain mainly in PRC and occasionally NSE, whereas smaller models may follow the format without improving progress understanding.
  • Training-based Progress Reasoning: PROGRESSLM consistently improves the base 3B model across answerable metrics, with PROGRESSLM-RL-3B achieving the strongest macro-averaged NSE and PRC.The result indicates that targeted supervision and optimization can produce effective progress reasoning at small scale.
  • Robustness to Viewpoint Changes: Cross-view observations yield higher NSE and lower PRC than same-view observations across models, especially for small and medium-sized models.The pattern suggests reliance on viewpoint-dependent visual similarity rather than viewpoint-invariant progress reasoning.
  • Robustness to Viewpoint Changes: PROGRESSLM-3B-RL shows smaller same-view-to-cross-view gaps, while training-free reasoning provides limited, capability-dependent cross-view gains.When training-free gains occur, they primarily appear in PRC and may come at the expense of same-view performance.
  • Unanswerable Case Recognition: Most models produce progress scores for inherently ambiguous inputs, whereas PROGRESSLM recognizes unanswerable cases under both demonstration settings.PROGRESSLM-3B-RL achieves the highest or near-highest unanswerable recognition accuracy, although high UDA can coexist with excessive rejection of answerable cases.

5 Further Analysis

Further analyses identify recurring prediction and error patterns and examine how the two-stage reasoning process operates. Explicitly trained models produce smoother score distributions and more robust per-sample errors.

  • Predicted Score Distributions: Predicted progress scores commonly collapse to extremes, cluster around heuristic anchors, concentrate near 50%, or vary smoothly across the full range.The four patterns distinguish coarse or uncertain responses from sensitivity to intermediate task states.
  • Per-sample Error Distribution: Smaller models have broad, heavy-tailed error distributions, while larger models produce errors more concentrated near zero.The distributions reflect instability in smaller models and more consistent estimates in larger ones.
  • Demonstration Modality: Text-based demonstrations require implicit state accumulation because action descriptions must be integrated over time rather than matched to explicitly shown world states.Steps involving the same object can differ through an unobserved state variable, such as whether the pumpkin was already placed on the plate.
  • Per-sample Error Distribution: PROGRESSLM-3B-SFT and PROGRESSLM-3B-RL tighten errors relative to the base 3B model, with RL especially suppressing extreme-error cases.Explicit progress learning improves both average accuracy and per-sample robustness.

Are the two reasoning stages truly coupled? Yes—the first-stage anchor directly constrains

The two reasoning stages are coupled: the retrieved episodic anchor constrains the later progress estimate. Training-based scaling also improves performance, including for a 7B model.

  • Are the two reasoning stages truly coupled?: A strong diagonal pattern links the Episodic Retrieval Anchor Index with the Score-Aligned Demonstration Index, confirming that anchor retrieval guides score estimation.The alignment indicates that the first stage is not auxiliary to the second-stage estimate.
  • Scalability: The training-based approach consistently outperforms no-thinking and training-free baselines on the larger Qwen2.5-VL model.The 7B model shows substantial gains despite only one RL training epoch, especially in the vision-based setting.

6 Related Work

Prior progress-estimation methods often depend on task-specific models, latent feature distances, or indirect sequence-based objectives. ProgressLM instead targets general progress reasoning from partial observations, a setting requiring long-horizon state tracking.

  • Progress Estimation: Early progress-estimation methods rely mainly on task-specific or expert models trained within fixed tasks or environments.This design limits generalization beyond the training distribution.
  • Progress Estimation: Other approaches estimate progress through latent feature-space distances, trajectory reordering, or pairwise relative comparisons.Indirect formulations couple estimates to sequence context or surrogate objectives rather than directly predicting progress from one observation.
  • Progress Reasoning in VLMs: Recent VLMs improve static visual reasoning but progress reasoning requires long-horizon evolution modeling, latent state-transition inference, and an implicit world model.This distinction separates snapshot-level perception from dynamic reasoning under partial observation.

7 Conclusion

The paper frames progress estimation as a dynamic reasoning problem and introduces PROGRESS-BENCH to evaluate it systematically. Experiments show that existing VLMs struggle, while robust estimation requires explicitly learning coarse anchor retrieval and fine-grained reasoning.

  • PROGRESS-BENCH evaluates progress reasoning from a single observation under controlled variations of modality, viewpoint, and answerability.
  • 14 VLMs exhibit sensitivity to modality and viewpoint changes, degenerate progress predictions, and weak handling of unanswerable cases.
  • Robust progress estimation emerges only when coarse anchor retrieval and fine-grained reasoning are explicitly learned.

8 Limitations

The benchmark and training data are centered on robotic manipulation with relatively clear, monotonic progress. Extending the findings and ProgressLM to open-ended or substantially different task families may require additional data or adaptation.

  • PROGRESS-BENCH focuses on robotic manipulation tasks with relatively clear and monotonic progress.This may limit generalization to open-ended scenarios with ambiguous goals or non-monotonic dynamics.
  • ProgressLM is trained on curated manipulation data with similar structural properties.Extending it to substantially different task families may require further data or adaptation.
  • The paper identifies anomaly detection, online reinforcement-learning rewards, and task-specific reward-model training as practical applications of progress reasoning.
  • The paradigm is also discussed for general agents such as web agents, where progress signals may support inference-time scaling and self-improving behaviors.

A.2 Vision-Based Unanswerable Data

Visual unanswerable samples are constructed by editing realistic manipulation images to violate the corresponding instruction, followed by human quality filtering. The broader pipeline organizes data across modalities, viewpoints, answerability settings, robot platforms, and evaluation stages.

  • Visual unanswerable sample construction: Visual unanswerable samples are generated through edit-prompt generation, image editing, and human filtering.The retained images must violate the corresponding instruction while maintaining visual realism.
  • Visual unanswerable sample construction: The editing strategies are color change, object replacement, and occlusion or removal of key objects.
  • Data organization: Samples are distributed across Benchmark, SFT, and RL splits and organized by demonstration–observation setting and robot platform.
  • Visual unanswerable sample construction: 23.5% of edited images meet the quality criteria and are retained as the visual negative dataset.
  • Human activity benchmark: The human activity benchmark uses demonstrations and test frames from different executions to introduce domain shifts in appearance, style, and environment.It contains 587 samples spanning 129 task goals across manipulation categories.
  • Data organization: The construction pipeline transforms trajectories from Franka, AgileX, Humanoid, and UR5e platforms into visual and text demonstration settings, including same-view, cross-view, and unanswerable cases.
  • Human activity benchmark: The human benchmark challenges models with domain, environmental, execution, and viewpoint variation beyond controlled laboratory settings.

B.2 Text-Based Unanswerable Data Generation.

The supplied passages describe how progress reasoning is grounded in retrieved demonstration anchors and local comparison across visual and textual modalities. Visual same-view inputs align most tightly, cross-view inputs introduce uncertainty, and text inputs remain feasible but more retrieval-sensitive.

  • Vision same-view: Same-view visual reasoning retrieves a near-completion demonstration step and compares subtle state differences to estimate progress.This behavior aligns with a strong diagonal diagnostic structure and low NSE.
  • Vision cross-view: Cross-view reasoning can retrieve a semantically aligned late-stage anchor despite viewpoint changes by using task-relevant state changes rather than pixel similarity.
  • Modality comparison: Vision-based demonstrations provide dense state information for episodic retrieval and local mental simulation, yielding higher PRC and lower AFRR than text-based inputs.
  • Mechanistic interpretation: Visualizations indicate that effective progress estimation relies on a semantically aligned visual anchor and local reasoning around that reference.Same-view coupling is tighter, while cross-view uncertainty weakens but does not break it.
  • Text-based reasoning: Text-based demonstrations can support progress estimation when the model correctly grounds an abstract step to the observed physical state.In the cup-stacking example, the model retrieves Step 3 and predicts 60%, matching the ground truth.
  • Text-based reasoning: Text inputs show higher NSE and AFRR because multiple physical states may correspond to the same textual instruction.Correct textual-anchor retrieval nevertheless permits accurate local progress reasoning.
  • Text-based reasoning: Text-based progress estimation remains feasible but is more sensitive to retrieval errors, making explicit coupling between retrieval and mental simulation important.
  • Overall mechanism: Across settings, progress estimation behaves as a coupled process in which episodic retrieval precedes mental simulation and score estimation.Uncertainty increases from vision same-view to cross-view and text without removing this coupling.

C.4 In the Wild Generalization Analysis

In-the-wild human activities expose substantial domain-shift challenges for progress estimation, but coupled reasoning with episodic anchoring improves calibration and robustness across modalities.

  • In-the-wild generalization: Human activities degrade most models’ NSE and PRC while increasing AFRR, reflecting harder calibration and abstention under subtle hand–object interactions.These conditions differ from robotic demonstrations in embodiment, motion dynamics, object appearance, and execution variability.
  • In-the-wild generalization: Larger models improve PRC and reduce AFRR, yet their NSE remains high, showing that capacity alone does not ensure fine-grained progress calibration.Smaller models show compounded errors in episodic alignment and progress estimation.
  • Demonstration modality: Vision-based demonstrations outperform text-based demonstrations on NSE and PRC, while text inputs produce higher AFRR because ambiguous instructions weaken episodic retrieval.The modality gap is tied to multiple physical states mapping to one textual step in human activities.
  • Coupled progress reasoning: PROGRESSLM-RL-3B achieves the lowest average NSE with competitive PRC and controlled AFRR, and reinforcement learning consistently improves calibration over SFT.The gains suggest that coupling episodic retrieval with progress estimation is especially beneficial under domain shift.
  • Qualitative evidence: Qualitative human-activity examples show accurate estimates when the model retrieves semantically aligned demonstration steps and compares fine-grained state differences.A jar-opening example predicts 43% against a 41% ground truth.
  • Unanswerable recognition: Explicit inference-time reasoning improves unanswerable detection, especially for text demonstrations, by requiring semantic comparison before assigning progress.Training-based supervision reinforces this verification process, and their combination yields more reliable recognition across modalities and scales.
Loading 2601.15224v2…