PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoTSR++: Egocentric Spatiotemporal Reasoning for Task Progress Understanding

Xiaoda Yang, Can Wang, Yuxiang Liu, Pengfei Zhou, Jianwen Lou, Shuicheng Yan

arXiv:2609.31751v1cs.CV

TL;DR

Egocentric task-progress judgments should compare task-relevant visual states with the goal, rather than rely on timestamps or presentation order. EgoTSR combines order-swapped evaluation, bidirectional supervision, and a CoT-to-Tag curriculum; it reports 92.4% long-horizon accuracy with a 0.1-point Gap.

  • Problem

    Vision-Language Models remain unreliable at judging egocentric task progress, where retries and corrections can make chronological order diverge from progress toward the goal.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    EgoTSR pairs order-swapped benchmark queries with bidirectional trajectory supervision and a CoT-to-Tag curriculum for evidence-grounded, order-robust state comparison.

    Source: §1 · pp. 1–2, §5.1 · p. 5, §5.2 · p. 5

  • Results

    92.4% long-horizon average accuracy with a 0.1-point long-horizon Gap; failure-aware supervision raises Failure Avg. by 11.8 points and Recovery Acc. by 11.2 points.

    Source: §8 · p. 12

  • Takeaways & Limitations

    Goal-conditioned state comparison is an explicit and transferable formulation for egocentric task-progress understanding.

    Source: §8 · p. 12

    Core training assumes successful, approximately monotonic executions; evaluation centers on manipulation, with closed-loop validation and broader recovery coverage left for future work.

    Source: §3.1 · p. 3, §7.8 · p. 12

Abstract

from arXiv · show

Vision-Language Models (VLMs) have advanced rapidly in static visual understanding, yet remain unreliable when judging how an egocentric task is progressing. Given a task instruction and two visual observations, a model should determine which state is closer to the goal by analyzing task-relevant object configurations and spatial relations, rather than relying on timestamps or presentation order. This distinction is critical in manipulation, where retries, corrective actions, and temporary regressions make progress inherently non-monotonic. We introduce EgoTSR, a unified framework for diagnosing and improving order-robust task-progress understanding. First, SpatialLogic-Bench evaluates each physical state pair in both original and order-swapped presentations across short- and long-horizon settings, exposing whether a model follows task-state evidence or chronological shortcuts. Second, our data construction pipeline converts successful, approximately monotonic manipulation and first-person trajectories into bidirectional supervision; LongTag further preserves intermediate subtask structure for long-horizon comparison, while failure-aware data extend learning to regressions and recoveries. Third, a progressive CoT-to-Tag curriculum first supervises evidence-grounded interpretation of task-relevant state changes and then consolidates the comparison rule through scalable label-only training. Experiments reveal substantial input-order bias in representative VLMs. EgoTSR achieves 92.4% long-horizon accuracy with a 0.1-point forward-inverse Gap. Failure-aware supervision further improves accuracy on non-monotonic trajectories by 11.8 points and Recovery Accuracy by 11.2 points, while maintaining broad visual and spatial capabilities. These results establish goal-conditioned state comparison as an explicit formulation of egocentric spatiotemporal reasoning for task-progress understanding.

1 INTRODUCTION

EgoTSR frames egocentric task progress as goal-conditioned comparison between visual states, with the preferred state determined by task evidence rather than presentation order. It diagnoses order sensitivity, constructs bidirectional supervision, and trains an order-robust comparison rule.

1 INTRODUCTION

  • The task is to identify which of two observations is closer to a goal from task-relevant object configurations and spatial relations.Manipulation is the primary testbed; the formulation assesses state rather than policy execution and also provides an interface for human first-person trajectories.Source: §1 · p. 1
  • 87.5% short-horizon average accuracy, 92.4% long-horizon average accuracy, and a 0.1-point long-horizon Gap are reported for EgoTSR.Failure-aware training raises overall failure-aware accuracy from 64.1% to 75.9% and Recovery Accuracy from 58.6% to 69.8%.Source: §1 · p. 2
  • Failure-aware training extends the comparison rule to failures, regressions, corrections, and recoveries, while external evaluations show transfer to visual and spatial reasoning and first-person interaction.These experiments address non-monotonic executions beyond the core training pool of successful, approximately monotonic trajectories.Source: §1 · p. 2
  • SpatialLogic-Bench presents each physical pair in both orders to test whether models follow state evidence or chronological shortcuts.The benchmark covers short- and long-horizon settings, with the latter comparing progress across omitted intermediate steps.Source: §1 · pp. 1–2
  • EgoTSR-Data creates bidirectional comparison pairs from manipulation and first-person trajectories, while LongTag preserves ordered subtask structure across omitted steps.The pipeline converts robot, human, and egocentric trajectories into supervision.Source: §1 · p. 2
  • The CoT→Tag curriculum first supervises evidence-grounded reasoning about task-relevant state changes, then scales direct comparison training with labels.The intended comparison rule remains stable across input orders.Source: §1 · p. 2

2 RELATED WORK

Related work spans visual and temporal benchmarks, structured reasoning supervision, and robot-learning approaches that connect representations to objectives. EgoTSR focuses on reliable goal-conditioned state comparison, including when execution departs from chronological progress.

2.1 Goal-Conditioned Progress Benchmarks

  • Prior benchmarks cover compositional scenes, 3D environments, video perception, and hallucination or state-change diagnostics; task progress asks a goal-directed comparison question.SpatialLogic-Bench evaluates matched physical pairs in both orders, linking spatial perception, cross-step reasoning, and order robustness.Source: §2.1 · p. 2

2.2 Multimodal Chain-of-Thought Supervision for Reasoning

  • Prior CoT and prompting work motivates structured reasoning; EgoTSR trains a reusable path from task evidence through state-change interpretation to judgment, then scales it with Tag labels.Training-time supervision complements inference-time prompting by organizing visual evidence into a reusable decision path.Source: §2.2 · p. 3

2.3 Task Progress Understanding in Egocentric Execution

  • VIP learns video-based value representations, while SayCan and Eureka ground or generate robot objectives; EgoTSR instead targets reliable state comparison.Temporal order can guide successful demonstrations but does not define progress under failures, regressions, interruptions, and ineffective actions; LongTag retains subtask structure for cross-step comparisons.Source: §2.3 · p. 3

3 PROBLEM FORMULATION

The paper formulates egocentric task-progress understanding as goal-conditioned comparison of two observations from one execution. The comparison is relational, uses successful-trajectory ordering for supervision, and is evaluated under reversed input order.

3 PROBLEM FORMULATION

  • Given a task instruction and two observations from the same execution, the model identifies which state is closer to successful completion.The output is a relational progress judgment rather than action classification or video captioning, and the interface applies to robot-mounted and human first-person video.Source: §3 · p. 3
  • The comparison maps task-relevant spatial changes to a goal-conditioned progress judgment, with manipulation as the primary controlled-evaluation setting.The same interface applies to robot-mounted cameras and human first-person video.Source: §3 · p. 3

3.1 Goal-Conditioned State Comparison

  • For successful, approximately monotonic trajectories, the state with the larger original time index is labeled closer to completion.This defines the preferred physical state independently of its input position.Source: §3.1 · p. 3

3.2 Order-Robust Evaluation

  • The same physical state pair is evaluated in forward and inverse input orders to separate state comparison from input-position heuristics.The physical answer remains unchanged while the input position is swapped.Source: §3.2 · p. 3
  • Forward and inverse accuracy, their average, and the forward–inverse gap measure performance and sensitivity to input order.A model using task-relevant state changes should preserve its physical judgment across the two orders.Source: §3.2 · p. 3
  • Pairwise comparison accommodates different trajectory lengths, subtask decompositions, and recording viewpoints within one evaluation interface.The paper also reports a single-image progress baseline for comparison.Source: §3.2 · p. 3

4 SPATIALLOGIC-BENCH

SpatialLogic-Bench evaluates whether models compare task-relevant physical states relative to a goal, rather than relying on chronology or input position. It spans local and long-horizon comparisons and tests both input orders.

4.1 Benchmark Scope and Diagnostic Goal

  • SpatialLogic-Bench targets whether one observed manipulation state is closer to the task goal than another, a relation distinct from action recognition or generic video understanding.Its diagnostic tests local spatial differences and their mapping to goal-conditioned progress judgments.Source: §4.1 · pp. 3–4

4.2 Multi-Granularity Pair Construction

  • Temporal spans from Δ=5 through Δ≥12 sample local state changes and comparisons across increasingly many omitted intermediate steps.The benchmark uses 10× temporal downsampling before constructing candidate pairs.Source: §4.2 · p. 4
  • Each physical pair is tested in forward and inverse orders, swapping only image positions so chronology or a fixed slot cannot serve as the comparison shortcut.The physical state closer to completion stays the same while its position label changes.Source: §4.2 · p. 4

4.3 Evaluation Protocol and Metrics

  • The protocol reports average accuracy, forward–inverse Gap, and worst-case accuracy, with Gap measuring order sensitivity and Accworst the harder ordering.Failure Avg. aggregates accuracy across valid forward–inverse items; the benchmark contains 28,250 evaluation samples.Source: §4.3 · p. 4

5 EGOTSR-DATA CONSTRUCTION

EgoTSR-Data converts manipulation and first-person trajectories into goal-conditioned state-comparison pairs. Its formats support local and long-horizon reasoning, while bidirectional presentations align training with order-robust evaluation.

5.1 Data Sources and Pair-Level Objective

  • EgoTSR-Data turns video trajectories into task-conditioned pairs labeled by which state is closer to a specified goal.Robot, human egocentric, and broader egocentric datasets contribute controlled transitions, viewpoint variation, and long-tail action–object combinations; trajectory-level splits keep derived pairs within their source split.Source: §5.1 · p. 5

5.2 Dataset Partition and Long-Horizon Structure

  • Short-Horizon Data emphasizes fine-grained spatial changes, while Long-Horizon Data compares states across hidden intermediate steps and task procedure.General Ego Data adds viewpoint variation and long-tailed action–object combinations; a Subtask Planner predicts an ordered skeleton for long-horizon context, with ground-truth metadata confined to training.Source: §5.2 · p. 5

5.3 CoT, Tag, and LongTag Supervision

  • CoT supplies evidence-grounded reasoning, Tag retains the comparison label for direct training, and LongTag adds a subtask skeleton to long-horizon samples.The five combinations are short CoT, short Tag, long CoT, LongTag, and general Tag.Source: §5.3 · p. 5

5.4 Bidirectional Construction and Dataset Scale

  • Bidirectional construction presents the same physical answer in both input slots, changing the position label while preserving the task-goal comparison.CoT references flip with image order, while LongTag retains the subtask skeleton; the construction engine supplies candidate pools and reports split-level counts.Source: §5.4 · pp. 5–6

6 EGOTSR LEARNING FRAMEWORK

EgoTSR uses a shared goal-conditioned state-comparison objective, progressively training models to ground progress judgments in task-relevant evidence and then make direct comparisons. LongTag preserves subtask structure for long-horizon pairs.

6.1 Overview

  • The full framework applies one goal-conditioned comparison objective across data subsets, predicting which state in a task-instruction-conditioned pair is closer to completion.Tag samples provide comparison labels, LongTag adds a subtask skeleton, and CoT samples supervise the evidence path.Source: §6.1 · p. 6

6.2 CoT Supervision for Reasoning Path Construction

  • CoT supervision trains the model to identify task-relevant visual differences, interpret them against the manipulation goal, and produce the corresponding progress preference.Its target reasoning text describes evidence, state-change interpretation, and the resulting judgment; the comparison label is supervised alongside it.Source: §6.2 · p. 6

6.3 Tag Supervision for Direct Comparison

  • Tag supervision transfers the evidence-grounded comparison rule into direct predictions through scalable label-only training.LongTag adds structured task context to preserve long-horizon task logic.Source: §6.3 · p. 6, Figure 4 · p. 7

6.4 Progressive Training Procedure

  • The CoT→Tag procedure first builds explicit reasoning paths from short- and long-horizon CoT data, then uses short Tag, LongTag, and general-ego Tag data for direct prediction.The resulting model is evaluated on order-swapped pairs and on failure, regression, correction, and recovery trajectories.Source: §6.4 · p. 7

7 EXPERIMENTS

SpatialLogic-Bench evaluates task-conditioned state comparisons across short and long temporal windows and in both input orders. EgoTSR achieves the strongest reported accuracy at both temporal levels, while representative VLMs show substantial order sensitivity.

7.1 Experimental Setup

  • The benchmark compares task-conditioned state pairs at short and long horizons, with forward and inverse presentations to test local state discrimination and progress across omitted steps.Its paired protocol holds physical states and task instruction fixed while exchanging input positions; the reported metrics include average accuracy, Gap, and worst-case accuracy.Source: §7.1 · pp. 7–8, §7 · p. 7
  • 87.5% short-horizon average accuracy and 92.4% long-horizon average accuracy make EgoTSR the strongest model at both temporal levels on SpatialLogic-Bench.The short-horizon average covers eight window categories; the long-horizon forward and inverse scores remain high across all three difficulty groups.Source: §7.1 · p. 8, Table 2 · p. 8
  • 91.79% forward accuracy versus 15.81% inverse accuracy gives Qwen3.5 a 75.98-point Gap under direct inference over windows 5–10.Its overall accuracy is 53.80%; Qwen2.5-VL-7B and InternVL-8B show the same qualitative asymmetry.Source: §7.1 · p. 8, Table 2 · p. 8
  • CoT supervision raises short-horizon accuracy to 73.6%, Tag raises it to 86.8%, and LongTag supplies procedural structure for long-horizon comparisons.The full CoT→Tag model with LongTag combines 87.5% short-horizon and 92.4% long-horizon accuracy.Source: §7.1 · p. 8
  • The short- and long-horizon settings use the same comparison interface, while LongTag supplies procedural context for pairs whose intermediate states are omitted.The task is to identify the goal-closer visual configuration and locate it within the larger manipulation procedure.Source: §7.1 · pp. 7–8

7.2 Ablation and Analysis

The ablations show that progressive CoT→Tag training with LongTag combines high long-horizon accuracy with low order sensitivity. They also distinguish the contributions of supervision path, pair-order construction, and training cost.

7.2 Ablation and Analysis

  • 92.4% average accuracy, 92.3% worst-order accuracy, and a 0.1-point Gap are achieved by progressive CoT→Tag training with LongTag.The sequential curriculum exceeds mixed training at the same data budget; adding the subtask skeleton raises average accuracy from 81.1% to 92.4% and reduces Gap from 4.3 to 0.1.Source: §7.2 · p. 9, Table 3 · p. 9, Figure 5 · p. 9
  • On the matched short-horizon diagnostic subset, the full supervision path reaches 87.6% forward accuracy and 85.1% inverse accuracy, for 86.4% average accuracy and a 2.5-point Gap.CoT supplies task evidence, Tag balances presentation orders, and LongTag adds accuracy and stability; this diagnostic subset is distinct from the main benchmark average over eight windows.Source: §7.2 · p. 9, Table 4 · p. 9
  • Forward+inverse training reaches 81.2% average accuracy, 4.4 points above random-order training, and raises worst-order accuracy from 74.6% to 78.2%.Random-order training has the smaller Gap, 4.3 versus 5.9 points; the lowest Gap appears only with progressive supervision and LongTag, while single-order variants have Gaps of 19.4–23.1 points.Source: §7.2 · p. 9, Table 5 · p. 9
  • On the matched short-horizon diagnostic subset, the single-image baseline reaches 70.6% average accuracy at 0.18× relative cost, LoRA reaches 82.1% at 0.35×, and full fine-tuning reaches 86.4%.On a representative long-horizon video, the task-completion curve changes most strongly around grasping, transporting, and placing events.Source: §7.2 · p. 9

7.3 Robustness to Non-Monotonic Trajectories

Failure-aware training improves performance on a trajectory-disjoint split containing failures, regressions, corrections, and recoveries. Additional analyses report results by pair type, data scale, sampling window, and backbone.

7.3 Robustness to Non-Monotonic Trajectories

  • The primary failure-aware evaluation uses fixed Window 10 with 1,345 distinct state pairs; Window 5, Window 20, and adaptive 5+R10 are sensitivity settings.The adaptive 5+R10 policy yields 1,292 pairs and 75.5% overall accuracy.Source: §7.3 · pp. 9–10, Table 9 · p. 10
  • 75.9% failure-aware accuracy, 69.8% Recovery Acc., 23.4% Regression Error, and a 6.7-point Gap are reported after adding annotated failure trajectories.Compared with success-only training, accuracy rises from 64.1%, Recovery Acc. from 58.6%, Regression Error falls from 32.8%, and Gap falls from 13.5 points on the fixed-Window-10 split.Source: §7.3 · p. 10, Table 7 · p. 10
  • The failure-aware model reports 76.8% on Normal-progress, 76.6% on Regression, 73.6% on Failure-internal, and 69.8% Recovery Acc.These pair-type results characterize distinct non-monotonic relations; the overall item-level accuracy is reported separately.Source: §7.3 · p. 10, Table 8 · p. 10
  • Failure Avg. rises from 64.1% to 75.9%, Recovery Acc. from 58.6% to 69.8%, and Gap falls from 13.5 to 6.7 as failure-aware training data increase.The largest gains occur before the 50% data point, which the authors identify as a practical operating range.Source: §7.3 · p. 10, Table 10 · p. 10
  • Every tested backbone improves its failure-aware average and reduces its Gap, with gains ranging from 4.4 to 11.8 points.The recipe therefore transfers beyond the Qwen-7B implementation.Source: §7.3 · p. 10

7.4 What the Diagnostic Reveals

EgoTSR’s long-horizon comparison is accurate and stable under input reversal, while ablations connect that performance to its staged supervision. The evaluation compares identical physical pairs across input orders, isolating presentation order as the source of any Gap.

7.4 What the Diagnostic Reveals

  • 92.4% average accuracy, 92.3% worst-order accuracy, and a 0.1-point Gap show EgoTSR’s strong, order-stable long-horizon comparisons.Random-order training reduces order sensitivity, while the complete CoT→Tag configuration also raises comparison accuracy.Source: §7.4 · pp. 10–11
  • CoT identifies task-relevant state changes, Tag scales direct comparisons, and LongTag preserves intermediate task logic across distant observations.Their sequential combination improves accuracy and order stability together.Source: §7.4 · p. 11
  • Failure-aware training improves Failure Avg. by 11.8 points and Recovery Acc. by 11.2 points, while reducing Regression Error by 9.4 points and Gap by 6.8 points.The resulting model retains 84.1% accuracy on the success test, extending the comparison rule to retries, regressions, and repairs.Source: §7.4 · p. 11, Figure 7 · p. 11

7.5 Evaluation Controls

  • Each inverse query reuses the same images, instruction, and ground-truth preference, changing only the presentation slots.Thus, the Gap compares accuracies for the same state relation rather than different trajectory examples.Source: §7.5 · p. 11

7.6 External Generalization

EgoTSR transfers to visual, spatial, and first-person interaction benchmarks while maintaining a broadly stable capability profile. The reported results also indicate that failure-aware coverage improves recovery-related judgments, and the discussion frames pairwise comparison as a state-assessment primitive rather than a closed-loop controller.

7.6 External Generalization

  • EgoTSR-Tag obtains the strongest reported scores on CV-Bench and MMSI-Bench, while EgoTSR leads on VSI-Bench and the first-person interaction group.These evaluations cover visual and depth cues, object-centric spatial reasoning, and interaction understanding in unseen videos.Source: §7.6 · p. 11, Table 12 · p. 12
  • The capability profile shifts from 71.6 to 71.0 overall while temporal ordering improves by 1.1 points after task-progress training.The results are consistent with targeted adaptation that strengthens temporal and state-comparison behavior while maintaining broader visual capabilities.Source: §7.6 · pp. 11–12, Table 13 · p. 12

7.7 Reading the State-Relation Results

  • Broader failure-aware coverage improves overall and recovery accuracy while reducing Regression Error and input-order Gap across four backbone families.Recovery remains more demanding than regression because it requires relating a repaired configuration back to the task goal.Source: §7.7 · p. 12

7.8 Discussion and Future Work

  • Pairwise preferences can rank candidate checkpoints, form task-completion curves, and flag inconsistent judgments across reversed queries.The comparator does not itself integrate temporal history, confidence, or a controller; these are proposed next steps toward checkpoint selection and execution monitoring.Source: §7.8 · p. 12
  • The measured capability is limited to comparing two observations against a task goal; closed-loop validation and temporal memory remain future work.The benchmark emphasizes robotic manipulation, while training and external evaluation also cover human first-person activities.Source: §7.8 · p. 12

8 CONCLUSION

The paper frames egocentric task-progress understanding as goal-conditioned comparison between visual states, with predictions stable to presentation order. EgoTSR combines a matched-order benchmark, scalable robot and human egocentric supervision, and a progressive CoT→Tag curriculum; its reported results include high long-horizon accuracy and gains on failure-aware evaluation.

8 CONCLUSION

  • SpatialLogic-Bench uses matched forward and order-swapped queries, EgoTSR-Data supplies robot and human egocentric supervision, and CoT→Tag turns evidence-grounded reasoning into a stable inference rule.The formulation isolates goal-conditioned state comparison from chronological evaluation.Source: §8 · p. 12
  • EgoTSR reaches 87.5% short-horizon average accuracy and 92.4% long-horizon average accuracy, with a 0.1-point long-horizon Gap.Failure-aware supervision raises Failure Avg. by 11.8 points and Recovery Acc. by 11.2 points, with transfer to visual, spatial, and first-person interaction capabilities.Source: §8 · p. 12

Related papers

Loading 2609.31751v1…