PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Proactive AI Assistance with Hierarchical Procedural Understanding

Jin-Seop Lee, TaeYeon Won, SeongJun Jung, JungHoon Kim, Boyang Albert Li, JinYeong Bak, Jaehong Yoon, Jee-Hyong Lee

arXiv:2610.06505v2cs.LGcs.CV

TL;DR

Existing proactive-assistance datasets focus on detection or fixed-granularity guidance, limiting support for task progress and user-adjusted guidance. ProactiveCoach provides hierarchical training and evaluation and an adaptive guidance system; hierarchical supervision and routing outperform their respective baselines in the reported comparisons.

  • Problem

    Existing datasets focus either on detection-based proactive understanding or fixed-granularity procedural guidance, limiting support for progress tracking, procedural context, and guidance-level adaptation.

    Source: §1 · p. 2

  • Method

    ProactiveCoach provides phase-, step-, and action-level training and evaluation data, fine-tunes pretrained VLMs, and pairs a multi-level guide model with a lightweight router.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Hierarchical supervision improves overall performance across backbones by up to 9.6%p, and the adaptive system outperforms the in-context baseline by 57.1%p across four transitions.

    Source: §1 · p. 3, §5.3 · p. 10

  • Takeaways & Limitations

    The system supports user-requested guidance-level changes during ongoing tasks without additional fine-tuning of the guide model.

    Source: §1 · p. 3

    Evaluation uses ground-truth dialogue history; further evaluation is needed for model-generated histories, error accumulation, and plan revision.

    Source: I LIMITATIONS · p. 30

Abstract

from arXiv · show

Proactive AI assistants continuously observe a user's activity and decide whether to provide new guidance or remain silent. They should provide appropriate guidance for the task, determine when to provide the next guidance based on task progress, and adjust the guidance level to the user's expertise and needs. Supporting these capabilities requires training and evaluation data that reflect procedural structure and capture how guidance should adapt to task progress and user needs. However, existing datasets either focus on detection-based proactive understanding or provide procedural guidance at a fixed granularity. Fixed-granularity guidance provides limited information about fine-grained progress and broader procedural context, making it difficult to determine completion and adapt guidance granularity. To address these limitations, we introduce the ProactiveCoach suite, comprising ProactiveCoach-Instruct for training, ProactiveCoachBench for evaluation, and fine-tuned VLMs with an adaptive guidance system. ProactiveCoach-Instruct provides hierarchically structured guidance at the phase, step, and action levels for learning task progress and procedural context. ProactiveCoachBench evaluates whether models provide appropriate guidance at the right time across different guidance levels and adapt when the requested level changes. We fine-tune pretrained VLMs on ProactiveCoach-Instruct and demonstrate its effectiveness across backbones. Compared with fixed-granularity supervision, hierarchical supervision improves overall performance across backbones by up to 9.6%p. We further build an adaptive guidance system by combining our fine-tuned model with a lightweight guidance router. Without additional fine-tuning, our system outperforms the in-context adaptation baseline by 57.1%p across four guidance-level transitions. Our project page is available at https://jinsuby.github.io/ProactiveCoach/.

1 INTRODUCTION

Proactive assistance must track task progress, time guidance, and adjust its granularity to user needs. ProactiveCoach addresses limits in existing datasets with hierarchical training and evaluation, and reports gains from hierarchical supervision and adaptive routing.

1 INTRODUCTION

  • Proactiveness combines continuous observation with decisions about what guidance to give, when to give it, and whether to remain silent.Guidance should match task progress and the user’s expertise and needs.Source: §1 · p. 2
  • Existing data either focus on detection-based proactive understanding or provide fixed-granularity guidance, limiting progress tracking, procedural context, and adjustment to changing guidance needs.Fixed granularity gives limited information about what is completed and remains, and about an event’s purpose within the broader procedure.Source: §1 · p. 2
  • ProactiveCoach combines ProactiveCoach-Instruct for training, ProactiveCoachBench for evaluation, and fine-tuned VLMs with an adaptive guidance system.The suite tracks progress and completion at phase, step, and action levels and adapts guidance to the user’s request.Source: §1 · p. 2, Figure 1 · p. 2
  • A lightweight router selects which guidance level to present, allowing granularity to change during a task without additional guide-model fine-tuning.The VLM generates guidance at multiple levels, while the router selects the requested level.Source: §1 · pp. 2–3
  • Hierarchical supervision improves overall performance across backbones by up to 9.6%p, while the adaptive system outperforms the in-context baseline by 57.1%p across four transitions.The dataset also receives an overall human quality-review score of 4.27.Source: §1 · p. 3

2 RELATED WORK

Prior work spans hierarchical video understanding and proactive assistance, but hierarchical descriptions do not directly provide guidance, and proactive datasets leave gaps in procedural-level coverage and adaptation.

2 RELATED WORK

  • Hierarchical video descriptions represent content from actions to activities, but proactive assistance must also determine goal-relevant guidance, its timing, and its level of detail.Prior approaches include step descriptions, cross-video procedure structures, multilevel captions, and hierarchical segmentation.Source: §2 · p. 3
  • Proactive assistance has moved beyond detecting objects or events toward choosing what guidance to provide and when, based on task goals and ongoing context.The cited detection-based and task-context approaches are identified separately in the discussion.Source: §2 · p. 3
  • Existing proactive datasets and benchmarks mainly focus on step-level guidance, with limited support for procedural-level progress and guidance-level changes.Some also address recovery, completion verification, error detection, and corrective guidance.Source: §2 · p. 3

3 PROACTIVECOACH-INSTRUCT & PROACTIVECOACHBENCH

ProactiveCoach-Instruct and ProactiveCoachBench organize guidance hierarchically and time it against task progress. The section describes their design, construction pipeline, and human quality review.

3.1 DATASET OVERVIEW

  • The dataset and benchmark structure goal-relevant guidance at phase, step, and action levels, connecting lower-level progress to event completion and higher-level context to purpose.The hierarchy also supports different guidance granularities according to user needs.Source: §3.1 · p. 4
  • A phase achieves an intermediate goal through steps, and a step achieves an outcome through actions; guidance sentences link to event spans and timing at each level.The next guidance follows an event’s end at that level; chunks with no new guidance are labeled silent.Source: §3.1 · p. 4
  • The dataset draws on 7,232 first-person videos across eight source datasets and covers assembly, repair, electronics, food and beverage preparation, household, and gardening activities.It is divided into 10,007 training samples and 1,112 evaluation samples.Source: §3.1 · pp. 4–5

3.2 CONSTRUCTION PIPELINE

  • Construction proceeds through procedural data curation, hierarchy construction, and proactive guidance generation.Curation selects suitable videos and revises video-grounded captions; hierarchy construction organizes goal-relevant annotations into refined phase–step–action structures; guidance generation assigns level-specific guidance and timing, labeling other frames silent.Source: §3.2 · pp. 5–6

3.3 QUALITY REVIEW

  • Human review of 50 benchmark examples scores annotations above 4.0 on most criteria and 4.27 overall.Review criteria are video alignment, hierarchical coverage, goal relevance, and guidance quality.Source: §3.3 · p. 6

4 PROACTIVECOACH

ProactiveCoach jointly learns phase-, step-, and action-level guidance in one vision-language model, preserving hierarchical relationships and guidance timing in its training targets. Its training also emphasizes response–silence transitions and uses guidance at finer and broader levels to inform what to say and when.

4 PROACTIVECOACH

  • ProactiveCoach trains one vision-language model to produce phase-, step-, and action-level guidance in a single output sequence that preserves their hierarchy and timing.Each guidance sentence is paired with its level and a hierarchical ID encoding parent–child structure.Source: §4 · p. 6
  • Training uses <silent> when no new guidance is needed and <response> with guidance otherwise, combining guidance when multiple levels need a response.The model receives previous ground-truth outputs as history during teacher forcing.Source: §4 · p. 6
  • To limit frequent silence labels from dominating the loss, supervision selects every response chunk, the following silent chunk, and an equal number of additional silent chunks sampled from the remainder.The loss is computed only on selected output tokens and normalized by the number of selected chunks.Source: §4 · p. 6
  • Hierarchical training uses lower-level events to indicate what is completed and remains, while higher-level context clarifies the guided event’s purpose and expected outcome.Together with the observed video, this understanding supports decisions about what guidance to provide and when at each level.Source: §4 · pp. 6–7

5 EXPERIMENTS

The experiments evaluate proactive guidance across benchmarks and guidance levels, then test hierarchical supervision and guidance-level adaptation. Results favor hierarchical supervision, while a lightweight router supports level changes without fine-tuning the guide model.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers ProactiveCoachBench at phase, step, and action levels, plus the step level of EgoProactive’s validation set, against four groups of baselines.The groups are proprietary frontier, multimodal, guidance assistant, and streaming models.Source: §5.1 · p. 7
  • At response anchors models should guide, and at silent anchors they should remain silent; evaluation uses recent video and ground-truth dialogue history.ProactiveCoachBench response anchors use ground-truth guidance timestamps, while silent anchors are sampled within event segments.Source: §5.1 · p. 7
  • The metrics assess guidance alignment and coverage (sF1), response-versus-silence decisions (gF1), and both decisions and guidance quality (PQS).PQS scores silence at silent anchors and evaluates relevance, specificity, actionability, and conciseness at response anchors.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • Across guidance levels, the method outperforms baselines on ProactiveCoachBench and achieves higher Avg. scores than base models on EgoProactive.Table 2 covers both benchmarks; the reported comparisons are consistent across backbones.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 ABLATION STUDIES AND ANALYSIS

  • Hierarchical supervision improves sF1, gF1, PQS, and Avg. over step-only supervision across backbones, including step-level proactive assistance.The comparison jointly trains phase-, step-, and action-level guidance.Source: §5.3 · p. 8, Table 3 · p. 9
  • Adding either phase- or action-level supervision improves step-level performance over step-only supervision, while all three levels achieve the highest performance.Action-level supervision helps represent actions within steps and assess completion; phase-level supervision adds purpose and expected-outcome context.Source: §5.3 · pp. 8–9, Table 4 · p. 9
  • Fine-tuning substantially improves zero-shot Qwen3.5-4B performance across all three guidance levels compared with prompting alone.The comparison uses the same multi-level prompt and three-level ground-truth history.Source: §5.3 · p. 9, Table 5 · p. 9
  • 57.1%p: the adaptive system’s average overall-performance gain over in-context adaptation across four guidance-level transitions.A lightweight router selects which level to display while the guide model generates guidance at three levels, without additional fine-tuning.Source: Table 6 · p. 9, Figure 6 · p. 9, §5.3 · p. 10

6 CONCLUSION

The conclusion presents hierarchical procedural understanding as a direction for proactive AI assistance, with ProactiveCoach datasets and a jointly trained multi-level guidance method. It reports improvements across backbones and guidance-level changes without additional guide-model fine-tuning.

6 CONCLUSION

  • ProactiveCoach-Instruct and ProactiveCoachBench provide phase-, step-, and action-level guidance for training and evaluation.The paper describes them as a useful foundation for future proactive-assistance research.Source: §6 · p. 10
  • The jointly learned multi-level guidance method consistently improves proactive assistance across different backbones.Source: §6 · p. 10
  • The adaptive guidance system supports guidance-level changes without additional fine-tuning of the guide model.Source: §6 · p. 10

Appendices

The appendix reports the dataset’s scale, duration, and hierarchical guidance-target counts.

A MORE DETAILS OF PROACTIVECOACH-INSTRUCT AND PROACTIVECOACHBENCH

  • 11,119 samples come from 7,232 first-person videos across eight source datasets, totaling 1,452.1 hours.The samples split into 10,007 for ProactiveCoach-Instruct and 1,112 for ProactiveCoachBench; samples from the same video stay in one split.Source: A MORE DETAILS OF… · p. 15
  • Each sample averages 4.14 phases, 3.35 steps per phase, and 2.47 actions per step.The dataset contains 582,084 guidance targets: 46,074 phase-, 154,507 step-, and 381,503 action-level targets.Source: A MORE DETAILS OF… · p. 15
  • Sample durations range from 40 seconds to 18 minutes, with a reported average of 7.8 minutes and median of 8.8 minutes.Source: A MORE DETAILS OF… · p. 15, Figure 7 · p. 15

B DATASET CONSTRUCTION DETAILS

The appendix details the dataset-construction pipeline, its prompts and timing rules, and examples of the benchmark’s hierarchical guidance.

B DATASET CONSTRUCTION DETAILS

  • Dataset construction follows procedural data curation, procedure hierarchy construction, and proactive guidance generation, with prompts and a dataset-split table provided.Gemini 2.5 Flash handles video-grounded recaptioning; GPT-4.1-mini handles remaining generation, filtering, and hierarchy refinement, while timing assignment is rule-based.Source: B DATASET CONSTRUCTION DETAILS · p. 15, Table 7 · p. 15

B.1 PROCEDURAL DATA CURATION

B.2 PROCEDURE HIERARCHY CONSTRUCTION

B.3 PROACTIVE GUIDANCE GENERATION

  • Guidance timing is assigned independently at each level after the current guided event ends; simultaneous levels are combined in phase–step–action order, and unassigned frames are labeled silent.The first guidance at each level is assigned to the video’s first frame.Source: §B.3 · p. 18, Figure 13 · p. 18

C EXAMPLES OF PROACTIVECOACHBENCH

D DETAILS OF QUALITY REVIEW

A human review of 50 test examples assessed hierarchical annotations at the phase, step, and action levels using four criteria. Scores indicate strong hierarchical coverage and generally positive ratings for alignment, goal relevance, and guidance quality.

D DETAILS OF QUALITY REVIEW

  • The review interface presented task videos and hierarchical annotations for evaluation on video alignment, hierarchical coverage, goal relevance, and guidance quality.The review covered 50 test examples at all three guidance levels.Source: D DETAILS OF QUALITY REVIEW · p. 22, Figure 20 · p. 22
  • Video-alignment scores were 4.05, 4.14, and 3.80 at the phase, step, and action levels, respectively.Reviewers considered content, temporal spans, and guidance timing; excessively early or late guidance and unnecessary repetition scored lower.Source: D DETAILS OF QUALITY REVIEW · p. 22
  • Coverage scores were 4.35, 4.50, and 4.48 at the phase, step, and action levels, respectively.Reviewers checked whether phases covered goal-related activities, steps covered phases, and actions covered steps without omitting essential execution.Source: D DETAILS OF QUALITY REVIEW · p. 22
  • Goal-relevance scores were 4.48, 4.41, and 4.39 at the phase, step, and action levels, respectively.Necessary preparatory, safety-related, and recovery events counted as relevant; unrelated events received lower ratings.Source: D DETAILS OF QUALITY REVIEW · p. 22
  • Guidance-quality scores were 4.37, 4.16, and 4.16 at the phase, step, and action levels, respectively.Reviewers assessed clarity, accuracy, and whether each sentence matched its intended level of detail.Source: D DETAILS OF QUALITY REVIEW · p. 22

E.1 TRAINING CONFIGURATIONS

The appendix specifies training context, evaluation prompts, and metrics for semantic guidance quality, response–silence decisions, and overall proactive-assistance quality.

E.1 TRAINING CONFIGURATIONS

  • Training used time-ordered video chunks and outputs with sliding-window causal attention over the latest five chunks, or 10 seconds, while retaining task instructions and prior output text within the context limit.The original task instructions were kept.Source: §E.1 · p. 23, Table 8 · p. 23

E.2 EVALUATION PROMPTS

  • All baselines shared the task goal, latest 10 seconds of video, ground-truth dialogue history, and requested guidance level, with prompt variants for frontier, streaming, proactive, and proposed models.The shared-input and prompt examples are shown in Figures 21–24.Source: §E.2 · p. 23, Figure 21 · p. 23, Figure 22 · p. 24

F DETAILS OF METRICS

  • sF1 measures semantic alignment and ground-truth guidance coverage, gF1 measures response–silence decisions, and PQS combines decision correctness with generated-guidance quality.The metrics are computed independently for each dataset and guidance level.Source: F DETAILS OF METRICS · pp. 24–25

F.1 SOFT F1 (SF1)

  • sF1 matches predicted and ground-truth guidance using semantic and temporal information, then computes the harmonic mean of semantic precision and recall.Silent anchors have no ground-truth guidance; their semantic term is set to 1 when matching a predicted response, and only response–response pairs enter the final matching set.Source: §F.1 · p. 25

F.2 G-MEAN F1 (GF1)

  • gF1 is the geometric mean of response F1 and silent F1 at anchor points, and does not assess response content or use temporal matching.An inaccurate response at a ground-truth response anchor still counts as a true positive.Source: §F.2 · p. 25

F.3 PROACTIVE ASSISTANCE QUALITY SCORE (PQS)

  • PQS scores correct silence as 1, true-positive responses by their guidance-quality score, and false positives or negatives as 0, then averages across anchors.The quality score is based on GPT-5.2 ratings of relevance, specificity, actionability, and conciseness; because always-silent prediction yields PQS 50 on this benchmark, the authors interpret PQS alongside gF1.Source: §F.3 · p. 25

G ADDITIONAL QUANTITATIVE RESULTS

Additional results report metric-level patterns and qualitative examples, while the limitations identify evaluation settings and task variations that remain to be addressed.

G ADDITIONAL QUANTITATIVE RESULTS

H QUALITATIVE RESULTS

  • Across six examples from ProactiveCoachBench and EgoProactive, the model more consistently guides at response anchors, stays silent at silent anchors, and aligns responses with ground truth.The examples cover food preparation, bicycle repair, fence maintenance, antigen testing, fan cleaning, and dog harnessing.Source: H QUALITATIVE RESULTS · p. 26, Figure 25 · p. 27, Figure 26 · p. 27

I LIMITATIONS

  • Evaluation uses ground-truth rather than model-generated dialogue history, and further work is needed on error accumulation, automatic expertise adaptation, alternative valid execution orders, long-term tracking, and plan revision.The authors also identify personalized guidance as a future direction.Source: I LIMITATIONS · p. 30

Related papers

Loading 2610.06505v2…