Source-linked AI summary
Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision
Logesh Kumar Umapathi
TL;DR
The paper addresses how a wearable assistant should decide whether to intervene after each egocentric-video segment when labelled data are limited. It separates intervention timing from utterance generation with single-token classification and adds visually grounded agentic supervision. The resulting submissions ranked first in the large division and second in the ≤2B division, while the paper reports stronger transfer from grounded than narration-derived supervision.
Problem
The task is to decide after each approximately 8 s egocentric-video chunk whether a wearable assistant should interrupt with assistance or remain silent, under limited labelled data.
Method
The approach uses single-token classification for intervention timing and an agentic pipeline that generates visually grounded supervision from video.
Results
Single-token classification improved macro-F1 by +0.249 and G-mean by +0.30 over direct generation, while the final entries ranked first in the large division and second in the ≤2B division.
Takeaways & Limitations
Visually grounded supervision transferred better than larger narration-derived supervision, and released in-domain performance was not a reliable proxy for cross-domain transfer.
Takeaways & Limitations
The released in-domain set was insufficient for reliable model selection because it could favor fit to released data over cross-domain transfer.
Abstract
from arXiv · showhide
We present our submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, which ranked first in the large-model division and second in the <=2B division. The task requires a wearable assistant to decide after each eight-second segment of egocentric video whether to intervene or remain silent. Our approach has two main components. First, we reformulate intervention timing as single-token classification. Rather than generating either $interrupt$<utterance> or $silent$, the model predicts yes or no, and we derive the decision from the renormalised probabilities of these two tokens. This formulation improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Second, because labelled data were limited to the released validation set, we generated additional supervision using a tool-calling video agent that inspects each clip and assigns intervention timestamps. A narration-only alternative was four times larger and ten times cheaper, but transferred worse than supervision from an unrelated real corpus, suggesting that visual grounding is more important than annotation volume for this task.
1 Introduction
The paper treats proactive assistance as a timing decision after each egocentric-video segment, separating intervention choice from utterance generation. It combines single-token classification with visually grounded agentic supervision under limited labelled data.
- Task formulation: The EgoProactive track asks a wearable assistant to choose after each approximately 8 s video chunk between interrupting and remaining silent.Submissions are scored with macro-F1 over the two decisions.
- Task formulation: Balancing intervention and silence is central because the task penalizes both intervening too often and missing needed assistance.The geometric mean of per-class F1 scores is used for internal model selection because it rejects policies collapsing to either extreme.
- Data regime: The released data comprise 700 validation videos spanning 135 procedural tasks and 1,947 scored decisions, with 54.3% requiring intervention.The test set remained withheld.
- Contributions: A single-token verbalizer decouples the binary intervention decision from utterance generation, improving macro-F1 by +0.249 and G-mean by +0.30 over direct generation.The formulation also provides a tunable operating point.
- Contributions: An agentic annotation pipeline supplies visually grounded proactive supervision and compares grounding against narration-derived labels.The paper also reports a release-set-only measurement protocol and a lossless vocabulary prune from 2.2132B to 1.9977B parameters with 544/544 identical chunk predictions.
2 Method
The method separates the intervention decision from utterance generation and augments limited labels with visually grounded agent supervision. It evaluates annotation policies and training safeguards designed to improve transfer while meeting the ≤2B constraint.
- The verbalizer reformulation: The generative baseline entangles whether to intervene with what to say, diluting the binary learning signal and producing interrupt precision 1.000 at recall 0.138.Its macro-F1 was 0.4517 and G-mean was 0.4004.
- The verbalizer reformulation: The model instead emits one yes/no token, using the renormalised probability over those logits to decide whether to speak.The utterance is discarded during training and templated at inference because the metric scores timing, not content.
- Input construction: The per-chunk loop accumulates frames to at most 32, thresholds the yes/no probability at τ, and feeds each decision into subsequent history.This history prevents repeated interventions, while the threshold remains tunable after training and requires no decoding loop.
- Agent-generated supervision: The agentic annotation route uses timeline reasoning, tool calls, and vision confirmation to produce timestamped intervention and explicit silence events binned into 8 s chunks.The pipeline applies a −0.5 s onset shift when mapping events to chunks.
- Agent-generated supervision: The narration route was cheaper and larger, but its weak visual grounding underperformed an unrelated real corpus for transfer.The comparison used 953 narration clips versus 234 agent clips, while the agent labels were tied to evidence visible to the student model.
- The annotation policy, and how it was tuned: The annotation policy was tuned on disjoint videos to cover setup actions, place cues at onset, merge repeated motions, and emit at most one cue per 8 s interval.The final policy avoided prescribing a target intervention density, while the dense policy matched the validation interrupt frequency within 1.3 points without being given it.
3 Measurement
The final submission reduced the labelled data available for unbiased evaluation, so full released-set performance was no longer a reliable generalisation signal. Development therefore used a fixed held-out split and ranked candidates on an excluded cross-domain benchmark.
- Training on part of the released validation set reduced the labelled data available for unbiased evaluation.
- Candidate models were evaluated on 210 development videos, 490 held-out released videos, and 140 excluded HoloAssist videos.The released-video split was stratified by domain and interrupt-rate tercile, while prompt-tuning videos stayed in development.
- Candidates were ranked by G-mean on the cross-domain benchmark rather than by the released in-domain set alone.
4 Results
The submission achieved the top large-division result and second place in the ≤2B division on the hidden-test macro-F1 leaderboard. The large-division comparison also showed that substantially larger models did not necessarily score better, while the selected recipe benefited from cross-domain supervision.
- 0.7179 macro-F1 won the large division with the 4.54B entry, while 0.6866 placed the 1.9977B entry second in the ≤2B division.The smaller entry finished 0.0045 behind the division winner; both values are hidden-test macro-F1.
- The official leaderboard reports macro-F1 on the held-out test set, which is not directly comparable with the G-mean figures used for model selection.
- A 4.54B model outscored 27B and 28.9B entries, whose large-division scores were 0.7055 and 0.5452.Across independent teams, the reported sixfold parameter advantage did not translate into a better score.
- Both submitted entries used the same recipe at two scales and were selected using the cross-domain benchmark alone.
- The added corpus lowered in-domain score by 0.042 but raised cross-domain score by 0.021, with interrupt recall increasing by 0.064 F1.
5 Negative results
Several interventions failed to improve transfer: changing the synthetic-data mixture and increasing backbone size produced little or negative benefit under the reported training setup. These results point to supervision and data composition as more consequential than simply adding diversity or parameters.
- Replacing 6.6k of 13.7k procedural examples reduced cross-domain G-mean from 0.590 to 0.548 at fixed compute and dataset size.The replacement used agent-annotated household and sightseeing footage.
- Synthetic data from Ego4D, HoloAssist, and Ego-Exo4D, along with temporal augmentation, produced no consistent improvement.
- Increasing source diversity at the expense of procedural examples was detrimental in this setting, and increasing backbone size provided little benefit.
- The 27B model scored 0.002 below the 4B model, while the 2B model remained within 0.01 of it under the same verbalizer recipe.
6 Conclusion
The results are driven by separating intervention timing from utterance generation and by using visually grounded supervision. However, released in-domain performance was not a reliable proxy for cross-domain transfer under limited labelled data.
- Single-token classification improved G-mean by approximately 0.30 over free-form generation by separating intervention decisions from response generation.This simplified the binary timing decision.
- Visually grounded agent supervision transferred better than a larger, cheaper narration-derived corpus.Narration labels often relied on information unavailable in the student’s visual input.
- Released in-domain performance was not a reliable proxy for cross-domain transfer, with some models reaching only 0.49 cross-domain.Adding the agent-generated corpus reduced in-domain G-mean by 0.042 while improving cross-domain performance, motivating separate cross-domain model selection.