PaperlayerView exact revision on arXiv

Source-linked AI summary

ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video

Ruotong Liao, Meikel Kokowski, Xiaoxiao Sun, Yongqin Xian, Laura Bravo Sánchez, Shiye Su, Federico Tombari, Serena Yeung-Levy, Volker Tresp

arXiv:2610.30089v1cs.CV

TL;DR

Wearable assistants must remember useful details as surroundings evolve, but doing so requires proactive acquisition rather than only interpreting prompted visual evidence. ProactEgoMem benchmarks this capability across ambient-memory tasks and notification or intervention protocols; current models still struggle to act both timely and accurately.

  • Problem

    Wearable assistants need to maintain useful memory as surroundings change, including transient details and evolving states or locations, while users focus on other activities.

    Source: §1 · pp. 1–2

  • Method

    ProactEgoMem evaluates streaming egocentric memory through three ambient-memory tasks and Notification and Intervention levels, across daily and scientific tracks.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    No model dominates proactive memory acquisition: the best Notification Level F1 is 19.0% on ST, the best Intervention Level overlap is 17.2% mIoU on ST, and the strongest strict-timing score is 28.8% Start@0.2 on LT.

    Source: §5.3 · p. 7

  • Takeaways & Limitations

    The benchmark provides an evaluation foundation for proactive multimodal assistance and video agents that co-exist with their users.

    Source: §6 · p. 9

Abstract

from arXiv · show

Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.

1 INTRODUCTION

Wearable assistants could reduce the burden of remembering by proactively maintaining useful memory as surroundings change. ProactEgoMem benchmarks this capability through ambient-memory tasks and two levels of proactiveness.

1 INTRODUCTION

  • ProactEgoMem benchmarks proactive memory in streaming egocentric video through Information Capture, State Tracking, and Location Tracking.The tasks preserve transient visual information, follow changing object states, and record object movements.Source: §1 · p. 2
  • The benchmark separates a timely, intent-matched notification from intervention that maintains memory tools on the user’s behalf.Intervention requires background tracking and memory-tool operation.Source: §1 · p. 2
  • ProactEgoMem-Daily contains 300 clips and 1000 questions, while ProactEgoMem-Sci contains 100 clips and 500 questions.The tracks cover everyday recordings and wet-lab co-scientist scenarios, respectively.Source: §1 · p. 2
  • The benchmark evaluates whether agents decide when to intervene, what to remember, and how to maintain background processing, while scoring timing and semantics.It benchmarks proprietary, open-source, video-language, and streaming models and identifies missed calibrations, delayed initiation, false alarms, lifecycle-maintenance issues, and invalid tool calls.Source: §1 · p. 2

2 RELATED WORK

ProactEgoMem differs from prior video and proactivity benchmarks by evaluating autonomous memory acquisition and maintenance rather than query-driven responses or a single response decision.

Streaming and egocentric video understanding

  • Prior egocentric and video QA benchmarks are query-driven, whereas ProactEgoMem lets the agent decide when to act and sustain memory through a start–maintain–stop lifecycle.The related work covers long-horizon and long-context video QA, streaming QA, and egocentric episodic memory.Source: §2 · p. 3

Video-language models

  • Existing video-language and streaming systems generally produce whole-clip answers or reduce proactivity to a single respond-or-not event.The cited systems include online respond-versus-wait, real-time memory, and when-to-speak approaches.Source: §2 · p. 3

Proactive assistance and proactivity benchmarks

  • ProactEgoMem is presented as the first benchmark in which an agent decides when to act, sustains memory, and is evaluated on later question answering.Earlier proactivity benchmarks mainly score whether to respond, with refinements for user profiles, out-of-plan recovery, or one-second grading.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation treats proactive memory as online initiative in egocentric streams: agents can notify users or operate background memory tools, with tasks spanning transient information, object states, and locations.

3.1 STREAMING SETTING

  • At each time step, the agent sees the egocentric stream only up to the present and chooses tool operations, a user message, or no action.The episode is timestamped and frames arrive oldest to newest.Source: §3.1 · p. 3
  • Tool calling starts or stops background recording, state-tracking, or location-tracking tools, and multiple tool instances may run in parallel.The agent outputs a JSON object containing a possibly empty list of tool calls.Source: §3.1 · p. 3
  • Proactive messages are grounded in observations and acquired memory, and the agent stays silent when no action is required.Messages may be notifications or interventions.Source: §3.1 · p. 3

3.2 TWO PROACTIVE MODES

  • At the Notification Level, the agent infers intent and sends a timely message in response to a visual cue.The ground truth is a timestamped opportunity paired with a semantic target message.Source: §3.2 · p. 4
  • At the Intervention Level, the agent maintains a memory-tool lifecycle for the inferred task and reports its resolution.For example, it can track objects while they remain on, stop trackers when they turn off, and report completion.Source: §3.2 · p. 4

3.3 THREE AMBIENT-MEMORY TASKS

  • Across 4,000 hours of video, the benchmark defines three tasks for proactively preserving transient information, object states, and location movements.The task families are Information Capture, State Tracking, and Location Tracking.Source: §3.3 · p. 4
  • Information Capture preserves dense visual details for later recall, while State Tracking follows object-state changes during multitasking.Questions test captured content or state counts, active states, activation order, and duration.Source: §3.3 · p. 4
  • Location Tracking records an object’s original and subsequent locations, including when it is picked up, placed, or last observed elsewhere.The composition figure also presents scenario, track, and task distributions and target-object frequencies.Source: §3.3 · p. 4, Figure 4 · p. 5

4 PROACTEGOMEM: BENCHMARK CONSTRUCTION

ProactEgoMem builds daily and scientific tracks from clips where ambient memory could help, using event annotation and reviewed questions to evaluate later recall.

Dataset source and annotation

  • Daily clips are drawn from Ego4D, while the Science track combines wet-lab egocentric recordings and Ego4D lab videos for a co-scientist setting.Annotations on raw streams and evaluation QA pairs are newly created to avoid label leakage from pre-training corpora.Source: §4 · p. 5
  • The construction pipeline pairs native action labels into candidate events, densely captions Daily clips with a VLM, and has annotators correct events and task families before QA generation.Ego4D lab clips are annotated from scratch, and event review involves at least three annotators.Source: §4 · p. 5

ProactEgoMem-QA

  • ProactEgoMem-QA pairs each annotated event with multiple-choice questions testing whether acquired memory supports later recall.IC questions are LLM-generated from dense human annotations and reviewed in three rounds; ST and LT questions are chiefly instantiated from templates.Source: §4 · p. 5
  • Every QA question is reviewed by at least three annotators with education above bachelor’s degree level.Table 3 reports data statistics and inter-annotator agreement, with paired entries ordered Daily/Sci.Source: Table 3 · p. 6

5 EXPERIMENTS

The experiments evaluate proactive memory across tasks, model families, proactive levels, and daily and laboratory settings. Results show that models struggle to act both accurately and on time, and that memory acquisition remains a major challenge.

5.1 METRICS AND BASELINES

  • The evaluation scores timing, detection, and semantic matching at both levels, plus lifecycle overlap and downstream QA accuracy for interventions.The baselines span proprietary, open-source, video-language, and streaming models.Source: §5.1 · p. 6

5.2 STREAMING EVALUATION PROTOCOL

  • Each baseline receives one request per video second with recent frames and prior tool-call or notification context; unparseable output counts as silence.The stream is sampled at 1 fps, with up to 10 recent frames per turn.Source: §5.2 · p. 6

5.3 MAIN RESULTS

  • Best Notification Level F1 reaches only 19.0% (Qwen3.6-27B on ST), best Intervention Level overlap 17.2% mIoU (Gemma4-31B on ST), and strict timing 28.8% Start@0.2 (VideoLLaMA3-7B on LT).No model dominates, and the Intervention Level is uniformly harder than the Notification Level.Source: §5.3 · p. 7

5.5 FROM DAILY LIFE TO THE LABORATORY

  • In the laboratory, IC acquisition gaps nearly double to 45–52 points, while open-source ST notification recall falls from 35.3% to 8.9%.IC and ST are harder in the lab, whereas LT is easier because lab objects rarely move; LT gaps narrow to 22–27 points.Source: Figure 7 · p. 8, §5.5 · p. 8

5.3 MAIN RESULTS

  • Open-source models generally cover more metrics, while streaming models have the smallest Intervention-level start error (1.3 s versus 2.5–3.1 s).VLMs outperform native-trained video models on metric coverage, while prompted video-language models have the worst recall.Source: §5.3 · pp. 7–8

5.4 EFFECTS OF PROACTIVE MEMORY ACQUISITION

  • Proactive memory helps QA on IC, but oracle memory remains at least 20 accuracy points above proactive memory in 34 of 36 valid model–task cells.Oracle accuracy reaches 94.9% on IC, while ST and LT remain below 70%; even oracle memory does not make the benchmark easy to solve.Source: §5.4 · p. 8, Figure 6 · p. 8

5.6 FAILURE MODES

  • Recalled interventions are explained correctly 87.8% of the time on average, but 67–93% of Intervention-level starts are late.Intervention start-error medians range from +1.0 to +3.5 s across model families.Source: §5.6 · p. 9
  • Failure modes include false alarms, incomplete tool lifecycles, and video-language models that leave the tool protocol.Notification recall reaches 79.5% at single-digit precision; intervention recall peaks at 37.9%, and VideoLLaMA3-7B never closes a target track.Source: §5.6 · p. 9

6 CONCLUSION

ProactEgoMem evaluates proactive ambient memory in streaming egocentric video across two proactive modes, three task families, and daily-life and laboratory tracks. The experiments find that current multimodal models remain far from reliable proactive assistants.

6 CONCLUSION

  • Current multimodal models struggle to identify the right moment under strict timing, maintain event tracking, and acquire useful memory proactively.The benchmark evaluates IC, ST, and LT across daily-life and laboratory co-scientist tracks.Source: §6 · p. 9
  • ProactEgoMem provides an evaluation foundation for proactive multimodal assistance and video agents that coexist with users.Source: §6 · p. 9
  • The paper frames reliable wearable agents as needing to remember useful context without being asked.Source: §6 · p. 9

AI USE STATEMENT

The authors used generative AI tools in parts of the benchmark pipeline and manuscript preparation, reviewed the AI-assisted work, and take responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools produced dense captions, generated IC questions, judged semantic similarity and message correctness, and assisted with manuscript text, layout, and figure scripts.Source: AI USE STATEMENT · p. 10
  • The authors state they used no generative AI tools for other tasks requiring disclosure.Source: AI USE STATEMENT · p. 10
  • Three independent annotators marked event boundaries, generated questions received three rounds of review, and authors checked AI-assisted text and figure code.The authors state they take responsibility for the final content, including AI-assisted material.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY

The benchmark uses blind score-only submissions and combines query-free streaming proactivity, a tool lifecycle, and downstream questions to test acquired memory. Its task formulations, QA pairs, annotations, and minimal tool interface are designed for this evaluation.

REPRODUCIBILITY

  • An evaluation server keeps ground-truth answers private and returns only scores to blind submissions.Source: REPRODUCIBILITY · p. 10

A COMPARISON WITH EXISTING BENCHMARKS

  • Unlike benchmarks testing offline QA, timestamp-prompted streaming, supplied-query recall, or a single respond-now decision, ProactEgoMem combines query-free streaming with a start–maintain–stop lifecycle and downstream QA.Source: A COMPARISON WITH EXISTING BENCHMARKS · p. 14

B DATA LEAKAGE MITIGATION AND ANNOTATION NOVELTY

  • The benchmark adds proactive task boundaries and actions absent from source datasets, including real-time triggers and longitudinal IC, ST, and LT tracking.The paper presents these formulations as structural safeguards against solving the benchmark through frame memorization or textual pretraining priors.Source: B DATA LEAKAGE MITIGATION AND… · p. 14
  • It uses newly generated QA pairs and fresh human grounding, with three annotators per clip and at least three annotator reviews for each finalized QA.IC questions are generated from dense captions; ST and LT questions use human-verified timelines and templates.Source: B DATA LEAKAGE MITIGATION AND… · p. 14, Figure 11 · p. 15

C TOOL INTERFACE

  • The minimal tool interface provides a Recorder for visible information, a State Tracker for unresolved object states, and a Location Tracker until an object moves.Tracker start calls return IDs that stop calls close.Source: C TOOL INTERFACE · p. 15

D QUALITY OF THE PROACTIVE MESSAGE

Models usually explain a recalled intervention correctly, but the quality varies by model; successful detection, rather than message wording, is the main bottleneck.

D QUALITY OF THE PROACTIVE MESSAGE

  • Among models with Intervention Level recall above 5%, a text-only judge scores whether each closing message names the correct information, conditioned on the model’s own true positives.The reference message is templated from the ground-truth object, tool, and status.Source: D QUALITY OF THE PROACTIVE MESSAGE · p. 15
  • Conditional message accuracy is nearly independent of recall (Spearman r = 0.23), so models that fire rarely are not thereby more careful about what they say.Figure 12’s best group mean remains below 6% on intervention IoU@τ.Source: D QUALITY OF THE PROACTIVE MESSAGE · p. 16, Figure 12 · p. 16
  • 87.8% of recalled interventions are explained correctly on average, but conditional message accuracy ranges from 58% (VideoLLaMA3-7B on LT) to 97% (Inkling-small on ST).VideoLLaMA3-7B recalls 19.8% of LT lifecycles but explains barely half correctly, yielding an 11.5% joint rate, below Qwen3.6-27B’s 12.5%.Source: D QUALITY OF THE PROACTIVE MESSAGE · p. 16

E EFFECTS OF TIMING IN PROACTIVITY

Timing is typically late and difficult to calibrate, while laboratory transfer most consistently reduces opportunity coverage. The paired results show lower notification recall in Science across nearly all model–task cells.

E EFFECTS OF TIMING IN PROACTIVITY

  • Start@1.0 reaches 50–80% for most models at Notification Level and 30–60% at Intervention Level, but Start@0.2 falls to 10–30% and 5–20%, respectively.The timing curves are steeper at Intervention Level, which is uniformly harder than Notification Level.Source: E EFFECTS OF TIMING IN PROACTIVITY · p. 16
  • Near 13–16%: group-mean Start@0.2 means fewer than one matched notification in six lands within 200 ms, and all 11 valid pooled intervention distributions have positive median start error.Mean absolute error across emitting model–task cells is 2.07 s, with a range of 0.73–8.5 s.Source: E EFFECTS OF TIMING IN PROACTIVITY · p. 17, Figure 13 · p. 17
  • Mean absolute start error is 2.07 s at the Notification Level and 3.02 s at the Intervention Level; 22 of 28 paired cells become later after tool commitment.Six paired cells improve, so the added delay is usual but not universal.Source: E EFFECTS OF TIMING IN PROACTIVITY · p. 17, Figure 13 · p. 17
  • Mean start-error spread remains near 2 s at both levels (σ = 2.01 s and 2.21 s), while the distributions vary in width and location across models.A model-level mean shift would not reliably bring starts within 200 ms; the passage identifies a reliable boundary signal as missing.Source: E EFFECTS OF TIMING IN PROACTIVITY · p. 17, Figure 13 · p. 17

F PROACTEGOMEM-SCI: FROM DAILY LIFE TO THE LABORATORY

  • 28.5% to 15.5%: mean Notification Level recall falls from ProactEgoMem-Daily to ProactEgoMem-Sci, decreasing in 28 of 30 paired model–task cells and increasing in none.Notification F1 falls from 5.9% to 3.9%, while intervention, conditional-timing, and QA changes have mixed directions.Source: F PROACTEGOMEM-SCI: FROM DAILY LIFE TO… · p. 18, Figure 14 · p. 18
  • Notification recall decreases by 13.3 points on IC, 14.8 on ST, and 10.8 on LT; cross-track recall rank correlations remain high for IC (ρ = 0.930), ST (ρ = 0.828), and LT (ρ = 0.900).The largest loss is on ST, identified as the most acute transfer bottleneck for recognizing domain-specific state transitions.Source: F PROACTEGOMEM-SCI: FROM DAILY LIFE TO… · pp. 18–19
  • From Daily to Science, best notification F1 falls from 19.0% to 10.8%, intervention mIoU from 17.2% to 10.2%, and memory-backed QA from 66.5% to 57.2%.The higher best intervention recall and F1 come from different single-model LT spikes.Source: Figure 15 · p. 19, F PROACTEGOMEM-SCI: FROM DAILY LIFE TO… · p. 19
  • Mean oracle-to-acquired QA gap widens from 29.2 to 34.7 points, as Science has higher oracle mean accuracy (67.4%) but lower proactively acquired accuracy.The passage contrasts this acquisition limitation with Daily’s comprehension limitation.Source: Figure 14 · p. 18, F PROACTEGOMEM-SCI: FROM DAILY LIFE TO… · p. 19

G RESULTS AT THE TWO PROACTIVE LEVELS

Notification models often trade precision for coverage, while full tool lifecycles remain harder to sustain. Representative traces show failures to close tools, stopping inactive tools, over-triggering, and narrating instead of calling tools.

G RESULTS AT THE TWO PROACTIVE LEVELS

  • Notification Level recall reaches 79.5%, but precision is almost uniformly single-digit and the best notification F1 is 19.0% (Qwen3.6-27B on ST).Start@0.2 never exceeds 28.8%; strict-timing leaders have precision below 2%.Source: G RESULTS AT THE TWO PROACTIVE LEVELS · p. 20
  • Nine model–task scores contain no valid intervention, and the best Intervention Level Start@0.2 is 20.0%.Intervention recall falls from as high as 79.5% at Notification Level to at most 37.9%, while precision, recall, and lifecycle overlap collapse together.Source: G RESULTS AT THE TWO PROACTIVE LEVELS · p. 20
  • Intervention requires a tool call to remain open as the scene changes, with parallel perception and a decision about when to stop.The notification is a single shot; the two-level performance gap measures this background co-existence demand.Source: G RESULTS AT THE TWO PROACTIVE LEVELS · p. 20

H QUALITATIVE FAILURE CASES

  • VideoLLaMA3-7B opens tracks without closing them and also stops tools that are no longer live.Figures 16–17 present these as lifecycle-maintenance failure cases.Source: Figure 16 · p. 20, Figure 17 · p. 21
  • Gemma4-31B over-triggers location tracking, while VideoLLM-Online narrates and MMDuet describes its decision instead of calling the tool.These traces illustrate calibration and tool-following failures.Source: Figure 18 · p. 21, Figure 19 · p. 22

I RESULTS ACROSS AMBIENT-MEMORY TASKS

Task difficulty rises with the memory horizon: Information Capture is easiest, while Location Tracking is hardest. The task demands distinguish dense visual reading from concurrent state updates and long-horizon object movement.

I RESULTS ACROSS AMBIENT-MEMORY TASKS

  • Memory-backed QA accuracy reaches 36–66% on IC, above the 18–34% observed on ST and LT.The reported difficulty ordering is IC < ST < LT.Source: I RESULTS ACROSS AMBIENT-MEMORY TASKS · p. 22
  • IC benefits from dense-text reading strengths, whereas ST requires concurrent state lifecycles and LT requires updating object locations over the longest horizon.ST involves tracking multiple states and their order; LT requires retaining earlier locations while recording each move.Source: I RESULTS ACROSS AMBIENT-MEMORY TASKS · p. 22
  • ST and LT demand distinct temporal memory: concurrent state changes for ST, and persistent location updates across moves for LT.The passage characterizes LT as the hardest task and IC as the easiest.Source: I RESULTS ACROSS AMBIENT-MEMORY TASKS · p. 22

J PROACTEGOMEM-SCI: FULL MAIN RESULTS

This section presents ProactEgoMem-Sci results and evaluation procedures, covering proactive notification and tracking performance, memory comparisons with uniform sampling, and prediction matching across Intervention and Notification levels using an LLM judge.

J PROACTEGOMEM-SCI: FULL MAIN RESULTS

  • Science-track bests are 10.8% notification F1 (Gemma4-31B, LT) and 10.2% intervention mIoU (Gemma4-31B, ST), with 7/30 notification and 10/30 intervention cells emitting nothing.Overall means are 3.9% notification F1 and 1.8% intervention mIoU; no family exceeds single digits on any joint timing-and-semantics metric.Source: J PROACTEGOMEM-SCI: FULL MAIN RESULTS · pp. 22, 24, Table 6 · p. 23

J.1 FAMILY TRADEOFFS ON THE SCIENCE TRACK

  • Family leadership is fragmented: open-source models lead memory-backed QA and intervention, proprietary models lead notification recall and strict timing, and video-language models lead timing accuracy.The strongest family reaches only 6.8% notification F1, 2.9% intervention mIoU, and 6.4% intervention F1; notification precision is at or below 9.8% in 29 of 30 cells, while recall reaches 53.1%.Source: §J.1 · pp. 24–25, J PROACTEGOMEM-SCI: FULL MAIN RESULTS · p. 24

J.2 EFFECTS OF PROACTIVE MEMORY ON THE SCIENCE TRACK

  • Proactive acquisition is the dominant bottleneck: Oracle Memory exceeds Proact Memory in all 36 model–task cells, by mean gaps of 50.0 points on IC, 33.1 on ST, and 22.6 on LT.Compared with uniform-sampling No Memory, Proact Memory loses in 22 of 36 cells, gains in 12, and ties in 2; 11 of 12 models lose on ST, and stronger-baseline models lose on LT while weaker ones gain or hold.Source: §J.2 · p. 25

K PROMPTS

  • All baselines receive the same task-and-level prompt, wrapped in each model’s chat template, and respond at 1-fps steps using JSON context and video frames.Tool calls execute at the current step without timestamps.Source: K PROMPTS · p. 26

K.1 INTERVENTION LEVEL

  • Intervention prompts provide active-tool state and completed intervals; models start or stop object trackers using instance IDs, with successful tracking remaining active until stopped.The prompt policy calls for tracking relevant personal objects and acting only when useful support is timely.Source: §K.1 · p. 26

K.2 NOTIFICATION LEVEL

  • Notification prompts use previous messages rather than tool state and ask the model to notify about relevant personal objects and locations or do nothing.The policy explicitly cautions against notifying about every object or location unless relevant to user intent.Source: §K.2 · pp. 27–28

K.3 MEMORY QA

  • Memory QA compares no memory, the assistant’s tool-selected Proact Memory, and ground-truth Oracle Memory, using task-specific text or recording evidence and frames.For IC, memory is recordings plus up to 40 sampled frames; for ST and LT, it is tracked names, states or locations, and start/end times.Source: §K.3 · pp. 28–29

K.4 PREDICTION MATCHING

  • Prediction matching pairs outputs one-to-one with ground truth using a 1 − sim cost plus a time term |∆t|^1.5, subject to similarity and asymmetric time-window thresholds.Pairs require similarity of at least 0.5, with a hard cutoff outside the allowed time window.Source: §K.4 · p. 29

K.5 LLM JUDGES

  • A temperature-0 LLM judge returns a semantic-similarity score from 0 to 1 for each comparison.At Intervention Level it compares the ground-truth object with the predicted tool-call name; at Notification Level it compares a templated ground-truth sentence with the model’s message.Source: §K.5 · p. 29
  • For Intervention-Level object names, synonyms and more- or less-specific names for the same real-world object count as 1.0, while unrelated objects count as 0.0.The judge rates object identity only.Source: §K.5 · p. 29
  • For Notification-Level messages, the judge scores whether the same object and event are announced, ignoring phrasing differences and extra detail.The reference sentence is task-templated for Information Capture, State Tracking, or Location Tracking.Source: §K.5 · p. 29

Related papers

Loading 2610.30089v1…