PaperlayerView exact revision on arXiv

Source-linked AI summary

Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos

Shravan Chaudhari, William Paul, Suchi Saria, Rama Chellappa, Homanga Bharadhwaj

arXiv:2610.10538v1cs.CVcs.AIcs.RO

TL;DR

Spatial questions may concern objects or details encountered earlier in everyday activity, before an agent knows what will be asked. Ledger builds persistent 3D object memories from egocentric videos, and improves reported spatial-answering results while localizing returned Ego4D predictions. Its scope remains limited by benchmark settings and unresolved counting, interval, and scaling challenges.

  • Problem

    An agent must decide what to retain from incomplete egocentric observations to answer later spatial questions about objects or events it could not anticipate, without replaying the video.

    Source: §1 · p. 2

  • Method

    Ledger builds offline persistent 3D object records from egocentric videos, combining locations, movement histories, and contextual descriptions for later text-based answering.

    Source: §1 · pp. 3–4

  • Results

    Ledger raises HD-EPIC accuracy from 29.7% to 42.6%, scores 46.3% versus DirectMe’s graph at 42.9% on UCS-Bench, and returns VQ3D predictions with 0.99 m median error.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The results indicate that past object observations can be reused for spatial answering and localization, with persistence and contextual descriptions contributing more than coordinate refinement.

    Source: §5 · p. 11

    Recent frame readers outperform memory-only answering on HD-EPIC, while counting, continuous rest intervals, and construction at the most object-dense hour-long scale remain unresolved.

    Source: §5 · p. 11

Abstract

from arXiv · show

As we move through the world and carry out everyday tasks, we encounter objects that may become relevant only later. We are capable of recalling where we left something or what was inside a container, even without knowing we would need it later. Here, we study how an embodied assistant can build a similar memory from egocentric videos, by observing a person's day-to-day activities. We present Ledger, a persistent 3D object memory that combines object locations, their histories, and contextual descriptions. It associates observations across the recording and retains objects after they leave the view, including those the person never touches. It clusters each object's observations by resting locations and records a move only after repeated evidence, reducing the effect of localization noise. Short descriptions preserve details such as an object's contents or supporting surface. It saves these records to later answer spatial questions without having to access the original images or video. Our memory raises HD-EPIC accuracy from 29.7% to 42.6%, UCS-Bench accuracy from 33.8% to 38.5% and localizes Ego4D objects with a 0.99 m median error on returned predictions. Our analyses identify complementary roles for temporal persistence, contextual descriptions, and retrieval. Our study on 100 stitched streams of multiple scenes each further exposes failures in both retrieval and construction. Per-scene construction partially recovers the performance lost across scene changes compared to that of single scene streams.

1 Introduction

The paper asks what an embodied agent should retain from everyday egocentric videos to answer spatial questions about objects and places later, without replaying the video. Ledger combines persistent object records, movement histories, and contextual descriptions; evaluations report gains across benchmarks and expose challenges when streams span multiple scenes.

1 Introduction

  • The central challenge is retaining useful evidence despite occlusion, brief views, changing viewpoints, localization errors, and not knowing which objects or events later questions will concern.The paper asks what must be preserved to answer spatial questions without revisiting scenes or replaying video.Source: §1 · pp. 2–3
  • The paper compares object memories, scene representations, language-model records, and frame-based systems to distinguish which retained evidence supports later questions from retrieval and reasoning failures.It highlights that interaction-only memories can omit visible objects relevant later, and that positions alone do not preserve contents or movement history.Source: §1 · p. 3
  • Ledger builds persistent 3D object records from egocentric videos, retaining objects after they leave view, including untouched objects, with locations, movement histories, and contextual descriptions.A new rest segment records a move only after repeated observations support displacement, reducing false motion detections from localization noise; descriptions capture details such as contents or supporting surfaces.Source: §1 · pp. 3–4
  • On HD-EPIC, adding Ledger raises GPT-5.4’s prototype-averaged accuracy from 29.7% to 42.6%; on UCS-Bench, Ledger scores 46.3% versus 42.9% for DirectMe’s graph, and VQ3D median error is 0.99 m on returned predictions.The evaluations cover all 2,400 HD-EPIC 3D Perception and Object Motion questions and 2,771 timestamped UCS-Bench questions.Source: §1 · p. 4
  • Across recording lengths and 100 paired stitched streams, memory retains gains up to 75 minutes, while scene changes expose construction and retrieval failures that per-scene construction partly repairs.The paper also identifies complementary roles for temporal persistence, contextual descriptions, and retrieval.Source: §1 · p. 4

2 Related Work

Related work spans object tracking and scene representations, systems that read stored records, and approaches that answer from video frames or visual-token memories. The paper evaluates these memory choices across spatial tasks and distinguishes representation limits from retrieval and reasoning failures.

2 Related Work

  • Prior object memories track identities and locations through absence, while scene representations organize object relationships, locations, or scene changes in different ways.Examples include OSNOM, Whareformer, AMEGO, ESOM, ConceptGraphs, and Khronos.Source: §2 · p. 4
  • The study examines how retained objects, estimated positions, movement histories, and descriptions affect spatial answers, alongside geometry errors’ effects on answers and localization.It modifies memory construction and retrieval to identify why performance changes.Source: §2 · pp. 4, 6
  • Memory-based readers search captions, entity records, or structured graphs, while video-language systems answer from sampled frames and streaming models compress past frames into visual tokens.The paper compares object records and event descriptions with caption memory under shared retrieval procedures to separate stored information from readers’ ability to find and use it.Source: §2 · p. 5
  • Spatial benchmarks cover object localization, positions and motion, timestamped reasoning, scene-graph questions, environment questions, long-video reasoning, and evidence spanning days.The paper complements these benchmarks by testing individual recordings and streams formed by joining recordings.Source: §2 · p. 5

3 Method

Ledger constructs a reusable object memory offline from posed egocentric recordings, then grounds queries and answers from retrieved textual records. Its records combine timestamped observations, resting-location segments, and context, with repeated evidence required to record displacement.

3 Method

  • Ledger is built once per recording, independently of questions, and reused for spatial question answering and named-object localization.On UCS-Bench, reading the memory costs about 2.0k input tokens per question, compared with 9.6k for 48 sampled frames.Source: §3 · p. 6
  • Each object record stores a label and timestamped observations, then summarizes its location history as rest segments with a time span, estimated 3D location, and contextual description.Segments summarize observed resting locations; positions during gaps between observations are unknown, and unrelated scenes are not registered into a shared frame.Source: §3 · p. 6

3.1 Offline memory construction

  • Offline construction detects objects whether or not they are touched, associates observations across time, and keeps records after objects leave view; a new rest segment requires k successive observations beyond threshold τ.This persistence rule suppresses false moves from localization noise but may miss brief ones; descriptions capture contents, supporting surfaces, and interactions, while triangulation can refine positions under specified viewpoint and stability conditions.Source: §3.1 · p. 6

3.2 Query-time reasoning

  • At query time, HD-EPIC retrieves nearby instances around a lifted 3D query point, VQ3D returns a selected named instance’s stored position, and UCS-Bench ranks inferred targets by word overlap and temporal proximity.HD-EPIC fixture matching uses fixed synonym groups and camera heading for clock directions.Source: §3.2 · pp. 6–7
  • Retrieved labels, ranking signals, and segment histories are rendered as text for a single language-model call, without showing the answerer images.For timestamped UCS-Bench queries, snapshots remove later observations and descriptions and truncate histories and the wearer path; construction itself is not causal.Source: §3.2 · p. 7

4 Experiments

The experiments test whether stored object records support spatial answering, how persistence and geometry affect the records, and how retrieval and scene changes shape performance. Ledger helps across answering models and recording lengths, while scene changes expose failures in both memory construction and retrieval.

Three memories on one stream

  • In one illustrative stream, Ledger answers all five questions, compared with one for DirectMe’s graph and none for ReMEmbR; its Q2 description recovers the held object despite a wrong detector label.For Q5, Ledger’s chair position is 0.73 m from the annotation, while ReMEmbR returns only the wearer’s position.Source: §4 · p. 7

RQ1

  • On HD-EPIC, the memory gains points over blind controls and, when read by GPT-5.4, outscores every rebuilt memory; on UCS-Bench, it improves the 48-frame reader.The clearest benefit is fixture location. Interaction counting and the onset of rest remain weaker when event evidence or stable identities are needed.Source: §4 · p. 7

RQ2

  • Requiring three consecutive observations beyond a 0.3 m radius distinguishes repeated movement evidence from localization noise better than a distance threshold alone.Half of annotated moves are shorter than 0.8 m, while the 95th percentile of localization error is 0.84 m; an 0.8 m threshold captures only 57.3% of counts within ±1 of truth.Source: §4 · p. 7

RQ3

  • Camera-reference errors matter more to spatial answering than object-coordinate errors: estimated poses cost fixture-location questions 11 points, or 29 without gravity levelling.Triangulation adds 2.2 points overall, less than persistence (5.5) or descriptions (4.9); its localization improvement depends on pose quality.Source: §4 · p. 9

RQ4

  • With the same agent reading both records, Ledger beats ReMEmbR captions by 5.0 points on HD-EPIC and by 26 points on VQ3D queries within 1 m, but is level on UCS-Bench.Reader choice changes comparisons: ReMEmbR leads natively only on UCS-Bench, while the single-shot reader beats the agent by 7.8 points on the same HD-EPIC record.Source: §4 · p. 10

RQ5

  • Ledger stays 10–18 points above the blind control in every HD-EPIC duration bin through 75 minutes, while scene changes bring stitched-stream performance to the blind level.On UCS-Bench, the 48-frame reader loses accuracy as recordings grow while memory does not; across scenes, retrieval and construction both fail, and per-scene construction recovers about half the loss in emulation.Source: §4 · pp. 10–11, Figure 5 · p. 10

5 Conclusion and Limitations

The paper concludes that persistent object records help spatial answering, while missing event evidence, fragmented identities, and scene changes constrain their use. Its experiments use sampled frames, a bounded object vocabulary, and detector-based lifting; the paper also directs readers to appendices for protocols and analyses.

5 Conclusion and Limitations

  • Ledger makes past object observations reusable for spatial answering and localization, but missing event evidence and fragmented identities limit interaction and counting.Persistence and contextual descriptions contribute more than coordinate refinement; per-scene construction partly repairs the losses associated with scene changes.Source: §5 · p. 11
  • The scope is bounded by offline construction, limited metric scale on UCS-Bench, unresolved counting and continuous rest intervals, and poor scaling to the most object-dense hour-long recordings.HD-EPIC uses calibrated poses, and question-time filtering does not establish causal online memory.Source: §5 · p. 11

Reading guide

  • The reading guide maps the method, benchmarks, five research questions, failure analysis, and memory-versus-frames discussion to their main-text sections and appendices.It points to dedicated analyses of movement noise, geometric errors, retrieval, and scene changes.Source: Reading guide · p. 18

A.1 Sampling, vocabulary, and detection

  • The detection pipeline samples one frame every four seconds, names visible objects, merges synonyms into at most 80 labels per recording, then detects matching objects.UCS-Bench uses at most 80 frames; which objects enter memory depends on the vocabulary, detector confidence threshold, and 3D-lifting budget.Source: §A.1 · p. 19

A.2 Camera poses and object lifting

The pipeline estimates camera poses and lifts detected boxes into 3D, then associates observations and records persistent resting locations with contextual descriptions. Multi-view refinement estimates object positions from viewing rays, subject to stability checks.

A.2 Camera poses and object lifting

  • Camera poses come from calibrated HD-EPIC trajectories or registered EgoLoc poses on VQ3D; estimated trajectories use overlapping chunks, and metric scale recovery fails for most UCS-Bench recordings and all TeleEgo recordings.WildDet3D supplies a metric 3D box centre, but scene-coordinate positions are metric only when camera poses have known scale; the pose estimators have similar median errors on VQ3D, though Depth Anything 3 fails to join chunks on 11 clips.Source: §A.2 · p. 19

A.3 Association and consolidation

  • Trajectory memory associates detections chronologically by label and proximity, then consolidation reconnects likely tracks using propagated-mask agreement while blocking incompatible merges.Even after consolidation, long absences or large moves may give an object a new identity.Source: §A.3 · p. 20

A.4 Persistence and contextual descriptions

  • Persistence confirms a new resting segment only after k successive observations lie beyond radius τ; descriptions preserve contents, supporting surfaces, and interactions that positions alone miss.For up to six segments per object, a representative frame is described; in the reported HD-EPIC runs, persistence segments inherit descriptions from the distance-threshold segment with greatest temporal overlap.Source: §A.4 · p. 20

A.5 Multi-view refinement

  • Multi-view refinement estimates the point minimizing squared distances to viewing rays, and accepts it only when ray geometry is stable and the estimate remains close to the monocular prediction.HD-EPIC requires at least three rays, a camera baseline of at least 0.10 m, a minimum eigenvalue of 0.25, and a shift no greater than 2.0 m.Source: §A.5 · p. 20

B.1 Evaluation scope and per-prototype results

The evaluation reports HD-EPIC results across prototypes, compares memory systems and answering setups, and tests components that may affect performance.

B.1 Evaluation scope and per-prototype results

  • 10.8 points [8.2, 13.3] for Qwen3.5-9B and 13.0 points [10.4, 15.5] for GPT-5.4 are the prototype-averaged HD-EPIC gains over matched blind controls.The evaluation covers all 2,400 questions in 3D Perception and Object Motion; memories were built from 150 videos totaling 40.7 hours.Source: Table 6 · p. 21, §B.1 · p. 21

B.2 Implementation and query adapters

  • 97.8% of 365 validation questions receive the correct fixture direction using ground-truth positions; with memory, retrieval finds a name-matched fixture for 83% of 500 questions.Across all 500 questions, choosing the option closest to the first candidate’s clock direction scores 59.2%, versus 56.0% for GPT reading candidate histories.Source: §B.2 · p. 23

B.3 Published baselines and aggregation

  • Memory exceeds the cited video baselines and 2025 challenge entries under prototype averaging, but trails two 2026 challenge reports; per-prototype results vary by task.Fixture location is the only prototype exceeding every reported baseline, while object-contents retrieval is among the weakest results.Source: §B.3 · p. 23, Table 8 · p. 24, Table 9 · p. 24

B.4 Memories rebuilt on our setup

  • The rebuilt memory’s 95% bootstrap interval is [40.5, 44.8], compared with [28.8, 32.8] for DirectMe, [27.8, 31.8] for ReMEmbR, [28.2, 32.2] for OSNOM, and [32.6, 36.7] for AMEGO.The methods were rebuilt on the same 150 videos and 2,400 questions, with GPT-5.4 answering from each memory under the same prototype-averaged metric.Source: §B.4 · pp. 24–25

B.5 Component ablations

  • After correction, descriptions improve object location by 8.4 points [3.0, 13.4] and fixture-interaction counting by 7.8 points [3.8, 12.0], while persistence improves movement counting by 15.5 points [8.1, 23.2].Each condition uses a single GPT-5.4 run; repeating identical prompts changes whether 15–20% of answers are correct.Source: §B.5 · p. 25

B.6 Design ablations

The analyses test design choices, retrieval and geometry, movement and stationary-state reasoning, localization protocols, and triangulation.

B.6 Design ablations

  • Only estimated poses without gravity levelling significantly change mean accuracy after correction; individual prototype scores vary by about ±7–10 points in single runs.The pose-free variants still use ground-truth information to rescale camera centres, and the GPT-5.6 gain combines a model change with added reasoning.Source: §B.6 · p. 26

B.7 Earlier retrieval, oracle, and geometry analyses

  • Triangulation reduces median 3D error from 0.39 to 0.35 m and raises the share within 0.25 m from 13.6% to 37.3% on 389 matched detections.In the earlier 339-question analysis, retrieval criteria differ by at most 0.3 accuracy points, while changing the answerer changes accuracy by 7.4 points; the trajectory oracle does not isolate failure sources.Source: §B.7 · pp. 27–28

B.8 Movement and stationary-state analysis

  • At a 0.3 m threshold, the rule over-counts movement in 64% of questions; larger thresholds suppress localization noise but can miss small moves.Only 22% of stationary-localization answers are correct when choosing the timestamp nearest a rest-segment start, versus 20% chance.Source: §B.8 · p. 28, Table 16 · p. 28

B.9 Development data and cost

  • WildDet3D takes 0.85, 0.99, and 0.94 seconds per detection at 1802, 960, and 480 pixels, respectively; these timings exclude end-to-end throughput.Segmentation, retrieval rules, triangulation thresholds, and fixture synonym groups were developed on 5–10 separate pilot videos.Source: §B.9 · p. 28

C.1 Protocol and interpretation

  • Localization metrics are conditional on returning a candidate: the 0.99 m median error does not establish success across all queries or an end-to-end improvement on official VQ3D.The variants are scored on their own returned queries, and conditional success can be near-ceiling despite errors of several metres.Source: §C.1 · p. 29

C.2 Implementation settings

  • The localization setup samples one frame every four seconds, caps detections per tag and scene, and uses GPT-5.4 to choose among at most four candidates.Qwen is evaluated for tagging and coverage, not as a complete localization pipeline.Source: §C.2 · p. 29

C.3 Localization across memory variants

  • With EgoLoc poses, consolidation lowers median error by 0.029 m for SAM 3 and 0.026 m for YOLO-World, but this single-position task does not test identity continuity or later question answering.A small localization difference does not establish that tracking is unnecessary for those tasks.Source: §C.3 · p. 29, Table 17 · p. 30

C.4 Triangulated against monocular positions

  • With EgoLoc poses, triangulation lowers median error by 0.09–0.12 m on eligible queries, but with FastVGGT poses the gain falls to about 0.03 m.Across all graded queries, mean error falls in every configuration, while median error does not improve; the refinement requires at least three rays and other geometric guards.Source: §C.4 · p. 30

C.5 Published methods on the validation split

The validation-split comparison notes protocol differences, rebuilds prior memories under a shared evaluation harness, and examines detector and vocabulary effects.

C.5 Published methods on the validation split

  • With YOLO-World, 10% of queries have errors above 3 m and account for 26% of total error; excluding them lowers mean error to 1.19 m.The text-title memory query differs from the benchmark’s visual-crop input, limiting direct comparison; clip identity explains 42% of L2 variance, without establishing scale error as the cause.Source: §C.5 · pp. 30–31, Table 19 · p. 31

C.6 Memories built by prior methods on the same clips

  • The shared comparison uses the same 44 clips, text query, GPT-5.4 matcher, and scoring procedure, but reports methods on the 138 queries answered by every capable method.ReMEmbR returns a camera position associated with retrieved memory, localizing the wearer rather than the object; AMEGO cannot answer these queries because it stores neither object names nor 3D positions.Source: §C.6 · p. 32, Table 20 · p. 32

C.7 Detector and vocabulary analysis

  • More detections or greater box agreement with SAM 3 need not mean lower downstream localization error; tag coverage alone does not establish detection, instance identity, or correct position.Coverage estimates also vary by judge: a semantic judge can reject a tag set containing the exact query word.Source: §C.7 · pp. 32–33, Table 22 · p. 33

D.1 Benchmark, construction, and answering protocol

The appendix evaluates Ledger’s answering conditions and analyzes performance across counting, timestamp-filtered queries, and stitched streams. It finds benchmark gains alongside limitations in counting and cross-scene construction and retrieval.

D.1 Benchmark, construction, and answering protocol

  • The main UCS-Bench evaluation covers 2,771 timestamped questions, with 48 sampled frames for frame-based conditions and memory added to spatial questions by keyword routing.The stitched-stream experiment joins recordings from different places, wearers, and cameras at scene-change boundaries.Source: §D.1 · p. 34, §E.1 · p. 37
  • Ledger outperforms DirectMe’s graph by 8.6 points (p < 10−12) and the blind control by 4.7 points (p < 10−4) in paired tests.With frames and the same routing, Ledger also beats DirectMe’s graph by 3.4 points (p = 0.0005) and frames alone by 1.9 points (p = 0.047).Source: §D.1 · p. 34

D.2 Counting

  • Counting tracks gives the correct answer on 11% of questions and is within one on 22%; largest-single-frame track counts raise these rates to 16% and 45%.Counting remains unresolved: two memory configurations score 33.7% and 31.6% on counting questions, below the 35.8% blind score.Source: §D.2 · pp. 35–36

D.3 Development pilot and causal construction

  • On the timestamp-filtered pilot, Ledger scores 38.4% versus 31.9% blind, a 6.5-point gain with an interval of [−2.9, +15.9].Because construction still uses the full recording, filtering outputs does not establish that earlier memory states are independent of later footage.Source: §D.3 · p. 36

E.2 Where the loss appears

  • On scene-change streams, memory accuracy falls 8.0 points for EPIC-Kitchens questions (p < 10−4), while losses for TeleEgo and EgoBlind are smaller and nonsignificant.Across stitched streams, Ledger beats DirectMe’s graph by 6.8 points overall (p < 10−9), or 4.0 points excluding questions handled by DirectMe’s counting rule.Source: §E.2 · pp. 37–38

E.3 Construction and retrieval

  • Restricting retrieval and trimming histories to the current scene recover 3.0 accuracy points (p = 0.011), while tag canonicalization and pose checks also fail across scene boundaries.Separate per-recording builds reach 37.0%, 40.4%, and 36.6% by stream position, versus 37.2%, 42.9%, and 38.3% on individual recordings.Source: §E.3 · pp. 38–39

E.4 Scene-cut detection and per-scene construction

  • The cut detector finds 89 of 112 true boundaries within 60 seconds, and full per-segment rebuilding scores 36.5% versus 33.8% for whole-stream construction.The rebuild covers 39 of 40 scene-change streams; its 2.7-point gain has p = 0.08 and remains below the 39.5% individual-recording result.Source: §E.4 · p. 39

E.5 Event lines

  • On held-out streams, event lines improve accuracy by 2.0 points [0.5, 3.6] on individual recordings and 2.6 points [0.9, 4.4] on stitched streams.These gains are reported for the held-out 78 streams; nine retrieval or prompt variants were tested, and a development-only gain did not transfer.Source: §E.5 · pp. 39–40

E.6 Leave-one-out ablation

The ablations show that event lines and descriptions contribute to some stitched-stream question sets, while overall component-removal effects remain limited. Comparisons across records and readers indicate that retrieval affects measured performance, and delayed questions challenge memory-based systems.

E.6 Leave-one-out ablation

  • No component removal shifts overall accuracy by more than 1.3 points on questions every arm answered, though category-level changes reach 13 points.On a larger per-arm set, removing event lines costs 2.2 points (p = 0.006) and descriptions 1.8 points (p = 0.02); Figure 7a and Table 31 report the leave-one-out comparison.Source: §E.6 · p. 40, Table 31 · p. 40, Figure 7 · p. 41

E.7 Record against reader

  • Switching ReMEmbR’s captions to our reader lowers accuracy by 5.0 points (p < 0.001), while our records score 1.2 points below the reference with our reader and 0.8 above it with ReMEmbR’s agent.Routing spatial questions to our reader and other questions to ReMEmbR’s agent yields 42.0% accuracy; Figure 7b and Table 32 describe the crossed record-reader comparison.Source: §E.7 · p. 41, Figure 7 · p. 41, Table 32 · p. 41

E.8 Open answering models

  • Ledger’s gain over the Qwen3.5-9B blind control is 3.4 points (p < 0.001) on the stitched-stream evaluation.The comparison includes 3,367 questions, with 58 unparsed answers scored as wrong.Source: §E.8 · pp. 41–42

E.9 Delayed questions

  • On delayed questions about earlier scenes, memories lose 3 to 4 points net of the blind-control change, while the frame reader loses most.Ledger’s single-shot reader is weaker than ReMEmbR’s agent on these questions because its time bonus favors observations near the question, which is at stream end.Source: §E.9 · p. 42, Table 34 · p. 42

F Accuracy by Recording Length

Recording-length analyses find that performance patterns differ by benchmark: HD-EPIC accuracy declines with length, while UCS-Bench memory conditions do not show that trend. A pipeline analysis identifies substantial losses before the answerer reads the memory.

F Accuracy by Recording Length

  • On HD-EPIC, every arm loses accuracy on longer recordings, but Ledger’s gain over blind ranges from +9.9 to +18.0 points across the five length bins.The blind control declines fastest, and Ledger leads ReMEmbR by 12 to 19 points in every bin; participant and recording length are partly confounded.Source: F Accuracy by Recording Length · pp. 42–43, Table 35 · p. 43
  • On UCS-Bench single recordings, neither memory condition nor the blind control changes with length, while the 48-frame reader declines with each doubling.The frame-reader decline also appears within EPIC-Kitchens, the only source represented in every length bin.Source: F Accuracy by Recording Length · p. 43, Table 36 · p. 43
  • VQ3D is not a length sweep: its clips fall into two duration groups, and the measure counts all 164 queries localized within 1 m, with unanswered queries as misses.Table 37 reports the share localized within 1 m by clip length.Source: F Accuracy by Recording Length · p. 43, Table 37 · p. 44

G.1 When an answer is wrong, where was it lost?

  • On development videos, supplying the queried object’s ground-truth trajectory adds 12 points and naming its fixtures adds another 25, while a fixed rule adds 3 points on fixture location.The largest measured losses occur before the answerer reads the record; the analysis links weak categories to memory construction and retained information.Source: §G.1 · p. 44

G.2 Three memories on one stream

On an illustrative three-recording stream, Ledger answers all four selected questions, compared with one for DirectMe and none for ReMEmbR. The examples show how stored descriptions and object positions distinguish Ledger’s answers from the other memories.

G.2 Three memories on one stream

  • Ledger answers all four selected stream questions, while DirectMe answers one and ReMEmbR none.The questions were chosen to illustrate differences between the memories, not to measure how often they occur.Source: §G.2 · p. 44, Figure 8 · p. 46
  • For Q2, Ledger’s segment description identifies the held device as an immersion circulator rather than the detector’s coffee maker label, enabling a match to the heating rod.DirectMe has no node for the object, and ReMEmbR’s nearby captions do not name it.Source: §G.2 · p. 44
  • For Q5, the chair-position error is 0.73 m for Ledger, 1.81 m for DirectMe, and 2.37 m for ReMEmbR.ReMEmbR stores the wearer’s position rather than the object’s.Source: §G.2 · p. 44
  • Table 39 organizes evidence loss by pipeline stage, affected questions, and measurement.Figure 8’s Q2 example shows how the memories’ retrieved evidence differs.Source: Table 39 · p. 45, Figure 8 · p. 46

H Answer-Time Token Cost

Answer-time cost depends on whether a method reads frames or stored records, and on what its memory preserves. On UCS-Bench, frames cost substantially more than Ledger’s record alone, while adding the record to frames costs relatively little and improves accuracy.

H Answer-Time Token Cost

  • Answer-time tokens and calls are averaged per question; memory construction is excluded because each method builds its memory locally without API calls.The comparison concerns answering-model usage, not construction compute time.Source: H Answer-Time Token Cost · p. 47
  • On HD-EPIC, Ledger’s record takes one call of about 1.8k input tokens, while ReMEmbR uses about 14 times as many tokens over roughly four calls and scores at the blind-control level.ReMEmbR’s captions lack object positions; the table covers 2,400 questions and reports prototype-averaged accuracy.Source: Table 40 · p. 47, H Answer-Time Token Cost · p. 47
  • Reading 48 UCS-Bench frames costs about 9.6k input tokens per question, versus 2.0k for Ledger’s record; adding the record to frames costs 0.9k more and recovers 1.9 points over frames.Frames alone buy 5.9 points over the record alone; the table covers 2,772 questions, with frame arms reading 48 uniformly sampled frames.Source: Table 41 · p. 47, H Answer-Time Token Cost · p. 47
  • On the 2,183 questions ReMEmbR answered, it scores 41.3% against Ledger’s 38.5%, while using 18,917 input tokens over 3.7 calls per question versus 2,041 tokens in one call.The comparison is restricted to questions ReMEmbR answered; the passage notes that whether its accuracy is worth the cost depends on the budget.Source: H Answer-Time Token Cost · p. 48

Related papers

Loading 2610.10538v1…