PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrospective Open-Vocabulary Memory for Long-Term Object Search

Jiaming Wang, Zhiwei Xue, Chen Jizhuo, Peng Shiqi, Harold Soh

arXiv:2610.00330v2cs.ROcs.CV

TL;DR

Long-term object search must learn where movable objects usually occur despite uneven observation, since detections alone confound prevalence with opportunity. ECROM combines open-vocabulary detector evidence with per-location observation opportunity to infer prevalence for queries supplied after mapping and use it as a search prior. It improves occurrence estimation and active-search results over competing memories, while relying on a heuristic geometric proxy for visibility.

  • Problem

    Uneven observations make detection counts confound object occurrence with observation frequency, complicating estimates of where movable objects usually occur.

    Source: §1 · p. 2

  • Method

    ECROM combines open-vocabulary detector evidence with per-support observation opportunity in a censored model to infer query-time prevalence and use it as an active-search prior.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    ECROM raises long-term occurrence AP from .171 for the strongest baseline to .216, and reaches .381 SPL versus .339 for PredictiveGraphs-style memory, with paired intervals including zero.

    Source: §6.3 · p. 8

  • Takeaways & Limitations

    Separating semantic evidence from observation opportunity yields better prevalence estimates and more effective search priors in simulation and on a real robot.

    Source: §9 · p. 10

    Exposure is a heuristic geometric proxy for visibility, and the model shares one likelihood ratio across queries, view counts, and exposure levels.

    Source: §8 · p. 10

Abstract

from arXiv · show

Long-term object search requires learning where objects usually appear from repeated but uneven observations of a changing environment. We formulate retrospective open-vocabulary memory as probabilistic inference from censored observations, where the key idea is to reason with evidence per opportunity: a detection or non-detection should influence belief only in proportion to the robot's opportunity to observe the corresponding location. We introduce ECROM, which uses this principle to estimate long-term prevalence for concepts specified only at query time and converts the resulting belief directly into an active-search prior. To evaluate this problem, we introduce a controlled long-term benchmark in ten HM3D homes that independently varies object placement and observation opportunity across repeated traversals. ECROM improves support-level AP on held-out queries by 4.5 points and search SPL by 4.2 points over the strongest competing memory in each metric. The benchmark, dataset, and code will be open-sourced. Project page: https://jiaming.im/ecrom/

1 Introduction

Long-term search needs to learn where movable objects usually occur, but uneven observation makes detection counts confound occurrence with opportunity. ECROM estimates prevalence from opportunity-calibrated evidence for queries supplied after mapping and uses the resulting belief to guide search.

1 Introduction

  • Existing approaches mainly learn priors for unseen environments, ground queries in spatial maps, or reuse maps across goals, often assuming static objects or that the latest observation reflects current location..Source: §1 · p. 2
  • Repeated observations are uneven: detection counts and visit frequencies confound object occurrence with how often a location could be observed.Equal detection counts can imply different prevalence when observation opportunities differ.Source: §1 · p. 2
  • ECROM separates semantic evidence from observation opportunity to estimate long-term prevalence for concepts specified only at query time.Its resulting belief becomes an active-search prior.Source: §1 · p. 2
  • The paper introduces a controlled benchmark in ten realistic homes with independently varying object occurrence and observation opportunity, and reports improved distribution estimation and active search over several memory types, including physical-robot deployment..Source: §1 · p. 2

2 Related Work

Prior work addresses search in unseen or static environments, changing scene memories, and benchmarks for search or relocation. The paper distinguishes its goal as estimating where an after-the-fact concept usually occurs under uneven observation.

2 Related Work

  • Object-navigation priors and open-vocabulary maps primarily address searching unseen environments or grounding queries spatially, while this work estimates where objects usually occur in changing environments..Source: §2 · p. 2
  • Temporal memories retrieve past observations, maintain the latest state, or predict changes in predefined states or relations; they do not estimate after-the-fact concept prevalence while accounting for observation opportunity..Source: §2 · p. 3
  • ObjectNav and GOAT-Bench evaluate static or map-reused search, while RIO evaluates relocation; LOOP-Bench independently varies object locations and viewpoints and tests distribution recovery plus active search.Queries are specified after the history is recorded.Source: §2 · p. 3

3 Problem Setup

The task is to build a query-independent memory from repeated traversals, then infer the likely occurrence of an arbitrary object and use that estimate to search. The inferred prevalence is assumed stationary across episodes.

3 Problem Setup

  • A place–view atlas organizes traversal history into executable places and viewing directions, while geometric surface supports represent candidate object locations.Supports are fixed-size, class-agnostic patches of up-facing surface; an instance is attributed within 45 cm of a support voxel.Source: §3 · p. 3
  • For each support, the memory infers a posterior over an arbitrary query’s long-term prevalence from recorded history, assuming prevalence is stationary across episodes.The posterior mean supplies the search prior over supports.Source: §3 · p. 3

4 Evidence per Opportunity

ECROM updates prevalence beliefs by combining detector evidence with the opportunity to observe each support, then uses those beliefs to choose and revise active-search views. This separates genuine non-detections from locations that were not meaningfully observed.

4 Evidence per Opportunity

  • Each episode contributes semantic evidence and observation opportunity separately, so non-detections at unobserved supports need not count as evidence of absence.The resulting posterior becomes the search prior.Source: §4 · p. 3, Figure 2 · p. 4

4.1 A censored observation model

  • The censored model treats visible presence as physical presence multiplied by observability, with detector evidence affecting belief only when the object could have been observed.The model assumes object placement is independent of whether the robot observes the support.Source: §4.1 · p. 4, Figure 3 · p. 4

4.2 Observation opportunity and semantic evidence

  • Exposure is the fraction of support voxels returning depth within detection range, used as a query-independent geometric proxy for observation opportunity.It is heuristic: visibility also depends on object placement, size, viewing distance and angle, and occlusion.Source: §4.2 · pp. 4–5

4.3 Posterior over long-term prevalence

  • The posterior combines episode evidence and opportunities under conditional independence; zero-exposure episodes leave belief unchanged, and duplicating frames within an episode does not change the stored evidence.The posterior mean estimates long-term prevalence for the queried concept at each support.Source: §4.3 · p. 5

4.4 From memory to active search

  • For active search, posterior prevalence weights guide a greedy choice of views by exposed search weight per unit cost, with beliefs updated after each failed look.Failed looks discount well-observed supports more strongly, partially observed supports weakly, and unobserved supports not at all; the criterion is used as a heuristic under changing travel costs.Source: §4.4 · pp. 5–6

5 LOOP-Bench: Long-Term Object Occurrence Prediction

LOOP-Bench tests whether robots can infer where objects usually occur from repeated, uneven observations. Ten HM3D-Semantics homes independently vary hidden object placement and sampled routes and viewpoints, separating prevalence from observation opportunity.

5 LOOP-Bench: Long-Term Object Occurrence Prediction

  • LOOP-Bench contains ten homes with hidden long-term object distributions and independently sampled routes and viewpoints, separating object prevalence from observation opportunity.Each home has 20 mapping traversals; five of eight objects per home are evaluation queries, yielding 50 queries overall.Source: §5 · p. 6

6 Simulation Experiments

Simulation evaluates long-term occurrence estimates and active search on shared supports and trials. ECROM leads the reported distribution metrics and improves search metrics, while ablations show the roles of exposure modeling and graded detector evidence.

6.3 Results

  • AP rises from .171 for the strongest baseline to .216 with ECROM, which leads all four distribution metrics; search reaches .587 SR and .381 SPL versus PredictiveGraphs-style memory’s .533 SR and .339 SPL.Paired intervals against PredictiveGraphs-style memory include zero for SR and SPL; paired AP, Top-5, and AUROC intervals exclude zero against the strongest distribution-estimation baseline.Source: Table 1 · p. 7, §6.3 · p. 8

6.1 Setup

  • Evaluation averages AP, Top-5 mass, AUROC, and Bernoulli NLL over 50 queries, and search success rate and SPL over 150 trials with a 120 m budget; 22 absent-target trials count as failures.All methods receive the same observations, supports, and detector scores, and use the same planner for search.Source: §6.1 · p. 7, Table 1 · p. 7, §6.2 · p. 8

6.2 Baselines

  • Baselines include detection count, visit frequency, opportunity-normalized count, retrieval, latest-state memory, PredictiveGraphs-style filtering, and a uniform-prior Coverage search.Opportunity-normalized count improves every ranking metric among the baselines, but its Bernoulli NLL (.0773) is worse than Coverage’s constant prior (.0357).Source: §6.2 · p. 7, §6.3 · p. 8

6.4 Ablations

  • Setting exposure to full observation everywhere drops AP from .216 to .191 and SR from .587 to .520, with the AP gain largest for the least-observed third of queries (+.057).This variant treats non-detections on partially or unobserved supports as fully observed; its prevalence sums to 2.32 per query versus ECROM’s 5.10 and the true 4.76.Source: §6.4 · p. 8
  • Binary exposure still ignores unobserved traversals but treats partial views as full inspections; it matches ECROM’s Bernoulli NLL while lowering AUROC from .760 to .733 and reducing search success.Thresholded binary sensor evidence also underperforms graded evidence on every metric: threshold .25 has the lowest variant success rate (.473), while threshold .15 has the lowest AP (.175) and SPL.Source: §6.4 · pp. 8–9

7 Real-Robot Experiment

A Stretch 3 robot learned object prevalence from ten mapping traversals on an approximately 500 m2 office floor, then searched for five queried objects. ECROM succeeded more often and used less travel and fewer inspected views than the two comparison memories.

7 Real-Robot Experiment

  • The Stretch 3 maps an approximately 500 m2 office floor over ten traversals, with object locations resampled from hidden support distributions before each traversal.Search evaluates five objects with three test placements each; all methods share the start, detector, candidate views, and planner.Source: §7 · p. 9
  • ECROM succeeds in 14 of 15 real-robot trials, versus 10 for detection count and 11 for PredictiveGraphs-style memory, while reducing mean travel from about 29 m to 19.4 m and inspected views from about nine to 4.7.The trials cover five queried objects with three independently sampled placements each and a 20-view budget.Source: §7 · p. 9, Table 2 · p. 9
  • Travel reductions are largest for the red cup (25.4 m versus 42.2 m for the better baseline) and scissors (9.0 m versus 25.6 m), while methods perform similarly for headphones and cupcakes.For luncheon meat, ECROM finds the object in two of three trials, versus none for detection count and one for PredictiveGraphs-style memory; its remaining failure reaches the support but the detector does not fire.Source: §7 · p. 9

8 Limitations

The paper identifies approximations in the observation model and boundaries in the representation and search procedure.

8 Limitations

  • Exposure is a geometric proxy for observability, although actual visibility also depends on object location, size, viewing distance and angle, and occlusion.The proxy equates seeing half a patch with a one-in-two chance of observing the object.Source: §8 · p. 10
  • A single likelihood ratio is shared across queries, view counts, and exposure levels, though weak responses carry more negative evidence when a support has many views.The paper identifies a learned visibility model and observation-process-conditioned likelihood ratio as future directions.Source: §8 · p. 10
  • Supports lack receptacle semantics, prevalence is assumed stationary, and the planner is greedy rather than optimizing the full route.The paper notes receptacle structure could add prior information when observations are sparse.Source: §8 · p. 10

9 Conclusion

ECROM estimates long-term object prevalence from uneven observations by separating semantic evidence from the opportunity to observe each support. The appendix explains this censored model, its special cases, and why the search index is used as a heuristic in navigation.

9 Conclusion

  • ECROM gives better long-term prevalence estimates and search priors in simulation and on a real robot; ablations show contributions from opportunity weighting and graded detector evidence.The conclusion describes the approach as a retrospective memory for estimating where objects usually occur from repeated, uneven observations.Source: §9 · p. 10

A Derivations for the Censored Observation Model

  • The opportunity-weighted likelihood uses detector evidence to favor presence or absence, while observation opportunity controls how strongly that evidence affects prevalence.The appendix derives the opportunity-weighted likelihood and prevalence posterior under the censored observation model.Source: §A.1 · p. 13, A Derivations for the Censored… · p. 13

A.2 Derivation of the prevalence posterior

  • The prevalence posterior combines episode evidence under conditional independence, given the prevalence and each episode’s recorded opportunity.The derivation identifies the resulting posterior as Eq. 6 and Proposition 2.Source: §A.2 · pp. 13–14

A.3 Episode-level presence

  • The same stored representation supports retrospective episode-level presence queries and long-term queries about where an object usually occurs.Episode-level presence is obtained by integrating over inferred prevalence.Source: §A.3 · p. 14

A.4 Special cases and invariances

  • With zero observation opportunity, an episode contributes no information, so an unobserved support is not treated as an observed absence.The model generalizes ordinary prevalence estimation to partial observation and uncertain semantic evidence.Source: §A.4 · p. 14

A.1 Derivation of the opportunity-weighted likelihood

  • The likelihood factor’s direction depends on whether the detector likelihood ratio is above or below one, and its effect vanishes as opportunity approaches zero.A likelihood ratio of one makes the observation uninformative.Source: §A.4 · p. 14

A.4 Special cases and invariances

  • Duplicating identical frames leaves the posterior unchanged because opportunity uses the union of observed surface voxels and the episode score uses the maximum localized score.Distinct views can change the distribution of the maximum detector score.Source: §A.4 · p. 15

B Motivation for the Probability-per-Cost Index

  • The probability-per-cost index is optimal only in a simplified setting; navigation violates its assumptions, so ECROM recomputes it as a greedy heuristic after each view.In navigation, view costs depend on current position, observations expose multiple supports, and detections require verification.Source: B Motivation for the… · pp. 15–16

C Compared Memories: Formulations and Implementation

The section compares memories on shared stored keyframes, surface supports, and detector scores, isolating how each summarizes history. ECROM estimates physical occurrence per chance to observe, unlike retrieval, latest-state, count, or relation-persistence approaches.

Compared Memories: Formulations and Implementation

  • All methods use the same keyframes, supports, and OWLv2 scores, while differing in their history statistic.Retrospective estimates use all 20 mapping traversals; none of the methods sees evaluation queries.Source: C Compared Memories: Formulations and… · p. 16
  • Visit frequency divides detections by place visits, so it estimates detection per visit rather than occurrence per useful look.A place visit may not include viewing the support.Source: C Compared Memories: Formulations and… · p. 16
  • The opportunity-normalized count replaces visits with geometric opportunity but remains a thresholded, perfect-sensor estimator without a prior or graded evidence.RAVEN-style retrieval instead uses a recency-weighted maximum over retrieved frames and ignores exposed non-retrievals.Source: C Compared Memories: Formulations and… · pp. 16–17
  • ECROM estimates physical occurrence per chance to observe, whereas retrieval and latest-state methods retain salient or recent positives and persistence filters forecast established relations.The PredictiveGraphs-style filter is adapted to shared 0.5 m supports and traversal-index timing; its candidate set includes only supports with an accepted positive, holding 0.51 of ground-truth occurrence mass on average.Source: C Compared Memories: Formulations and… · pp. 17, 19

D Benchmark Construction Details

LOOP-Bench varies object placement independently from observation opportunity across repeated traversals in ten HM3D homes. It evaluates held-out queries against known placement distributions and uses shared conditions for memory comparisons and search trials.

Benchmark Construction Details

  • The benchmark contains ten HM3D-Semantics homes, with eight household objects per home and five held-out evaluation queries per home.Three objects per home are reserved for calibration, yielding 50 evaluation queries.Source: D Benchmark Construction Details · p. 19
  • Object placements are sampled independently across episodes from hidden distributions covering unimodal, bimodal, near-uniform, and rare-or-absent routines.Placement is independent of observation generation.Source: D Benchmark Construction Details · pp. 19–20
  • Distribution evaluation maps receptacle placements to surface supports using 15 cm-radius footprints and allocates placement mass by sampled footprint area.The resulting normalized support mass supplies labels for AP and AUROC and mass for Top-5.Source: D Benchmark Construction Details · p. 19
  • Each home has 20 mapping traversals with varied broad routes, local patrols, and camera behavior, so supports may be fully observed, partly observed, or missed.Across the benchmark, the observation process comprises 200 traversals, 79,741 RGB-D frames, and 9.01 km.Source: D Benchmark Construction Details · p. 20
  • All memories use the 20 mapping traversals, while distribution metrics compare estimates with known placement distributions and search uses three test traversals per query.The 50 queries therefore yield 150 search trials; two shifted-distribution test traversals per home are released but not used in this paper.Source: D Benchmark Construction Details · p. 20

E Representation and Implementation Details

ECROM represents surfaces as 0.5 × 0.5 m support patches and uses depth returns to measure episode-level opportunity. Its detector evidence, online detection model, and sparse prior are calibrated separately, with the likelihood ratio audited across view counts and opportunity levels.

E Representation and Implementation Details

  • Supports are 0.5 × 0.5 m patches of up-facing depth voxels, and episode opportunity is the fraction of patch voxels returning depth within detection range.Detector regions are assigned to supports when within 45 cm of reliable surface voxels.Source: E Representation and Implementation… · p. 20
  • A composite 400-node grid resolves the sparse Beta prior near zero, reproducing exact conjugate Beta–Bernoulli means and intervals.The uniform grid overestimates the no-evidence mean for Beta(.05, 8) as .026 versus the exact .0062.Source: E Representation and Implementation… · p. 20
  • Scoring half the frames halves query time to 40 s but lowers AP from .216 to .191 and SR from .587 to .533.The system treats unprocessed frames as censored and recomputes opportunity from processed-frame depth only.Source: E Representation and Implementation… · p. 21

E.1 Calibrating the model: likelihood ratio, detection model, prior

  • The score likelihood ratio is calibrated from episode–support responses labeled by presence and visibility, with absent detections assigned score zero.Simulation calibration uses three designated objects per home; score histograms are smoothed, made monotonic, and clipped.Source: §E.1 · p. 21
  • Weak-response evidence depends on view count: scores from .05 to .15 are neutral with few views (+.09) but negative with many (−1.13).Across opportunity levels, score-bin ordering stays the same; below .05, evidence is less negative at low opportunity (−1.43 versus −2.26).Source: §E.1 · p. 22, Table 5 · p. 22
  • The sparse Beta(.05, 8) prior has mean about .0062 and mainly sets beliefs where evidence is limited; the online update separately uses a calibrated visible-target detection rate.Across houses, the detection-rate estimate ranges from .05 to .73; applying the traversal-maximum likelihood ratio to a single live frame is not equivalent.Source: §E.1 · pp. 22–23

F Paired Uncertainty, Present-Target Search and Per-House Results

This block reports paired uncertainty and present-target search analyses, then details per-query results and the real-robot system’s differences from simulation.

F Paired Uncertainty, Present-Target Search and Per-House Results

  • On absent-target trials, four methods stop after 58–81 m when scores are zero on most supports and candidates run out; other methods use the 120 m budget.The four methods are detection count, visit frequency, opportunity-normalized count, and RAVEN-style retrieval.Source: F Paired Uncertainty, Present-Target… · p. 23, Table 7 · p. 24
  • Table 6 reports ECROM-minus-method paired differences with 95% intervals from bootstrapping houses and then queries within each house.This resampling preserves dependence among queries within a house and among the three trials of a query.Source: Table 6 · p. 23, F Paired Uncertainty, Present-Target… · p. 23

G Real-Robot Experiment: Per-Query Results

  • The robot experiment reports 15 trials per method and uses a detector event rather than verified discovery for belief updates.After each scored frame, only the assigned support’s belief changes; other supports retain their beliefs, and the update is not exposure-weighted.Source: G Real-Robot Experiment: Per-Query… · pp. 23–24, Table 11 · p. 25

F Paired Uncertainty, Present-Target Search and Per-House Results

G Real-Robot Experiment: Per-Query Results

Related papers

Loading 2610.00330v2…