Source-linked AI summary
COOL: Curiosity-Driven Object Ownership Learning for Personalized Robotic Assistance
Samira Huber, Ruben Hammele, Sören Pirk
TL;DR
Robots need to resolve object instances, people, locations, and ownership to carry out personalized commands, but ownership is rarely explicitly labeled. COOL learns ownership from long-term observations and maintains spatial memory with curiosity-driven data collection. Offline and real-world evaluations report ownership inference from interactions and use of that knowledge for navigation; its exploration strategy achieves 85.0% F1.
Problem
Robots must identify intended object instances and their owners to execute personalized commands, but ownership is rarely labeled and can be ambiguous in shared environments.
Source: §1 · pp. 1–2Method
COOL maintains long-term spatial memory of people, object instances, locations, and interactions, then uses agents to infer ownership and collect informative observations.
Source: §1 · p. 2Results
Offline and real-world evaluations report ownership inference from interactions and use of that knowledge for navigation, while curiosity-driven exploration achieves 85.0% F1.
Source: §1 · p. 2, §4.4 · p. 8Takeaways & Limitations
COOL can adapt ownership estimates as new evidence accumulates, including when ownership changes.
Source: §6 · p. 9Ownership can remain uncertain with sparse or ambiguous evidence, and deployments in shared spaces raise privacy and social-acceptability concerns requiring consent and data safeguards.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Robots are increasingly expected to provide personalized services in everyday environments. To do so, they must ground natural-language commands such as "Where is my backpack?" or "Find my bottle" and execute them by reasoning about object instances, people, locations, and ownership. This is challenging because ownership is rarely labeled explicitly and must be inferred from long-term, behavioral evidence of human-object interactions. To address this, we present COOL, a novel robotic framework for autonomously learning object ownership from everyday observations and maintaining a long-term spatial memory of its environment. To keep its memory current, COOL uses an agent-based curiosity-driven data collection strategy that guides the robot toward the most promising locations to gain information and refresh stale observations. Offline experiments, ablation studies, and real-world evaluations show that COOL can infer ownership relations from real-world interactions and use this knowledge for ownership-conditioned navigation and task execution.
1 Introduction
Personalized object search requires robots to identify object instances, infer who they are associated with, and locate them despite ambiguous ownership. COOL learns ownership from long-term observations and uses that evidence in personalized tasks.
1 Introduction
- Existing systems address memory-based search or personalized search separately, leaving autonomous ownership learning from long-term observations and its use in task execution unaddressed.Memory-based navigation does not model persistent ownership, while personalized search often relies on user input, demonstrations, or visual references.Source: §1 · p. 2
- COOL treats ownership as a latent relation inferred from long-term observations, linking people, object instances, locations, and interaction histories in persistent spatial memory.It clusters repeated observations, detects human–object interactions, and uses a curiosity-driven agent to refresh stale or incomplete memory.Source: §1 · p. 2
- Offline experiments, ablations, and real-world tests assess ownership inference from interactions, location updates, and use of this knowledge for object search and navigation.The evaluations assess associations between repeated observations of people and objects as well as ownership relations inferred from human–object interaction evidence.Source: §1 · p. 2
- The framework uses accumulated evidence to infer ownership and support ownership-conditioned questions, object search, and navigation without explicit labels, demonstrations, or dialogue.Its contributions include persistent spatial memory, a task-execution agent, and an autonomous memory-refresh agent.Source: §1 · p. 2
2 Related Work
Prior work addresses memory-based navigation, entity re-identification, object personalization, and human–object interaction. COOL combines these capabilities to infer person–object ownership from long-term observations for personalized assistance.
Memory-based Navigation
- Memory-based navigation stores spatial, semantic, or temporal observations for search and question answering, but does not model persistent ownership relations [1] [2] [3] [4] [5] [6] [7].Retrieval-augmented systems reuse prior episodes or perceptual streams, while general LLM memory systems focus on textual recall and consolidation [7] [18] [19] [20].Source: §2 · p. 2
Re-Identification
- Re-identification and tracking methods link observations across time using visual embeddings, identity matching, or motion and appearance cues [22] [23] [24] [25] [26] [28] [29].COOL uses tracking and re-identification within long-term memory to connect people and object instances to spatial observations and interaction histories.Source: §2 · p. 2
Object Ownership
- Personalized search systems use probabilistic inference, demonstrations, dialogue, feedback, visual references, or limited examples [10] [12] [13] [14] [15].COOL instead infers owner–object associations from long-term robot observations and uses them to select objects for person-specific search or navigation.Source: §2 · p. 3
3 Method
COOL combines persistent spatial memory with agents for ownership inference, task execution, and curiosity-driven memory refresh. It associates repeated observations, weighs interaction evidence to infer ownership, and actively gathers new observations where they can strengthen or update that evidence.
3 Method
- Two specialized language agents operate on shared spatial memory: a user agent retrieves evidence for ownership reasoning and navigation, while a curiosity agent expands and refreshes memory.The framework grounds observations in a metric map and stores them in spatial memory.Source: §3 · p. 3, Figure 3 · p. 4
3.1 Persistent Person and Object Memory
- Persistent memory links repeated people and object-instance detections through track continuity, spatial-cache recovery, embedding-based nearest-neighbor retrieval, and position matching for predefined static classes.Periodic centroid comparisons can merge fragmented clusters; face embeddings provide specific person matches while body embeddings preserve continuity when faces are unavailable.Source: §3.1 · pp. 3–4
3.2 Agent-Based Ownership Inference and Navigation
- Ownership is inferred at query time from retrieved interaction histories, with the agent weighing interaction specificity, repetition, recency, and consistency to label pairs owned, shared, or unknown.Direct interactions provide stronger evidence than weak interactions or co-presence; possessive commands use the inferred relation to select an object instance and its latest reliable map position.Source: §3.2 · pp. 4–5
3.3 Curiosity-Driven Data Collection
- Curiosity-driven collection balances likely ownership evidence, expected scene changes, and stale or sparse observations, using stay commands to gather repeated detections and move commands to select informative rooms.New detections and interactions remain active in the perception and memory pipeline during navigation and are added to long-term memory.Source: §3.3 · pp. 5–6
4 Experiments and Results
COOL is evaluated on synthetic office reasoning, sitcom footage, and real-world office navigation, alongside tests of its curiosity-driven exploration policy. Across these settings, the experiments examine ownership reasoning, robustness to perception noise, navigation, and memory refresh.
4.1 Offline Reasoning Benchmark: Synthetic Dataset
- In the oracle-memory synthetic benchmark, COOL achieves 100% ownership accuracy, while Rule-based, Plain LLM, and Heuristic baselines score 0%.The benchmark uses 79 questions to test reasoning over time, space, interaction histories, ownership, and navigation targets.Source: §4.1 · p. 6
4.2 Video-Based Evaluation: Sitcom Dataset
- 81.48% accuracy under 40% dropout, 78.51% with 10% false positives, and 79.63% with 50% face occlusion; severe dropout and identity swaps reduce accuracy to 61.12% and 50.01%.The robustness evaluation perturbs sitcom observations with eight perception failure modes.Source: Table 2 · p. 7, §4.2 · p. 7
4.3 Real-World Experiments
- At least 90% of real-world navigation trials succeed in every scenario, with 100% success for Room Navigation and Owner-based Distinction.The five tasks use 10 runs each; the Ownership-Preserving Update task tests whether location updates retain the ownership association.Source: §4.3 · pp. 7–8, Table 3 · p. 7
4.4 Curiosity-Driven Navigation and Exploration
- COOL’s curiosity evaluation tests change detection and stay-or-relocate decisions against strategies based on observation statistics.It uses a VLM to judge whether continued observation is warranted and evaluates 15 synthetic office logs spanning environmental conditions and change distributions.Source: §4.4 · p. 8
- 85.0% F1: COOL has the highest score among navigation strategies and is the only one with both precision and recall above 80%.On 24-room logs, COOL reaches 85.4% F1 versus 78.1% for the strongest baseline.Source: Table 4 · p. 8, §4.4 · p. 8
- Replacing COOL’s relocation policy with unconditional score argmax raises moves from 30 to 105 and lowers F1 from 85.0% to 76.4%.The argmax variant leaves fewer changes unobserved in absolute terms, but at 3.5× the movement cost; the navigator’s contribution is deciding when to relocate under a movement budget.Source: §4.4 · pp. 8–9
- Entropy achieves 93.3% precision with 14 moves, but its 70.7% recall leaves 818 change episodes unobserved per log, versus 400 for COOL.The comparison characterizes Entropy as exploiting known activity while COOL distributes observations across the environment.Source: §4.4 · p. 9
- During 4.5 hours of real-world exploration, the robot navigates a four-room office and spends more time in rooms with observed human activity.The trajectories show when the robot moves between rooms and when it remains in place, with charging breaks between runs.Source: §4.4 · p. 9
5 Limitations
COOL’s ownership estimates can remain uncertain when interaction evidence is sparse or ambiguous, and its real-world evaluation is limited in scope.
5 Limitations
- Sparse or ambiguous evidence, such as borrowing, moving, or brief handling, can leave ownership uncertain; clarification questions, visible name labels, or user confirmation may be needed.The system can report ownership as unknown or shared in these cases.Source: §5 · p. 9
- Storing long-term observations of people, objects, locations, and interactions raises privacy and social-acceptability concerns in shared spaces.The authors call for clear consent procedures, data minimization, access control, and retention policies.Source: §5 · p. 9
- Real-world evaluation covers one office, a small number of people and objects, and few navigation tasks; broader studies are needed to assess generalization.Figure 5 shows trajectories from a 4.5-hour deployment in a four-room office.Source: §5 · p. 9, Figure 5 · p. 9
6 Conclusion
COOL learns ownership from long-term observations and combines persistent spatial memory with agents for evidence collection and task execution. Its conclusion reports office-based assistance outcomes and identifies broader deployments and manipulation as future work.
6 Conclusion
- On a physical robot in an office, COOL improved ownership reasoning, resolved personalized search requests, collected relevant evidence more efficiently, and handled visually similar objects and changing locations.The authors conclude that interaction-aware memory and active observation can support personalized robotic assistance.Source: §6 · p. 9
- COOL infers ownership from accumulated evidence about people, object instances, and interactions, while two agents collect evidence and execute ownership-conditioned requests using spatial memory.The curiosity-driven agent refreshes stale or incomplete memory; the user agent grounds requests and directs navigation toward inferred target locations.Source: §6 · p. 9
- Future work will explore interactive clarification, broader multi-user deployments, and mobile manipulation beyond object localization toward retrieval and handover.Source: §6 · p. 9
A Object and Person Clustering
- Fixed person- and object-memory parameters govern re-identification, matching, spatial and temporal gating, and cluster consolidation, with the same values used across experiments unless stated otherwise.Table 5 summarizes the inference-time clustering parameters.Source: A Object and Person Clustering · p. 16, Table 5 · p. 16
B Structured Database Tools
The agents use structured tools to query memory and spatial context, with navigation occurring only after a reachable target is resolved. The user and curiosity-driven agents have distinct tool sets for request execution and room selection.
B Structured Database Tools
- Structured database tools provide typed inputs and outputs for searching entities, inspecting observations, querying interaction histories, and grounding decisions in the robot’s pose.Navigation occurs only after a target is resolved to a reachable map position.Source: B Structured Database Tools · p. 16
B.1 Tool Calls for the User Agent
- The user agent’s tools identify people and objects, retrieve histories and locations, inspect interaction evidence, and execute navigation after target selection.Table 6 covers tools for object search, inspection, interaction retrieval, scene understanding, and navigation.Source: §B.1 · p. 16, Table 6 · p. 17
B.2 Tool Calls for the Curiosity-Driven Agent
- The curiosity-driven agent queries room visitation history, recent interaction density, and time since observation to prioritize rooms by spatial coverage and expected information gain.Table 7 summarizes the tools used to select the next room.Source: §B.2 · p. 17, Table 7 · p. 17
C Offline Reasoning Benchmark
The offline benchmark evaluates reasoning over a structured synthetic office memory trace from one simulated workday. It tests the user agent on held-out questions covering interactions, locations, navigation, lookups, and ownership.
C Offline Reasoning Benchmark
- The synthetic office benchmark uses one simulated workday of rooms, people, objects, timestamped observations, map positions, and human–object interactions, with stable identities for people and objects.Table 8 presents the question distribution across evaluation categories.Source: C Offline Reasoning Benchmark · p. 17, Table 8 · p. 17
- The benchmark evaluates the user agent through structured-tool access to memory, with natural-language answers scored by semantic similarity against manually specified references.Questions are held out until final evaluation.Source: C Offline Reasoning Benchmark · pp. 17–18
- Question categories test interaction events, observed locations, navigation goal resolution, entity lookup, and ownership inferred from interaction and observation histories.Source: C Offline Reasoning Benchmark · p. 18
C.1 Robustness Evaluation Details
The robustness evaluation perturbs observations returned by memory queries while keeping the underlying database fixed, separating query-time perception errors from data collection and database-construction errors. The real-world evaluation also includes five task setups targeting distinct robot-memory and tool-agent capabilities.
C.1 Robustness Evaluation Details
- Query-time perturbations leave the underlying database unchanged, isolating perception errors from errors introduced during data collection or database construction.Perturbations are applied independently for each evaluation seed unless otherwise specified.Source: §C.1 · p. 18
- Table 9 covers detection dropout, false positives, embedding noise, track fragmentation, face occlusion, temporal jitter, action noise, class confusion, and identity swaps.These simulate missed or spurious detections, weaker retrieval, identity inconsistency, synchronization errors, incorrect interaction labels, and category confusion.Source: §C.1 · p. 18, Table 9 · p. 19
D Real-World Tasks
- The real-world experiments comprise five task setups, each targeting a capability of the robot memory system and tool-using agent.Source: D Real-World Tasks · p. 19
D.1 Detailed Tasks
The real-world tasks test room grounding and navigation, ownership-based object search, and updating an object's location without changing its inferred ownership. Success requires selecting the correct target and reaching its corresponding navigation goal.
D.1 Detailed Tasks
- Room navigation tests grounding a named room to a map region and reaching a goal at least 1 m inside it from multiple starting positions.Source: §D.1 · p. 19
- Object relocation tests re-identifying a ball and person across observations, inferring their association, and navigating to the ball's latest known position.Source: §D.1 · p. 19
- Owner-based distinction and visual-ambiguity tasks require selecting the correct person-associated bottle or visually similar backpack and navigating to its latest position.The bottle scenario separates people and objects by interaction and workplace; the backpack scenario distinguishes similar instances using prior interaction evidence.Source: §D.1 · p. 19
- Ownership-preserving update tests re-identifying a moved backpack and updating its location without changing its person–object association.Source: §D.1 · p. 19
D.2 Success criterion
- A run succeeds only if the correct room or entity is selected and the robot reaches the corresponding goal; wrong entities, stale locations, or wrong goals count as failures.Each task is evaluated over 10 runs after 10 hours of naturalistic office activity involving 10 people, 20 objects, and four rooms; runs vary starting pose and sometimes target instance.Source: §D.2 · p. 19, §D.3 · p. 20
E Curiosity-Driven Evaluation
The curiosity-driven evaluation compares navigation strategies for detecting room changes across simulated office worlds, with attention to coverage and movement as well as observed-step F1. It defines ownership-relevant changes, describes the simulator and evaluation protocol, and examines both policy ablations and destination-choice effects.
E Curiosity-Driven Evaluation
- The evaluation uses 15 synthetic worlds at 6-room and 24-room scales, with 183 decision steps per run and procedural person, object, and room consistency.Room activities, temporary items, stochastic movement, scripted events, and hot, mild, or flat occupancy priors vary the simulated conditions.Source: E Curiosity-Driven Evaluation · pp. 20–21, Table 10 · p. 20
E.1 Change Definition
- Major changes include person arrivals or departures, fixture changes, or changes to at least two objects; activity-only changes are minor.Changes are determined by comparing consecutive observations of each room, and major changes are treated as ownership-relevant.Source: §E.1 · p. 21
E.3 Change Distribution
- Change distributions vary across worlds; in 24-room worlds, the busiest room contains 7–14% of major changes, making camping expensive at scale.Hot-prior worlds have quiet storage rooms, while flat-prior worlds remove that gradient.Source: §E.3 · p. 22, Table 14 · p. 23
E.4 Metrics
- Observed-step F1 is conditional on presence and can be maximized by camping, so the evaluation reports move budgets and unconditional metrics alongside it.Source: §E.4 · p. 23
E.5 Protocol
- Nine strategies are compared across all 15 worlds, with each table cell averaging 10 seeds and reporting change-detection, yield, movement, and coverage measures.The protocol shares COOL's VLM change detector while each policy controls stay-or-move decisions; per-world tables report F1 variability and other metrics.Source: §E.5 · p. 23, §E.5.1 · p. 24, Table 15 · p. 24
E.6 Ablations
- At 7.7 moves, the all-three-signals ablation enters 22% of rooms and misses 812 changes per run; in the adversarial world, removing staleness drops F1 from 91.6% to 62.3%.The table notes that higher mean F1 in some ablations reflects camping, while the signals support worst-case robustness and coverage.Source: Table 30 · p. 28
E.7 Isolating the destination choice
- Granting the stay-or-move decision improves every baseline by 0.120 F1 on average, while changing movement rates in different directions across policies.Entropy moves increase from 13 to 45, sweeping policies decrease from 182 to 74, and the hazard policy decreases from 112 to 8.Source: §E.7 · p. 28
- Granting the detector’s stay-or-move decision improves every baseline by 0.120 F1 on average (range 0.056–0.162); under the evaluated condition, hazard-greedy reaches 90.7% F1 at 8 moves, leaving 715 changes unobserved.COOL leaves 389 changes unobserved at 28 moves, the fewest among strategies with a comparable movement budget.Source: §E.7 · p. 28
F Perception and Re-Identification Ablations
The perception and re-identification studies select models for noisy office-robot imagery and assess person and object clustering. Person re-identification favors fine-tuned OSNet on whole images for several metrics, while upscaling yields the highest recall.
F.1 Object Detection
- Object detection models are compared on a self-labeled office dataset using their best-performing confidence thresholds, with F0.5 chosen to prioritize precision over recall.False positives are considered more problematic than false negatives because incorrect detections may lead to erroneous robot behavior.Source: §F.1 · p. 29, Table 31 · p. 29
F.2 Person Re-Identification
- Fine-tuned OSNet achieves the best overall person-clustering performance; original whole-image input leads in Pairwise F1, purity, and precision, while Real-ESRGAN upscaling leads in recall.The test set contains 20 identities and compares original, upscaled, and brightness/white-balance-preprocessed inputs.Source: §F.2 · p. 29, Table 32 · p. 30
F.3 Object Re-Identification
- Object re-identification compares feature extractors and three image-input variants on a labeled office dataset whose identity labels sometimes span detector classes.Results are averaged over runs using greedy clustering and reported as mean ± standard deviation; no comparative outcome is stated in the supplied text.Source: §F.3 · p. 30, Table 34 · p. 31, Table 33 · p. 31
G Fine-Tuning
- OSNet fine-tuning uses 808 detections from 19 identities, split into 605 training and 203 validation samples, with checkpoint selection by validation pairwise F1.Training uses identity-balanced batches, AdamW, a cosine learning-rate schedule, and combined cross-entropy and triplet loss.Source: G Fine-Tuning · p. 32, Table 35 · p. 32