PaperlayerView exact revision on arXiv

Source-linked AI summary

LOCUS: Landmark-Oriented Container Discrimination Using Spatial Graphs

Taylor Bergeron, Shibani Senthilbabu, Kevin Leahy

arXiv:2610.02803v1cs.RO

TL;DR

Finding occluded objects requires robots to use both semantic and spatial relationships to discriminate among plausible containers. LOCUS updates CLIP embeddings with a scene-graph GNN; in zero-shot RoboCasa evaluation, it achieved the best IRA on 14 of 17 queries and beat baselines despite a fivefold larger candidate set.

  • Problem

    Finding occluded objects requires reasoning about physical and semantic relationships to discriminate among multiple plausible containers, including instances of the same class.

    Source: §I · p. 1

  • Method

    LOCUS combines ontology-derived object roles and CLIP embeddings with a scene-graph GNN that passes semantic information according to spatial relationships.

    Source: §I · p. 1, A. Graph Construction and Ontology · p. 2

  • Results

    14 of 17 RoboCasa queries had LOCUS's best IRA, and zero-shot evaluation beat baselines despite a fivefold larger candidate set.

    Source: 2) Results · pp. 6–7

  • Takeaways & Limitations

    LOCUS enables a robot to choose among containers, including same-class instances, to retrieve an occluded object.

    Source: §VI · p. 8

    LOCUS ranks a fully built scene graph rather than a partial one, so it requires an exploration pre-step.

    Source: §VI · p. 8

Abstract

from arXiv · show

As robots are increasingly deployed in unstructured, real-world environments, the ability to reason about complex spatial and semantic relationships among objects remains a fundamental challenge in enabling robust and generalizable manipulation and navigation. For example, deciding where to search for an object that is not in plain sight depends on where it is physically plausible as well as semantically likely. Popular methods such as cosine similarity with CLIP struggle to disambiguate between spatially and semantically similar objects that could contain a target object. We propose Landmark-Oriented Container Discrimination Using Spatial Graphs (LOCUS). To jointly reason about spatial information and semantics, we train a GNN to update CLIP embeddings on a full environment scene graph by passing embedding information between nodes based on proximity. CLIP embeddings, augmented by semantic knowledge from a fusion of household ontologies provide a robust semantic signal. We evaluate in simulation on a noiseless scene graph from simulator metadata, making the approach detection-agnostic. Our approach outperforms random, CLIP, Tidybot, and an LLM planner in the majority of room classes and configurations in simulation. To show our approach is robust to scene graph node placement and label noise, we demonstrate a physical mobile manipulator running in an exploration pipeline start-to-finish including scene graph labels from Detic.

I. INTRODUCTION · II. BACKGROUND AND PROBLEM FORMULATION

LOCUS addresses the challenge of choosing between same-class containers when searching for occluded objects. It combines ontology-informed semantic knowledge with spatial context from scene graphs to update CLIP embeddings for instance-level discrimination.

I. INTRODUCTION

  • Occluded-object search requires physical and semantic reasoning: with a floor plan but no room-item knowledge, robots must order plausible containers such as multiple drawers that could hold a fork.The paper posits that an approach needs both the generality of LLMs or VLMs and structured ontology knowledge.Source: §I · p. 1
  • LOCUS updates CLIP embeddings with spatial context learned over an environment scene graph, combining formal ontologies with general semantic knowledge to distinguish containers by both place and appearance.The approach pretrains a GNN over scene graphs and uses ontologies to learn typical spatial relationships.Source: §I · p. 1, §II · p. 2
  • The contributions include an object-container-landmark ontology, an ontology-informed iTHOR placement dataset, an ontology-using scene-graph builder, and a room-agnostic GNN for updating CLIP embeddings.The GNN is trained on explored environment scene graphs to pass semantic information from the environment into the embeddings.Source: §I · p. 1

II. BACKGROUND AND PROBLEM FORMULATION

  • CLIP retrieval methods,, give nearly identical embeddings to same-class containers, while prior container methods [12],, reason by class rather than instance.TidyBot selects a receptacle type from LLM preferences and therefore cannot order instances of the same class.Source: §II · pp. 1–2
  • Prior search approaches use pretrained models and LLMs,, [7], [8], or scene graphs [10], [11], but lack ontology-grounded semantics or scalable placement categories.The paper notes that LLM-generated training data can hallucinate, while manually crafted categories can be sparse and poorly reflect real-world distributions.Source: §II · pp. 1–2
  • LOCUS targets the gap of discriminating between same-class container instances under full occlusion, using a floor plan and relevant ontologies to generalize across configurations and domains.The paper frames the problem as choosing which container instances to search to physically retrieve an occluded object.Source: §II · p. 2
  • At runtime, LOCUS ranks nodes by cosine similarity between an updated container embedding and the query text, retaining CLIP’s comparability while adding spatial and semantic reasoning.Source: §II · p. 2

III. TECHNICAL APPROACH · A. Graph Construction and Ontology · B. Graph Neural Network

LOCUS builds an ontology-based heterogeneous scene graph that encodes proximity and room context, then uses a GNN to propagate spatial and semantic information so nearby context can distinguish containers with similar CLIP embeddings.

A. Graph Construction and Ontology

B. Graph Neural Network

  • The GNN updates node representations by aggregating information from connected landmarks and the room node, helping distinguish same-class containers with nearly identical CLIP embeddings.Source: B. Graph Neural Network · p. 2

1) Semantic Information: · 2) Message Passing:

LOCUS represents containers using visual, semantic, and spatial features, then propagates context across a scene graph so container representations reflect nearby landmarks and rooms. The resulting embeddings are normalized and scored against query text embeddings by cosine similarity.

1) Semantic Information:

  • Each node combines visual appearance, semantic identity, spatial context, and relative position to distinguish objects of the same class.The representation captures both what an object looks like and where it is relative to the room and other object nodes.Source: 1) Semantic Information · p. 2
  • Container features combine max-pooled CLIP crop embeddings across viewpoints, a class-label text embedding, a full-frame embedding, and a four-dimensional height encoding.The full-frame embedding comes from the viewpoint where the container occupies the largest pixel area; height is encoded as normalized height plus floor, counter, and wall tier indicators.Source: 1) Semantic Information · pp. 2–3
  • A soft landmark-distance vector encodes proximity to each training landmark type as exp(−d), while deployment detections receive soft landmark-type assignments from CLIP prototype similarities and softmax.Landmark nodes use the same input layout as containers.Source: 1) Semantic Information · p. 3
  • Room features combine the room-class text embedding with mean text embeddings of member containers and landmarks; containers and landmarks share a residual encoder, while rooms use a separate linear projection.Source: 1) Semantic Information · p. 3

2) Message Passing:

  • Message passing lets containers absorb spatial-neighbor context, so identical-looking drawers can be represented differently depending on whether they are near a sink or stove.Source: 2) Message Passing · p. 3
  • APPNP propagates landmark-to-container messages weighted by exp(−d) and room-to-container messages with unit weight; a linear head, L2 normalization, and cosine similarity score containers against query text embeddings.Source: 2) Message Passing · p. 3

3) Loss:

LOCUS trains with multi-positive InfoNCE and soft ranking losses, alongside auxiliary losses for container prediction and preserving representation distinctions.

3) Loss:

  • Multi-positive InfoNCE contrasts every container holding the target against all scene containers, while landmarks appear only in the denominator and queries are weighted by inverse positive rate.Similarity uses cosine scores between post-GNN node embeddings and the query embedding, with temperature applied only inside the exponent.Source: 3) Loss · p. 3
  • The soft ranking loss penalizes the expected number of containers opened before finding the target, directly aligning training with Expected Containers Opened (ECO).The ranking loss uses a unit-temperature sigmoid.Source: 3) Loss · p. 3
  • A stop-gradient binary cross-entropy head predicts whether each node is a container, and a learned scalar β boosts predicted container scores while suppressing landmark scores.The score adjustment is +βŷ for containers and +β(ŷ−1) for landmarks.Source: 3) Loss · p. 3
  • A teacher head matches each container representation to soft K-means assignments from open-view CLIP embeddings, preserving distinctions obscured by closed-door appearance.The auxiliary target helps prevent the encoder from collapsing visual distinctions.Source: 3) Loss · pp. 3–4
  • The total objective combines InfoNCE and search losses with weighted containment and teacher auxiliaries, using λsf, λc, and λt for the latter three terms.L = LInfoNCE + λsf Lsearch + λc Lcontain + λt Lteacher.Source: 3) Loss · pp. 3–4

IV. EXPERIMENT … C. Metrics

The experiment asks a robot to find a text-labeled item, including when it is occluded in a container, and evaluates container selection using rank-based and semantic metrics. The setup defines containers and landmarks from simulator metadata and ontologies.

A. Task

  • The robot searches for a text-labeled item that may be visible in the scene graph or occluded inside a container, then chooses which containers to search and in what order.Source: A. Task · p. 4

B. Ontology

  • Containers are simulator objects flagged as both openable and receptacle [28], while landmarks are non-pickupable, nonopenable objects in landmark-relevant ontology classes.Landmark classes draw on SUMO, SOMA, and WordNet; countertops and stove burners qualify, but garbage bags do not.Source: B. Ontology · p. 4

C. Metrics

  • ECO is the primary metric: it ranks containers by score and measures how many must be opened before the first positive container is found.ECO covers all 201 (scene, query) episodes, while IRA is unattainable at 1.0 under partial information when identical drawers cannot be distinguished without priors.Source: C. Metrics · p. 4
  • ECO/N divides ECO by the scene’s container count, reporting the fraction opened before finding the target for comparisons across environments.The passage gives approximately 10 containers for iTHOR and 34 for RoboCasa.Source: C. Metrics · p. 4
  • S-IRA and S-ECO/N count any co-placement-equivalent container of the same class as correct, and the ECO/N–S-ECO/N gap measures errors selecting a functionally equivalent container.For example, two drawers that both hold forks are treated as interchangeable.Source: C. Metrics · p. 4
  • IRA tests whether the top-scored instance is the positive among same-class containers, and is computed only for discriminative queries with both positive and negative containers.IRA applies to 165 discriminative episodes, so Table III uses dashes where it is not applicable.Source: C. Metrics · p. 4
  • The 42 query classes produce 201 scene-query episodes; ECO uses all episodes, whereas IRA uses only the 165 discriminative ones.The passage notes that this difference accounts for dashes in Table III.Source: C. Metrics · p. 4

D. Baselines … 1) Setup:

LOCUS is evaluated against similarity, prior-based, TidyBot, LLM, and MLP baselines in a simulation setup that isolates graph reasoning from perception noise. The setup uses iTHOR scenes with landmark-sensitive semantic placements and held-out evaluation.

D. Baselines

  • Visual CLIP [1], class-text CLIP, and room/type priors—with optional landmark-distance scaling—provide similarity and prior-based container-ranking baselines.Visual CLIP uses the full-frame image embedding where each container is most visible; priors rank by P(q|room)×P(q|type), optionally scaled by e−d/σ.Source: D. Baselines · p. 4
  • TidyBot, two LLM ranking variants, and a full-spatial-feature MLP with K=0 complete the comparison baselines.TidyBot selects a receptacle type and randomly selects instances within it; this chance-IRA upper bound also covers ConceptGraphs. Claude Haiku 4.5 ranks either a serialized scene graph or raw detections.Source: D. Baselines · p. 4

1) Setup:

  • Training and evaluation use iTHOR metadata scene graphs, removing perception noise and eliminating simulation exploration or sim-to-sim transfer.The scene graph is generated directly from simulator metadata.Source: 1) Setup · p. 4
  • The household dataset spans 120 iTHOR scenes across four room classes, split into 80 training, 20 validation, and 20 test scenes, with dense multiview RGB coverage.The 120 scenes comprise 30 kitchens, 30 living rooms, 30 bedrooms, and 30 bathrooms; coverage captures reachable positions, four yaw rotations, and three camera tilts.Source: 1) Setup · p. 4
  • A semantic placement policy distributes 42 query-object classes into openable receptacles, favoring purpose-compatible containers and nearby landmarks while limiting crowding.Three placement seeds yield labels with 99.6% cross-seed agreement; landmark-sensitive placements create instance-level differences, such as drawers in one kitchen holding different objects.Source: 1) Setup · pp. 4–5
  • LOCUS trains for 500 epochs on 80 scenes and averages test results over 30 training seeds, using a 3-layer residual MLP and APPNP with K=2 hops.Training uses learning rate 10^-4, weight decay 10^-4, batch size 32, and τ=0.07; the held-out test set has 20 scenes, five per room class.Source: 1) Setup · p. 5

2) Results:

On the 201-episode iTHOR test split, LOCUS outperforms the baselines, while its ablations and error analysis show the value of spatial reasoning and the relative cost of its mistakes.

Results

  • TidyBot opens .338N containers per search versus LOCUS at .257N, a 24.0% improvement; their correct-class rates are 92.0% and 95.2%, respectively.The LLM is the strongest baseline on ECO search cost but trails LOCUS on every metric; its near-random IRA despite nearly LOCUS-level class prediction highlights instance discrimination as the bottleneck.Source: 2) Results · p. 5, Table I · p. 5
  • K = 0 reaches IRA 40% above the class-ranking chance upper bound, and APPNP at K = 2 adds another 7% across 30 seeds (p<.001).The authors attribute the first result to spatial features alone and the additional gain to inter-node context; counting functionally equivalent containers lowers ECO from 2.83 to S-ECO 2.17 containers.Source: 2) Results · p. 5
  • LOCUS’s ECO/N is 63% better than random in kitchens, while living rooms have the worst ECO/N at 0.471; fridge-stored foods are trivial with class priors (ECO=1.0).LOCUS dominates utensils and tableware, where spatial context helps resolve which drawer to open; IRA .367 means the correct instance ranks first about 37% of the time, so the method reduces search cost without reliably finding the target on the first opening.Source: 2) Results · p. 5, Table II · p. 5, Table III · p. 5
  • 90% of LOCUS errors are right-type, wrong-instance errors, while about 50% of baseline errors are costly wrong-type errors.LOCUS’s errors therefore often require a few additional openings rather than causing search failure.Source: 2) Results · pp. 5–6, Table IV · p. 6
  • LOCUS uses 3.18M parameters and runs in under 3 ms on CPU, offline and deterministically; its fully connected graph also handles RoboCasa scenes with 20–40 containers.The average iTHOR scene graph has 10 containers, 11 landmarks, and 122 edges.Source: 2) Results · p. 6

3) Circularity and Determinism:

Relabeling test episodes with HouseKeep votes leaves LOCUS’s search cost essentially unchanged, while cross-seed reproducibility and near-equal performance on novel and trained queries support the conclusion that it is not learning the generator’s rules or specific object–container pairings.

3) Circularity and Determinism:

  • Under HouseKeep relabeling, LOCUS’s ECO/N is essentially unchanged, though 89% cabinet labels make a query-independent cabinet-first ranker best there and near-worst under the authors’ labels.The relabeling replaces the landmark-proximity instance choice with a uniform draw and the type conventions with HouseKeep crowd votes.Source: 3) Circularity and Determinism · p. 6, Table VI · p. 6
  • 99.6% cross-seed reproducibility and near-equal RoboCasa performance on novel versus trained queries suggest LOCUS learns neither generator rules nor specific object–container pairings.The 99.6% figure measures the generator’s cross-seed reproducibility.Source: 3) Circularity and Determinism · p. 6

4) Ablations: … 2) Results:

LOCUS transfers zero-shot from iTHOR to RoboCasa kitchens, outperforming baselines despite a much larger candidate set and generalizing to unseen query categories. Ablations indicate that height features and landmark-context message passing contribute to performance.

4) Ablations:

  • Removing the 4-d height-tier features raises ECO, suggesting those features contribute to reduced search cost.Source: Table VII · p. 6, 4) Ablations · p. 6
  • Message passing improves ECO by approximately 0.28 with or without soft-ldist, while the strongest performance uses both features.The passage attributes the message-passing gain to landmark context carried by the graph.Source: Table VII · p. 6, 4) Ablations · p. 6
  • Uniform and e−d edge weights perform similarly, suggesting finer geometric weighting is unlikely to improve performance.Source: Table VII · p. 6, 4) Ablations · p. 6

B. Sim-to-Sim Transfer: RoboCasa — 1) Setup:

  • The zero-shot test uses 1,188 RoboCasa episodes across 10 layouts, 3 styles, and 5 seeds, with 20–40 openable containers per scene.RoboCasa uses a different asset library and floor-plan distribution from iTHOR, while keeping the placement policy and labeling convention fixed; its kitchen scenes are rendered in MuJoCo, and RoboCasa is cited as.Source: 1) Setup · p. 6

B. Sim-to-Sim Transfer: RoboCasa — 2) Results:

  • Trained only on iTHOR, LOCUS beats baselines zero-shot in RoboCasa kitchens despite a fivefold larger candidate set.No finetuning was used.Source: 2) Results · p. 6, Table VIII · p. 7
  • LOCUS maintains consistent performance across in-distribution and novel query splits, generalizing to unseen object categories without finetuning.Source: 2) Results · p. 6, Table IX · p. 7
  • LOCUS achieves the best IRA on 14 of 17 RoboCasa queries, with strong results for utensils and food.The passage links these results to transfer of spatial conventions, such as utensils near countertops and perishables in the fridge.Source: 2) Results · p. 7

C. Mobile Manipulator Hardware · 1) Setup:

The hardware trial tested the simulation-trained GNN on a Stretch 3 in a deliberately cluttered real kitchen, using Detic detections to build a scene graph. Across three queries, the robot ranked openable nodes and searched them, with real-world perception and communication noise absent from simulation.

1) Setup:

  • The simulation-trained GNN was tested on a Hello Robot Stretch 3 in a deliberately cluttered real kitchen, adding extraneous detections as scene-graph nodes.Detic supplied open-vocabulary detections, with 3D poses extrapolated from bounding boxes and depth images.Source: 1) Setup · p. 7
  • Ontology-matched container or handle labels were marked openable, while other nodes provided spatial context; the GNN ranked the full graph for robot search.Overlapping detections were deduplicated by confidence, retaining the highest-scoring box.Source: 1) Setup · p. 7
  • The evaluation averaged three runs per query–method combination from a fixed start, panning the camera to map the kitchen for fork, mug, and sponge.The completed scene graph and test kitchen are shown in Fig. 4.Source: 1) Setup · p. 7, Figure 4 · p. 7
  • Real detections introduced estimated rather than ground-truth poses, depth-sensor noise, and ROS message throttling over rosbridge, which reduced frames and could degrade oblique-angle detection.These perception and communication effects were absent from the noiseless simulation scene graph.Source: 1) Setup · p. 7

2) Results: · VI. CONCLUSION

LOCUS performs competitively on average and uses spatial reasoning to make “smarter mistakes” under label noise. The conclusion describes transfer to other simulators and hardware, alongside limitations in scene-graph completeness, spatial signal, and dataset coverage.

2) Results:

  • Table X shows that LOCUS performs competitively on average.Source: 2) Results · p. 7
  • For sponge retrieval, Detic mislabels the containing cabinet as a drawer, affecting LOCUS and Text CLIP, while spatial reasoning yields “smarter mistakes.”Source: 2) Results · pp. 7–8

VI. CONCLUSION

  • LOCUS is presented as the first approach enabling a robot to choose among containers, including same-class instances, to retrieve an occluded object.Source: §VI · p. 8
  • The GNN-updated CLIP embeddings transfer across simulators and to hardware.Source: §VI · p. 8
  • LOCUS ranks fully built scene graphs rather than partial ones, so it requires an exploration pre-step.Source: §VI · p. 8
  • Queries with weak spatial signal stay near-random, while landmark-sparse rooms such as living rooms offer little message-passing signal.Source: §VI · p. 8
  • Because no public dataset covers instance-level object-in-container ground truth across diverse room classes, the authors built an AI2-THOR dataset using a hand-crafted placement policy that omits real-home variability; prior work gives only class-level preferences.Source: §VI · p. 8

Related papers

Loading 2610.02803v1…