PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAME: Factored Retrieval via Attribute Readouts for Object-Centric Scene Memory

Woosang Jeon, Sanghyeok Choi, Minwoo Kim, Taehyun Jung, Taehyeong Kim

arXiv:2609.08886v1cs.CVcs.RO

TL;DR

Many object references specify multiple persistent attributes, but language-guided scene-memory retrieval has often emphasized spatial or relational references. The paper introduces a controlled protocol and FRAME, which composes attribute evidence from object embeddings; FRAME outperforms representative retrieval baselines, with lightweight post-decomposition scoring.

  • Problem

    Many everyday object references specify conjunctions of persistent attributes, while much language-guided scene-memory retrieval emphasizes spatial or relational references.

    Source: §1 · p. 1

  • Method

    The paper introduces a controlled fixed-memory evaluation protocol and FRAME, which converts language to attribute weights and composes learned per-attribute evidence from object embeddings.

    Source: §1 · p. 2

  • Results

    FRAME-Vis achieves the best overall result at 77.82% P@1, and FRAME outperforms representative scene-memory retrieval baselines in held-out scenes and object assets.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The results position attribute-compositional retrieval as a complementary way for robots to query scene memories using persistent object attributes as composable evidence.

    Source: §6 · p. 9

    The HSSD protocol uses a predefined attribute vocabulary, and some attributes, especially material, remain difficult to estimate reliably from a single view.

    Source: §6 · p. 9

Abstract

from arXiv · show

Language-guided robots need persistent scene memories to follow instructions, revisit objects, and resolve references to objects encountered over time. While much of language-guided scene-memory retrieval has emphasized spatial or relational references, many everyday object references specify objects by multiple persistent attributes, such as category, material, size, or surface appearance. We formalize this problem as attribute-compositional retrieval, where a fixed object-centric scene memory is queried with natural language to retrieve the object satisfying the requested attributes. To investigate this capability directly, we introduce a controlled evaluation protocol with fixed scene memories and attribute-defined targets, separating retrieval from perception and annotation ambiguities. We then propose FRAME, which turns language into query-relevant attribute weights, uses learned readouts to estimate per-attribute evidence from object embeddings, and ranks objects by aggregating this evidence according to the query. Across held-out scenes and object assets, FRAME outperforms representative scene-memory retrieval baselines while reducing post-decomposition object scoring to lightweight matrix-vector computation. These results position attribute-compositional retrieval as a complementary scene-memory capability for language-guided robots, showing that persistent object attributes can be exposed as composable evidence for accurate and efficient multi-attribute retrieval.

1 Introduction

Object-centric scene graphs support spatial and relational references, but everyday queries also identify objects through conjunctions of persistent attributes. The paper formulates this as attribute-compositional retrieval and proposes FRAME to compose attribute evidence from object embeddings, outperforming representative baselines in held-out scenes and assets.

1 Introduction

  • Object-centric scene graphs store object representations at nodes and spatial or geometric relations at edges, supporting spatial and relational language grounding [4].Prior 3D grounding work has consequently focused substantially on spatial anchors, relations, and graph-based reasoning.Source: §1 · p. 1
  • Many everyday references identify objects through conjunctions of persistent attributes, including category, material, size, and surface appearance.Unlike a wooden table or plastic chair, the target “wooden chair” must satisfy both the category and material attributes.Source: §1 · p. 1
  • The paper formalizes attribute-compositional retrieval as finding objects that satisfy query-specified persistent attributes in a fixed scene memory.Its controlled protocol fixes scene memories and defines targets using predefined attribute combinations, separating retrieval from perception and annotation ambiguity.Source: §1 · p. 2
  • FRAME decomposes language into attribute primitives, reads out attribute-specific evidence from object embeddings, and combines that evidence according to the query.In held-out scenes and object assets, FRAME outperforms representative scene-memory retrieval baselines while reducing post-decomposition object scoring to lightweight matrix-vector computation.Source: §1 · p. 2

2 Related Work

Prior work provides object-centric scene memories and 3D language-grounding benchmarks, but attribute composition is not evaluated in isolation. The paper studies persistent attributes as composable evidence for retrieval from fixed scene memories.

2 Related Work

  • 3D grounding datasets evaluate spatial, relational, viewpoint-dependent, and context-rich references, but their attribute cues are entangled with other language and scene context.The paper instead studies persistent-attribute composition directly over a fixed scene memory.Source: §2 · p. 2
  • Open-vocabulary and hierarchical scene-memory systems build object-centric memories from visual, textual, and geometric evidence, while other systems address long-horizon memory and episodic recall [17] [20].Prior work leaves open how constructed memories should be queried.Source: §2 · p. 2
  • LLM/VLM-mediated methods support flexible open-ended and relational querying, but typically reason iteratively over scene context or candidate objects [9].For persistent attribute-centric retrieval, attribute satisfaction is determined at inference time rather than represented as reusable object-side evidence.Source: §2 · pp. 2–3
  • Attribute recognition and compositionality have been studied in 2D, multimodal, and open-vocabulary 3D work, yet compositional retrieval remains challenging even when embeddings encode attribute-object associations [25].The paper treats attributes as composable object-side evidence for language-guided retrieval from persistent scene memory.Source: §2 · p. 3

3 Attribute-Compositional Retrieval Protocol

The protocol evaluates retrieval of objects satisfying fixed combinations of persistent attributes from identical, frozen scene memories. Its held-out test split contains unseen compositions, scenes, and object assets, with answerable query-scene pairs and near-miss distractors.

3 Attribute-Compositional Retrieval Protocol

  • The protocol uses fixed scene memories, predefined attribute-combination targets, and identical object inventories, frozen node features, and graph structure across approaches.This controls for scene representation and isolates attribute-compositional retrieval.Source: §3 · p. 3
  • The 47-attribute vocabulary covers fine and super categories, material, size, surface roughness, metallicity, and articulatability.The vocabulary derives from HSSD annotations and asset metadata; “attribute” includes categories and properties.Source: §3 · p. 3
  • Each target is a conjunction of one to three attributes, and the benchmark defines 60 compositions grouped into six classes.Ten constrained GPT-4o paraphrases are generated per composition while the target specification remains fixed; a correct object must satisfy every attribute.Source: §3 · p. 4
  • Testing uses 17 unseen compositions across 26 test scenes, with scene- and asset-disjoint splits and no use of unseen compositions for validation or model selection.The protocol also has 43 seen compositions across 25 validation scenes; primitive attributes in unseen compositions occur in training and validation.Source: §3 · p. 4, Table 1 · p. 4
  • Answerability filtering yields 3,720 test pairs and 8,510 validation pairs, and near-misses satisfy all but one required attribute.Near-misses are plausible distractors that require considering the full conjunction to reject.Source: §3 · p. 4

4 FRAME

FRAME represents a language query as attribute weights, estimates per-attribute evidence from object-node embeddings, and composes that evidence to rank objects. It can cache readouts at scene ingestion to reduce subsequent query-time scoring.

4 FRAME

  • FRAME factors retrieval into query-specified attribute weights, learned per-attribute readouts from object-node embeddings, and a rule that composes selected evidence.The evidence can be estimated once from stored representations and reused across queries.Source: §4 · p. 4, Figure 2 · p. 5
  • A single LLM call converts each natural-language query into sparse weights over the attribute vocabulary, emphasizing required attributes and down-weighting weak modifiers.Attributes irrelevant to the query are omitted.Source: §4 · p. 5
  • For each attribute, a trained two-layer MLP probe estimates evidence from an object-node representation.FRAME-Vis uses visual embeddings, while FRAME-VT concatenates visual and caption-derived text features.Source: §4 · p. 5
  • FRAME scores objects with a normalized weighted average of query weights and per-attribute evidence, then returns the highest-scoring object.Probes may run lazily for queried attributes or eagerly by caching all scores per scene at ingestion; eager mode trades a one-time ingestion cost for lower-cost query scoring.Source: §4 · p. 5

5 Experiments

Experiments evaluate attribute-compositional retrieval on held-out scenes and compare FRAME with embedding, LLM-mediated, and reranking baselines. FRAME-Vis achieves the strongest overall top-1 result, with further analyses of ranking quality, efficiency, attribute readouts, composition classes, and scene consistency.

5.1 Experimental Setup

  • All methods use the same captioned object-centric scene graph, with evaluation on 26 held-out test scenes and comparisons spanning embedding, LLM-mediated, reranking, and attribute-compositional retrieval.The test split contains 3,720 valid query–scene pairs for 17 unseen compositions.Source: §5.1 · p. 6

5.2 Compositional Retrieval Performance

  • 77.82% P@1: FRAME-Vis achieves the best overall result, 15.24 percentage points above CrossEncoder; FRAME-VT reaches 75.75% P@1.CrossEncoder is the strongest non-FRAME baseline; LLM-mediated and retrieval-then-reranking baselines remain below 44.57% P@1.Source: Table 3 · p. 7, §5.2 · p. 7
  • FRAME-Vis also leads Hit@5, MRR, and mAP; several baselines often place the correct object in their top five but struggle to rank it first.For example, ZS-Visual reaches 83.92% Hit@5 but 46.40% P@1, while BiEncoder reaches 87.15% Hit@5 but 48.87% P@1.Source: §5.2 · p. 7
  • Factoring and reliable learned readouts appear complementary: FRAME-ZS reaches 35.67% P@1, while task-tuned holistic BiEncoder and CrossEncoder models each remain below 63% P@1.FRAME-ZS factors queries but uses zero-shot attribute scores; BiEncoder and CrossEncoder use the same supervision as FRAME but collapse references into a single query–object score.Source: §5.2 · p. 7
  • FRAME caches object–attribute readouts per scene, making post-decomposition scoring a matrix–vector product taking approximately 0.01 ms.Embedding retrieval takes 10–24 ms, while LLM ranking and reranking take 2.0–6.3 s per query; FRAME latency is dominated by one query-decomposition call.Source: §5.2 · pp. 7–8

5.3 Performance Breakdown

  • Validation AP shows category and size readouts are reliable, other-property readouts are moderately strong, and material readouts are substantially weaker; material is therefore the main bottleneck for attribute-based retrieval.The paper suggests single-view material cues are confounded by texture, lighting, finish, and asset style.Source: §5.3 · p. 8
  • CrossEncoder falls from 95.5% P@1 on C1 to 38.5% on C6, while FRAME-Vis drops 38.1 pp (98.7 →60.5); among two-attribute classes, C3 is easiest and C4 hardest.C4 has the highest near-miss density among the two-attribute classes, a likely factor in its difficulty; FRAME variants outperform shown baselines across classes, with the largest gap on C6.Source: Figure 3 · p. 8, §5.3 · p. 8
  • Across 26 test scenes, FRAME-Vis has the highest median P@1 and FRAME-VT the tightest spread; 3DSSG results do not establish deployment-ready real-scene grounding.FRAME-VT's IQR is 9.13, compared with 19.43 for FRAME-Vis and 19.61 for CrossEncoder; weak size readouts on 3DSSG expose limitations from noisy geometry, absent depth or scale calibration, and imperfect segmentation.Source: Figure 4 · p. 9, §5.3 · p. 9

6 Conclusion

FRAME composes language-specified persistent attributes as evidence for retrieving objects, improving retrieval over representative baselines while making post-decomposition scoring lightweight. The controlled HSSD-based protocol leaves open broader vocabularies and richer references, and reliable robot-built memories remain a requirement for extending beyond the controlled setting.

6 Conclusion

  • FRAME improves attribute-compositional retrieval over representative scene-memory baselines while reducing post-decomposition object scoring to lightweight matrix-vector computation.It reads out evidence for persistent attributes from object embeddings and composes that evidence according to language-specified attributes.Source: §6 · p. 9
  • Attribute-compositional retrieval is positioned as a complementary way to query robot scene memories alongside spatial or relational querying.Source: §6 · p. 9
  • The HSSD-based protocol fixes the attribute vocabulary, leaving open open-vocabulary attributes, relational references, richer language, and more robust robot-built memories.Material is difficult to estimate reliably from a single view, limiting how far factored composition alone can compensate for unreliable evidence.Source: §6 · p. 9

A.1 Architecture

FRAME uses independent attribute-specific MLP probes to convert object representations into per-attribute evidence. Its FRAME-VT variant combines visual and caption embeddings, with separate probes and no parameter sharing or auxiliary losses.

A.1 Architecture

  • Each attribute has an independent two-hidden-layer MLP probe with hidden sizes 256 and 128, ReLU activations, and dropout after both layers.Dropout probabilities are 0.3 and 0.2, respectively.Source: §A.1 · p. 12
  • A probe maps an object representation to a scalar logit and applies a sigmoid at inference to produce an attribute score.Source: §A.1 · p. 12
  • FRAME-Vis uses 1152-dimensional inputs, while FRAME-VT concatenates separately ℓ2-normalized 1152-dimensional SigLIP visual and 1024-dimensional BGE caption embeddings.The probes are independent across attributes, with neither parameter sharing nor auxiliary losses.Source: §A.1 · p. 12

A.2 Training procedure

This section describes FRAME’s attribute-probe training and assessment, query and caption preparation, and adaptation of retrieval baselines to a shared protocol. It also outlines HOV-SG candidate filtering by room and category, followed by ranking against the original query.

A.2 Training procedure

  • Each attribute probe is trained as a binary classifier on the asset-disjoint pool, with positive-class weighting to address label imbalance.All 47 retained attributes pass the inclusion threshold, and the least frequent has 66 positive training objects.Source: §A.2 · p. 12

A.3 Validation reporting

  • Probes train for 80 fixed epochs, use the final-epoch checkpoint, and report per-attribute validation AP without selecting checkpoints by it.Composition-level P@1, Hit@5, MRR, and mAP are computed only during retrieval evaluation.Source: §A.3 · p. 12

A.4 Per-attribute validation AP

  • Material probes have the weakest family-mean validation AP, while other families are substantially stronger; FRAME-VT is lower than FRAME-Vis for most families.The grouped summary combines fine- and super-category probes as Category, and roughness, metallicity, and articulatability as Other properties.Source: §A.4 · p. 12, Table 6 · p. 13

B.1 FRAME query decomposition

  • FRAME makes one scene-independent LLM call per query, maps output to a fixed 47-attribute vocabulary, and weights required or preferred attributes while excluding irrelevant ones.Strict schema rules bar invalid keys and room-association attributes from object scores and prohibit adding unsupported or alternative constraints.Source: §B.1 · pp. 13–14

B.2 Paraphrase generation

  • The frozen paraphrase set contains 600 requests: 444 direct and 156 lightly indirect, with indirect wording preserving every required property.C1 compositions use 10 direct requests each; C2–C6 use seven direct and three lightly indirect requests, and generated requests undergo lexical checks and up to three retries.Source: §B.2 · pp. 14–15

B.3 Per-object caption

  • Each asset receives one offline caption reused by caption-based methods, with full or Type-only forms assigned according to the method.Captions use fixed fields for type, style, materials, features, atmosphere, and descriptors; the Type field serves as the short form.Source: §B.3 · p. 15

B.4 BiEncoder

  • BiEncoder learns query and object projections over frozen BGE and visual features, then retrieves by cosine similarity against cached projected object vectors.It uses near-miss and random negatives, samples one frozen query form per unit per epoch, and trains for 60 epochs with a final-epoch checkpoint.Source: §B.4 · pp. 15–16

B.5 CrossEncoder

  • CrossEncoder concatenates frozen query and visual representations and predicts relevance with an MLP, scoring each candidate jointly with the query.It shares BiEncoder’s negative sampling and fixed 60-epoch training policy but cannot use standalone cached object retrieval vectors.Source: §B.5 · p. 16

B.10 HOV-SG

  • HOV-SG ranks room- and category-filtered candidates by relevance to the original query and outputs their 1-based indices as a comma-separated list.Candidate captions are presented as a numbered list before ranking.Source: §B.10 · p. 19

C Active Attribute Compositions

The final protocol contains 60 active attribute compositions, including unseen test compositions. Each composition is represented by 10 frozen query forms.

C Active Attribute Compositions

D.1 Full test metric breakdown

The test breakdown reports retrieval metrics on unseen compositions and held-out scenes, with validation results kept separate. FRAME-Vis leads P@1 in five of six composition classes, while BBQ leads on C3.

D.1 Full test metric breakdown

  • The test split contains 17 unseen compositions and 26 held-out scenes; validation uses 43 seen compositions only for development diagnostics and readout-quality analysis.Source: §D.1 · p. 19
  • Table 8 reports ranking metrics on 3,720 valid query–scene pairs from the unseen-composition test split.Source: §D.1 · p. 19, Table 8 · p. 20

D.2 Per-class test P@1

E Real-scene Attribute Readouts on 3DSSG

The 3DSSG crop-level sanity check finds that learned attribute readouts retain real-scene signal unevenly: some semantic or visually distinctive attributes transfer, while size readouts remain close to random. The experiment is not a full end-to-end real-scene grounding benchmark.

Setup

Family-level results

Per-attribute pattern

Two layers of failure

  • Partial, distant, oblique, poorly segmented, or background-contaminated crops weaken readouts; absolute size is also difficult to infer from one RGB crop without depth or camera calibration.Source: E Real-scene Attribute Readouts on 3DSSG · p. 21

Implication

  • The sanity check does not establish deployment-ready grounding; full real-scene deployment is viewed as requiring better object-memory construction, including segmentation, view selection, and occlusion handling.Source: E Real-scene Attribute Readouts on 3DSSG · p. 21

Related papers

Loading 2609.08886v1…