OMAR: Open-World Multi-Attribute Reasoning for Scene-Conditioned Shelf Organization

• Robots must infer where novel objects belong even when household organization changes and instructions or known object-receptacle associations are unavailable.Source: §I · p. 1, Figure 1 · p. 1
• OMAR grounds unknown objects across views in an instance-level 3D scene graph that captures attributes beyond category labels.The representation includes content, packaging type, edibility, and orientation.Source: §I · pp. 1–2
• Its LLM infers latent shelf organization and ranks scene-consistent placements without user-provided preferences or predefined object-receptacle mappings.The ranked proposals guide geometric and kinematic feasibility checks.Source: §I · p. 2
• The evaluation reports improved instance mapping and placement coherence over single-view language-grounding and geometry-only baselines.Source: §I · p. 2

A. Classical and Learning-based P&P Approaches

• Classical and learning-based pick-and-place methods address grasp-placement compatibility or physical stability, while some omit semantics or struggle with confined spaces.Pick2Place, Shanthi and Hermans, and UniP address place-aware grasping; REAP guides search toward geometrically feasible actions.Source: A. Classical and Learning-based P&P… · p. 2

B. Foundation Model-based P&P Approaches

• Foundation-model systems learn user-specific placement preferences,, or reason over 3D scene graphs for object selection and grounding,.Source: B. Foundation Model-based P&P Approaches · p. 2
• Other foundation-model methods target grasping –, semantic placement,,, or confined-space rearrangement,, but do not explicitly couple placement semantics with grasp-dependent feasibility.Source: B. Foundation Model-based P&P Approaches · p. 2
• OMAR uses a persistent multi-view instance map with detailed attributes to consider semantic and geometric fit, reachability, and grasp feasibility across shelf boards.Source: B. Foundation Model-based P&P Approaches · p. 2

III. TASK DESCRIPTION AND ASSUMPTIONS

• The robot must place an unknown target at a semantically coherent location in an open-set shelf scene containing multiple boards.Source: §III · p. 2

IV. OUR APPROACH

• The pipeline discovers the target and shelf, builds an object instance map and 3D scene graph, then generates grasp and placement candidates.It matches geometrically possible placements with semantically suitable candidates, filters them through feasibility checks, and plans collision-free trajectories.Source: Figure 2 · p. 3
• Multi-view semantic instance mapping and LLM-based multi-attribute reasoning are combined with target-oriented antipodal grasp generation to select grasp-placement pairs.Source: §IV · p. 3

A. Autonomous Open-World Scene Perception

• At multiple viewpoints, a VLM predicts labels and attributes such as content, packaging type, edibility, and orientation for the target and shelf objects.The shared schema supports cross-view association and placement prediction without class-specific retraining or predefined mappings.Source: A. Autonomous Open-World Scene… · p. 3
• The system grounds predicted labels in image masks and depth, updates an attribute-annotated view set, and builds a target mesh for grasp generation and placement validation.Source: A. Autonomous Open-World Scene… · p. 3

B. Multi-View 3D Scene Representation

• The two-stage mapper and scene-graph conversion consolidate partial observations into view-independent object identities and spatial relations for inferring latent shelf organization.The map combines geometric association with VLM-derived extent cues; semantic disagreement rules out a match only when supported by sufficiently confident observations.Source: B. Multi-View 3D Scene Representation · p. 3, Multi-View Semantic Instance Mapping · p. 4
• The multi-view representation preserves alternative label hypotheses and accumulates evidence, allowing repeated support to outweigh isolated misclassifications while retaining alternatives for revision.Label evidence is weighted by segmentation confidence and a Gaussian extent-consistency term; unmatched detections initialize instances and unmatched instances remain available to later views.Source: Multi-View Semantic Instance Mapping · p. 4
• A 3D semantic scene graph stores each object's semantic attributes, bounding-box dimensions, and centroid, with edges encoding shelf-board membership and neighboring-object relations.Board assignment uses known board heights and centroid height; lateral and depth ordering encode left/right and in-front/behind relations.Source: 3D Semantic Scene Graph Generation · p. 4

• Antipodal grasp candidates are sampled on grids over the target mesh’s bounding-box faces, with up to six approach directions and rotations about each approach axis.The grid sampling is supplemented with perturbations of up to ±3 mm along the gripper approach, closing, and binormal axes.Source: C. Model-Aware Grasp Prediction · pp. 4–5
• Perturbations of up to ±3 mm along three gripper axes add candidates between the systematic grid samples.Source: C. Model-Aware Grasp Prediction · p. 5
• FoundationPose estimates the target’s 6-DoF pose for world-frame collision and inverse-kinematics checks, after which candidates are ranked by geometric quality and a vertical-approach preference.The quality score favors antipodal contact-normal alignment, finger-pad centering, and an enclosing aperture.Source: C. Model-Aware Grasp Prediction · p. 5

• The LLM infers shelf organization from the scene graph and target attributes, then ranks placement candidates anchored to existing shelf objects.It jointly considers content, packaging type, edibility, and orientation so attribute relevance can depend on the scene.Source: D. Multi-Attribute-Aware Placement… · p. 5
• Exponential rank weights prioritize higher-ranked semantic candidates while retaining lower-ranked alternatives if geometric constraints or motion planning prevent the preferred placement.The method samples 100 mesh-based poses per shelf board and associates them only with candidates on that board within 25 cm.Source: D. Multi-Attribute-Aware Placement… · p. 5
• Eligible pose-to-anchor associations maximize semantic score, with ordinal rank and anchor distance shaping proximity preference and ties favoring the higher-ranked candidate.The linear proximity term falls from 1 at the anchor to 0 at the association threshold; the resulting score lies in [0, 1].Source: D. Multi-Attribute-Aware Placement… · p. 5
• Grasp-placement pairs are sorted by a score combining geometric grasp quality, semantic placement score, and a feasibility indicator, and the motion planner tries them in descending order until a collision-free trajectory is found.The feasibility indicator excludes combinations that are not collision-free and reachable.Source: D. Multi-Attribute-Aware Placement… · p. 5

• Across three scene sets, the approach improves all mapping metrics over the single-view baseline and achieves higher semantic fit than both baselines, with success rates comparable to geometry-only planning.Source: §V · p. 5
• The mapping comparison uses label accuracy, geometric accuracy, and instance coverage against a single-view LLM baseline using the same segmenter.Source: §V · p. 5
• An additional placement-predictor attribute evaluation highlights benefits of LLM-based prediction over score-based prediction for finding semantically coherent placements.Source: §V · p. 5

• BL1 adapts EMPOWER to use the same SAM 3 segmenter, but operates on one RGB-D observation and extracts nouns while discarding descriptive adjectives.Discarding adjectives can remove semantically relevant object information for placement decisions.Source: A. Baselines · p. 5
• BL2 adapts UniP without stability prediction and samples geometrically feasible placements from the coarse map, without semantic placement decisions.The evaluation limits reorientation to packaged items with planar side faces for stable support.Source: A. Baselines · p. 5
• The Basic variant predicts labels from a basic prompt without the full approach’s detailed attributes.Source: A. Baselines · p. 5
• The simulation dataset has 60 packaged grocery and household objects, while three scene sets of 100 scenes each vary between single-row, multi-row occlusions, and changing shelf clearance.Objects are drawn with sampling biased toward same-category local groupings, and candidate poses must satisfy shelf bounds, height, and collision constraints.Source: Scene Dataset · p. 6, Figure 4 · p. 6

C. Metrics

• Semantic fit combines distance-weighted content, packaging, edibility, and orientation components from up to three closest shelf neighbors.Content similarity uses SBERT category embeddings; packaging and orientation are binary matches, while edibility compatibility accounts for unknown groups.Source: Semantic Fit (SF) · p. 6
• Success Rate (SR) is the percentage of runs in which all components execute and placement falls within 6 cm of the planned pose.The evaluation compares paired per-run metric values using a two-sided Wilcoxon signed-rank test.Source: Semantic Fit (SF) · p. 6, C. Metrics · p. 6
• SF-EQ weights all four semantic-fit components equally, while SF-LLM uses per-scene weights determined by three LLM experts.The equal-weight variant avoids a predefined attribute hierarchy; SF-LLM is intended to capture latent shelf organization.Source: Semantic Fit (SF) · p. 6
• Instance mapping is measured by canonical-label F1, 3D bounding-box IoU, and mapped-instance count relative to ground truth.F1-Label uses SBERT label-embedding similarity with a 60% threshold.Source: Instance Mapping… · p. 6

D. Experimental Results

• 24% to 63% canonical-label F1 and 29% to 80% mapped-instance coverage in multi-row scenes for OURS versus BL1.TABLE I reports that OURS also outperforms BL1 in BB-IoU across all scene sets; both methods use the same SAM 3 segmenter.Source: Quantitative Instance Mapping Results · p. 6, Table I · p. 7
• 20% to 95% SR, 34% to 63% SF-EQ, and 29% to 58% SF-LLM in multi-row scenes for OURS versus BL1.Across scenes, TABLE II reports the highest SF-EQ and SF-LLM for OURS while retaining success comparable to geometry-only BL2.Source: Table II · p. 7, Quantitative Placement Results · p. 7
• OURS improves F1 over attribute-free Basic across scenes while achieving similar BB-IoU and mapping nearly the same number of instances.The comparison supports a benefit of semantic attributes for cross-view association and label consistency.Source: Quantitative Instance Mapping Results · p. 7
• OURS exceeds the equally weighted Score predictor by 4, 5, 4, and 10 percentage points in mean SF-LLM for content, packaging, edibility, and orientation, respectively.Weights were determined using four sets of ten scenes and evaluated on 20 unseen scene sets.Source: Quantitative Placement Results · p. 7, Table III · p. 7
• A real-world rollout places a previously unseen object alongside shelf items with similar content and the same packaging type.The example integrates multi-view perception, scene-graph reasoning, and geometric feasibility assessment.Source: Qualitative Results · p. 7, Figure 5 · p. 8

VI. CONCLUSION

• The pipeline grounds labels and attributes across views in a 3D semantic scene graph, then uses LLM placement reasoning followed by geometric and kinematic feasibility checks.The conclusion reports improved instance mapping and higher semantic fit over single-view language grounding and geometry-only planning.Source: §VI · p. 8