PaperlayerView exact revision on arXiv

Source-linked AI summary

CLON: Cue-Calibrated Linguistic Object Onboarding for Zero-Shot 6D Pose Front-Ends

Seojin Ji, Yoojin Kwon, Hyung-Sin Kim

arXiv:2609.04784v1cs.CV

TL;DR

Zero-shot 6D pose pipelines remain constrained by front-ends that must retain partially visible targets while rejecting plausible distractors. CLON builds linguistic memories and fixed object-set cue weights from rendered templates for target-aware proposal generation and calibrated scoring. Across the reported benchmark and downstream evaluations, it improves front-end quality and pose performance over existing front-ends.

  • Problem

    Zero-shot 6D pose front-ends must preserve partially visible true positives while rejecting semantically plausible distractors under clutter and occlusion.

  • Method

    CLON converts rendered templates into linguistic memories for SAM 3 proposal generation and fixed object-set cue weights for semantic, appearance, and geometric scoring.

  • Results

    CLON improves front-end proposal quality and downstream pose AR across BOP-Classic-Core datasets and three downstream pose solvers.

  • Takeaways & Limitations

    Target-aware linguistic proposals and calibrated cue scoring improve zero-shot 6D pose front-ends in cluttered and occluded settings.

  • Takeaways & Limitations

    Linguistic prompts can remain ambiguous for visually similar objects or poorly named industrial parts, while entropy-based calibration may miss shape, symmetry, and appearance ambiguities.

Abstract

from arXiv · show

Zero-shot 6D pose estimation pipelines increasingly rely on strong downstream pose solvers, but their performance is often limited by the front-end: object proposals must preserve partially visible true positives while rejecting semantically plausible distractors. We introduce Cue-Calibrated Linguistic Object Onboarding (CLON), a front-end requiring no task-specific training for new objects. Given rendered templates of the onboarded object set, CLON constructs a linguistic semantic memory for top-down proposal generation and object-set cue weights for calibrated proposal scoring. The linguistic memory guides SAM 3 toward high-recall proposals for onboarded objects, while cue weights are computed once from the onboarded object set before scene inference and kept fixed during online scoring. On seven BOP-Classic-Core datasets, CLON improves detection AP by 8.1 percentage points (pp), segmentation AP by 6.2 pp, and downstream 6D pose AR by up to 4.1 pp over CNOS and SAM-6D front-ends.

1 Introduction

CLON addresses brittle zero-shot 6D pose front-ends by turning object onboarding into target-aware proposal generation and calibrated object scoring. It combines linguistic memories from rendered templates with fixed object-set cue weights to improve proposal quality under clutter and occlusion.

  • Motivation: Front-end failures can prevent strong downstream pose solvers from recovering missed, fragmented, or misidentified target objects.The front-end must preserve partially visible true positives while rejecting semantically plausible distractors.
  • Approach: CLON converts rendered templates into compact linguistic object memories for target-aware proposal generation without task-specific training for new objects.The memories guide SAM 3 toward regions likely to correspond to the onboarded object set.
  • Design rationale: CLON combines linguistic object onboarding with object-set-aware proposal scoring rather than relying only on generic objectness or post-hoc top-K pruning.This design changes proposal generation itself while assigning object identity through calibrated proposal-object scoring.
  • Approach: CLON computes fixed semantic, appearance, and geometric cue weights from template statistics before scene inference.During online inference, proposals remain identity-agnostic and are scored against onboarded objects using the calibrated cues.
  • Results: Target-aware linguistic proposals and cue calibration improve zero-shot 6D pose pipelines under clutter and occlusion.The front-end is evaluated across proposal quality, downstream pose estimation, and real robot-observation scenes.

2 Related Work

Related work establishes foundation-model proposals and template matching as effective for unseen objects, but CLON focuses onboarding on front-end object-set memory construction. It separates high-recall semantic proposal generation from calibrated template-based identity assignment.

  • Promptable segmentation: Promptable segmentation provides class-agnostic masks, while later systems add language-grounded search for noun phrases or image exemplars.These systems support semantic object search but do not by themselves establish onboarded-instance recognition.
  • Promptable segmentation: Semantic phrases can retrieve plausible distractors that do not match the onboarded object model or reference instance.CLON therefore uses SAM 3 as a high-recall proposal generator and resolves identity through calibrated template-based scoring.
  • Novel-object pose estimation: CNOS, SAM-6D, and related methods combine foundation-model proposals with rendered-template matching for unseen-object detection, segmentation, or pose estimation.These methods demonstrate the effectiveness of proposal generation and template matching without task-specific training.
  • CLON positioning: CLON differs by converting object templates into linguistic memories and computing cue weights before scene inference.The design targets generic proposal generation and proposal scoring as separate front-end limitations.
  • Novel-object pose estimation: Strong pose estimators still depend on sufficient localized object evidence because missed, fragmented, or misidentified candidates can leave them without a valid hypothesis.CLON addresses this upstream front-end dependency before downstream pose inference.

3 Method

CLON separates offline object-set onboarding from online scene inference, using template-derived linguistic memories to guide SAM 3 proposals and fixed cue weights to score proposal–object matches.

  • Object-Set Onboarding: CLON represents each onboarded object with rendered templates and constructs an object-set memory entirely before scene inference.The memory contains linguistic object information and cue weights used later in proposal generation and scoring.
  • Linguistic Semantic Memory: Linguistic memories are generated from subsampled template views using deterministic VLM prompting for persistent noun phrases describing object properties.The prompt can include optional object metadata and targets color, shape, material, and distinctive parts.
  • Object-Set Cue Weights: Cue weights adapt semantic, appearance, and geometric evidence to the onboarded object set’s visual variation and inter-object discriminability.High appearance entropy activates semantic-only scoring, while low entropy retains appearance and geometry as compensatory cues.
  • Semantic Proposal Generation: CLON combines generic and linguistic prompts to guide SAM 3 toward target-relevant regions, then pools masks with IoU-based non-maximum suppression.Generic prompts preserve recall under unusual viewpoints, weak linguistic evidence, or severe occlusion.
  • Calibrated Proposal-Object Scoring: Proposals remain identity-agnostic until every proposal is scored against every onboarded object using semantic, appearance, and geometric cues.Identity assignment uses the highest final score, and matched hypotheses above the threshold proceed to downstream 6D pose estimation without an additional top-K cap.

4 Experiments

CLON is evaluated across seven BOP datasets, multiple pose solvers, prompt configurations, and real-world robot scenes. It improves front-end proposal quality and downstream pose estimation, with ablations attributing gains to linguistic memory and calibrated cue weighting.

  • Benchmark and setup: CLON is evaluated on seven BOP challenge datasets containing 132 object instances under occlusion, textureless surfaces, illumination changes, and visual similarity.Detection and segmentation use AP, while downstream pose estimation uses AR under the standard BOP protocol.
  • Benchmark and setup: CLON is compared with CNOS and SAM-6D front-ends paired with GigaPose, SAM-6D, and FoundationPose under fixed global hyperparameters without per-dataset tuning.The evaluation isolates whether front-end improvements transfer to different downstream pose solvers.
  • BOP benchmark results: +8.1 pp detection AP and +6.2 pp segmentation AP over the strongest baseline yield APdet 0.551 and APseg 0.543 for CLON.Linguistic object memories provide target-aware SAM 3 anchors that suppress semantically plausible distractors.
  • BOP benchmark results: +4.1, +3.0, and +1.4 pp ARmean gains are obtained with GigaPose, SAM-6D, and FoundationPose, respectively.CLON reaches ARmean values of 0.634, 0.726, and 0.748 with those solvers; gains are not uniform for every dataset.
  • Ablation study: Linguistic memory raises APmean to 0.540, while cue calibration further raises it to 0.551; semantic-only scoring drops to 0.532.The ablation indicates that target-aware proposal generation provides the dominant improvement, while appearance and geometry remain useful when calibrated.
  • Ablation study: The full CLON prompt set reaches 0.551 APmean, outperforming a generic object prompt at 0.410 and the SAM-6D baseline at 0.448.Combining structural and color descriptors reaches 0.544 APmean, whereas multiple compact, view-consistent phrases perform best.
  • Real-world robot evaluation: CLON achieves 0.811 APmean, 0.849 ARmean, and a 36.0% grasp rate in 100 cluttered robot scenes, twice the strongest baseline grasp rate.The scenes include robot-arm occlusion, and each object is tested in 20 grasping trials.

5 Conclusion

CLON converts rendered templates into linguistic object memories for target-aware proposal generation and computes object-set cue weights before scene inference. Across BOP-Classic-Core datasets and downstream pose solvers, it improves proposal quality and pose estimation, while remaining limited by linguistic ambiguity, calibration gaps, and runtime scaling.

  • 5 Conclusion: CLON converts rendered templates into linguistic object memories and computes object-set cue weights before scene inference.The memories support target-aware proposal generation, while the weights calibrate semantic, appearance, and geometric scoring.
  • 5 Conclusion: CLON’s linguistic prompts can remain ambiguous for visually similar objects or poorly named industrial parts.The entropy-based cue calibration may also miss ambiguities caused by shape similarity, symmetry, or visually similar industrial parts.
  • 5 Conclusion: CLON is not optimized for worst-case runtime because SAM 3 processes multiple prompts and proposal-object scoring scales with the number of onboarded objects.The paper identifies prompt-efficient proposal generation and stronger object-set reliability estimation as future directions.
  • 5 Conclusion: Qwen-2.5-VL generates exactly four concise noun-phrase descriptors per object using unified prompt rules for identity, spatial visibility, and physical boundaries.The rules include spatial keywords for partial visibility and explicit descriptions of volumetric extent.
  • 5 Conclusion: Generic noun phrases provide recall-oriented anchors when unusual viewpoints or severe occlusion hide instance-specific visual cues.These generic anchors supplement the more specific descriptors in the linguistic memory.

B Implementation Details of Object-Set Cues

This section specifies the hyperparameters and implementation details for CLON’s semantic, appearance, and geometric cues and its weight distribution mechanism.

  • B Implementation Details of Object-Set Cues: The implementation details cover semantic, appearance, and geometric cues together with the relative-weight distribution mechanism.These details were omitted from the main text.

B.1 Detailed Computation of Discriminability Metrics

CLON computes object-set discriminability from semantic, appearance, and geometric statistics, then derives relative cue weights and applies texture and occlusion-specific rules. Across seven BOP datasets, the resulting weights are dataset-dependent.

  • B.1 Detailed Computation of Discriminability Metrics: DINOv3 class tokens averaged across template views form canonical semantic prototypes, with nearest-neighbor separation margins measuring semantic distinctiveness.The margin establishes how distinct each object’s global concept is from the rest of the dataset.
  • B.1 Detailed Computation of Discriminability Metrics: Appearance discriminability measures surface-pattern richness using joint Hue-Saturation entropy after masking template backgrounds.The histogram uses 180 Hue bins and 256 Saturation bins, totaling 46,080 bins.
  • B.1 Detailed Computation of Discriminability Metrics: Geometric discriminability measures spatial distinctiveness through maximum 3D IoU after aligning object bounding-box centers.This evaluates whether structural scale and aspect ratio provide discriminative cues without visual features.
  • B.1 Detailed Computation of Discriminability Metrics: Appearance entropy above τapp assigns weights (wsem, wapp, wgeo) = (1, 0, 0), while low entropy sets wapp directly to eapp with τapp = 0.25.This gating strategy specifically handles texture-rich versus textureless object sets.
  • B.1 Detailed Computation of Discriminability Metrics: Under heavy occlusion, semantic features receive three times the geometric priority through λsem = 3.The rule reflects the stated assumption that semantic features are more robust than distorted apparent 3D bounding profiles.
  • B.1 Detailed Computation of Discriminability Metrics: Table 6 reports computed discriminability metrics and relative cue weights across the seven BOP datasets.The final weights are determined from these object-set statistics and the stated strategy.

C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring

Linguistic semantic memory improves proposal anchoring by producing fewer noisy candidates while retaining recall, and its benefits transfer across proposal generators and are strongest with the full memory.

  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: The conditioned proposal generator reduces noisy candidates before visual verification, lowering computational overhead and false-positive risk.SAM-6D averages 174 masks on LM-O and 108 on ICBIN under dense grid prompting.
  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: The full linguistic memory achieves 0.551 APmean and improves performance substantially on textureless datasets such as T-LESS and ITODD.Different datasets benefit from different linguistic cues, but the full memory performs best overall.
  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: LM improves Grounding DINO + SAM from 0.310 to 0.446 APmean (+13.6 pp).This shows that the linguistic-memory benefit transfers beyond SAM 3.
  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: LM improves SAM 3 from 0.410 to 0.551 APmean (+14.1 pp), while SAM 3 remains the strongest tested proposal generator.The comparison separates the contribution of the linguistic memory from the proposal generator.
  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: Qualitative comparisons show SAM-6D producing dense, overlapping, and background-focused boxes, whereas CLON improves proposal signal-to-noise ratio.The comparison is presented through detection and segmentation proposal examples.
  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: Optimal semantic, appearance, and geometric weight ratios vary drastically across datasets, supporting object-set-specific calibration rather than fixed weights.The grid search evaluates how each dataset responds to different cue-prioritization strategies.
  • C.1 Impact of Linguistic Semantic Memory for Proposal Anchoring: CLON assigns higher appearance scores to small, occluded proposals than SAM-6D across varying occlusion levels.Figure 10 directly compares appearance-scoring robustness under occlusion.

D.1 Robustness across Visible Fractions

CLON maintains stronger proposal correctness across visible-fraction ranges, including severe occlusion and high visibility. Its largest advantages appear when objects are difficult to localize or baselines retain false positives.

  • Severe Occlusion: At 0%–33% visibility, CLON achieves at least twice the baseline accuracy, while baseline methods remain below 5%.Across the full visible-fraction range, CLON consistently outperforms existing baselines.
  • Across Visibility Ranges: At visibility above half, CLON outstrips other approaches by 16.0% to 47.7%.The margin persists beyond the most severely occluded cases.
  • Severe Occlusion: 25.3%–39.3% recall on ICBIN at 10%–30% visibility contrasts with baseline accuracies below 10%.The comparison illustrates CLON’s robustness when only a small portion of the object is visible.
  • Severe Occlusion: 90.5% accuracy on YCB-V at 30%–40% visibility compares with 4.8% for CNOS and SAM-6D.The result is attributed to dynamically balancing semantic, appearance, and geometric scores under severe occlusion.
  • High Visibility: In the 90%–100% visibility bin, CLON reaches 98.9% on LM-O, 99.3% on ICBIN, and 98.8% on YCB-V.Baselines remain bounded between 60% and 80% in some highly visible settings because of unsuppressed false positives.

D.2 Upper Bound Analysis with Ground Truth

Ground-truth proposals still improve pose-estimation performance over CLON proposals, showing that front-end localization remains a bottleneck. CLON nevertheless reduces proposal counts and can lower total latency on several datasets, with one-time onboarding overhead.

  • Upper Bound Analysis: Ground-truth proposals improve mean AR′ by 6.9–10.7 pp over CLON across three pose solvers.The largest residual gap occurs on T-LESS, where texture-less and highly similar objects challenge instance localization.
  • Upper Bound Analysis: On T-LESS, the ground-truth-versus-CLON gap reaches 16.6–33.0 pp.This dataset-specific gap identifies accurate instance localization as a remaining limitation.
  • Efficiency: CLON reduces proposals from 67.3 to 13.6 per image on average compared with SAM-6D.The reduction decreases subsequent pose-estimation time and yields lower total latency on LM-O, TUD-L, and IC-BIN.
  • Efficiency: Offline onboarding takes 73.7 s on average versus 34.9 s for the baseline.The stored linguistic memory has negligible memory and computational overhead during online inference.

D.4 Details on Robot Deployment

The deployment uses a fixed top-down RGB camera and a server-side CLON pipeline to localize targets for robotic pick-and-place. In real-world trials, CLON suppresses background false positives and isolates target instances more reliably than the baselines.

  • Hardware Configuration: A Logitech C920E webcam captures raw RGB bird’s-eye views without depth inputs in the SO-101 manipulation setup.The camera is mounted directly above the workspace in a fixed top-down configuration.
  • Control Pipeline: The server receives each RGB frame, generates 2D proposals, and converts the detected bounding-box center into robot coordinates for inverse-kinematics control.The resulting trajectory is open-loop and proceeds down to the surface for pick-and-place.
  • Quantitative Evaluation: The quantitative deployment evaluation covers five SenseShift6D objects with 20 trials per object, totaling 100 robot grasping trials.Ground-truth bounding boxes were manually annotated for the evaluated RGB frames.
  • Qualitative Results: CLON cleanly isolates the sandwich target, whereas CNOS and SAM-6D frequently localize background structures such as camera tripod legs.The comparison attributes the baseline errors to severe false positives caused by environmental noise and identity dilution.
  • Memory Construction: The linguistic semantic memory uses 10 template views from 42 available views per object and generates exactly four noun phrases per object.These settings describe the offline memory construction used for deployment.
Loading 2609.04784v1…