PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Scan-Derived Class Prototypes Add? Disentangling Supervision, Prototype Content and Query Protocol in Recognition over Frozen Foundation Features

Chenxi Tao, Hong-In Won, Seung-Kyum Choi

arXiv:2609.04381v2cs.CVcs.AIcs.RO

TL;DR

A scan provides both labeled images and a geometric reference, so accuracy gains alone do not identify which contributes. This study audits scan-derived prototypes as a supervised head's fixed output layer; their largest fused-accuracy advantage over either control is one percentage point in observed arm means.

  • Problem

    Because a scan supplies both object-labeled images and a geometric reference, improvement over image-prototype matching alone does not identify which product contributes.

    Source: §1 · p. 2

  • Method

    The audit uses scan-derived class prototypes as a fixed output layer in a recognizer with a frozen image encoder and a mean image-prototype branch.

    Source: §3 · p. 5

  • Results

    One percentage point was the largest fused-accuracy advantage of real prototypes over either control in observed arm means.

    Source: §6 · p. 24

  • Takeaways & Limitations

    Improved accuracy over image-prototype matching alone cannot identify a contribution from scanned shape.

    Source: §5 · p. 23

    Whether scanned geometry contributes independently remains an open question; its contribution was not separately measured.

    Source: §5 · p. 24

Abstract

from arXiv · show

A scan supplies labeled images and a geometric reference. We separate their contributions in a recognizer whose scan-derived prototype matrix acts as a supervised head's fixed output layer. On T-LESS, HOPE and 18 self-collected industrial parts, we test real, random and exactly permuted prototypes, matched geometry-free classifiers, stronger appearance rules and paired background protocols. Across DINOv2-giant and MetaCLIP-H with real-background queries, the largest fused-accuracy advantage of the real prototypes over either control is one percentage point; larger differences favor controls, by up to 2.8 points in arm means. On HOPE with DINOv2-giant the head alone is 2.8 points above exact permutations (95% interval: 0.8-4.7); this advantage does not reach fusion and is not observed on MetaCLIP-H. On DINOv2-giant, matched logistic regression comes within 0.5 points of fusion on T-LESS and exceeds it on HOPE and the self-collected parts. Against white cutouts, real HOPE query backgrounds lower image-prototype accuracy by 43 points on DINOv2-giant and 13 on MetaCLIP-H. The audit separates prototype content, label supervision and query protocol.

1. Introduction

A scan provides both object-labeled images and a geometric reference, so better accuracy alone does not identify which product contributes. The paper audits scan-derived class prototypes against matched supervision, arbitrary assignments, appearance rules, and query protocols.

1. Introduction

  • A scan supplies object-labeled images and a geometric reference, whose separate contributions matter when a reconstruction serves as the reference and CAD models are unavailable.Improvement over image-prototype matching alone does not identify which scan product contributes.Source: §1 · p. 2
  • The recognizer maps frozen image features to scan-derived class prototypes, which serve as a fixed output layer while cross-entropy trains the mapping with onboarding labels.Queries are RGB crops, with geometry entering through the prototype arrangement.Source: §1 · p. 2
  • The audit tests real, random, and class-permuted prototypes, geometry-free classifiers, global and patch-level appearance matching, and paired white-cutout and real-background queries.It covers T-LESS, HOPE, and self-collected industrial parts using DINOv2-giant and MetaCLIP-H.Source: §1 · p. 2
  • The paper proposes a reproducible audit of whether the original shape-to-class assignment adds predictive value beyond matched supervision and arbitrary prototypes.It also applies the assignment intervention to a published semantic-prototype recognizer trained end to end.Source: §1 · pp. 2–3
  • Most fused gains persist without the original prototype assignment, while a head-level advantage remains conditionally on HOPE with DINOv2-giant.The audit separates head and fused outcomes to assess the prototype matrix under matched conditions.Source: §1 · p. 3

2. Related Work

Related work spans geometric references, scan-derived descriptors, frozen image representations, and fixed prototype outputs. This paper distinguishes reference uses from per-class prototype matrices fitted through a supervised head, and tests them under matched controls and query protocols.

2. Related Work

  • Prior work includes geometric references for recognition and pose estimation, while this study focuses on per-class prototype matrices fitted through a supervised head.The distinction identifies which intervention can test the role of geometry.Source: §2 · p. 3
  • ULIP and OpenShape learn joint image, text, and point-cloud embeddings for cross-modal retrieval, unlike a per-class matrix fitted through a supervised head.Source: §2 · p. 3
  • Scan representations include reconstructed objects, point-cloud descriptors, global shape signatures, and learned point encoders; image representations include frozen self-supervised and image-text features.The study compares mean image prototypes with stronger appearance rules while keeping the feature extractor frozen.Source: §2 · pp. 3–4
  • Fixed random output layers and predefined semantic prototypes motivate controls that test whether scan-derived targets supply shape evidence or function as class codes.The cited precedents include fixed output layers and semantic prototype structure.Source: §2 · p. 4
  • The paired query analysis isolates background replacement by changing pixels outside the visible-object mask while retaining the crop and target identity.This connects background dependence and environmental shifts to a specific protocol change.Source: §2 · p. 4

3. System and audit design

The system and audit design separates the image-prototype branch from a trained head with a fixed, scan-derived geometric output layer, varying prototype content, class assignment and query backgrounds across T-LESS, HOPE and FGIP where paired protocols are available. It also specifies matched geometry-free controls and appearance-based references, including patch bag-of-words, and their feature, filtering and calibration protocols.

3. System and audit design

  • A frozen-encoder recognizer combines mean image prototypes with a trained head scored against fixed scan-derived geometry prototypes, with controls applied at distinct stages.The head is the only trained module; the audited fusion weights the geometry score by 0.35.Source: §3 · p. 5, Figure 1 · p. 6, §3.2 · p. 9

3.2. Class descriptors and prediction

  • Centering and scaling normalize translation and scale, but six pose-dependent terms and sampling-induced changes to G mean the implemented 45-dimensional descriptor is not exactly SE(3)-invariant.Source: §3.2 · pp. 7–9

3.3. Interventions and statistical units

  • Random prototypes test whether object-derived rows are needed but also change conditioning, whereas exact row permutations isolate class assignment while preserving the saved matrix.Real-versus-permuted comparisons use five-seed 95% t-intervals; real-versus-random intervals use five random matrices and combine between-matrix and real-seed variance, with no equivalence test.Source: §3.3 · pp. 9–10
  • Real-prototype versus control differences are reported with intervals, and “largest advantage” refers to observed arm-level differences.The passage does not specify the interval type or the statistical unit beyond arm-level comparisons.Source: §3.3 · p. 10
  • Matched logistic-regression references use fixed C = 1 and the softmax head’s optimization subset and checkpoint rule; full-budget references use all onboarding frames.Logistic regression is deterministic for a fixed subset, whereas softmax results use five seeds; full-budget softmax uses its final epoch.Source: §3.3 · pp. 10–11
  • Five global appearance rules use mean prototypes, best-template or best-five matching, eight k-means sub-prototypes, or temperature-weighted soft voting.Patch bag-of-words compares TF–IDF histograms using cosine similarity, following FoundPose’s retrieval stage (Örnek et al., 2024) without correspondence estimation or PnP.Source: §3.3 · p. 11
  • Patch bag-of-words uses a 2,048-word vocabulary fitted for 25 k-means iterations, with at most 40 onboarding frames per class and 200,000 tokens.It tests final-layer and hidden-state-30 features, and best-template or best-five matching over all onboarding frames.Source: §3.3 · p. 11
  • No semantic mask is applied; 68% of real-background HOPE queries retain all patches, and the appearance rules cost more storage and query computation than a mean prototype.Each rule calibrates its fusion weight on held-out onboarding frames over [0,2].Source: §3.3 · p. 11

4. Attribution results

This section attributes recognition outcomes to prototype content and geometric-head supervision by comparing real prototypes with controls and geometry-free references, while testing the effects of query backgrounds, appearance rules, reconstruction and descriptor choices, shape matching, and semantic prototype assignments. The reported results show little consistent fused-accuracy advantage for real prototypes, and note a second-backbone audit without reporting its outcomes here.

4.1. Prototype content and class assignment

  • On HOPE, the real head beats exact permutations by 2.8 points, but its fused advantage stays within −0.3 to +0.3 points across weights; T-LESS and FGIP show no detected head-level difference.The HOPE head advantage does not reach fusion; on T-LESS and FGIP, paired intervals include zero.Source: §4.1 · pp. 12–13, Table 5 · p. 12, Figure 3 · p. 13

4.2. Matched label supervision

  • Matched logistic regression comes within 0.5 points of fixed fusion on T-LESS and exceeds it on HOPE and FGIP; the HOPE head advantage over matched references is 2.6 to 3.8 points.The softmax reference trails logistic regression on T-LESS and HOPE but leads on FGIP.Source: Table 6 · p. 14, §4.2 · p. 14

4.4. Paired query backgrounds

  • On FGIP, white cutouts change mean-prototype and best-template matching by −33 and +33 correct queries, respectively, while the audited system changes by less than two points either way.The cutout changes the task, and its effect on FGIP depends on the appearance rule.Source: §4.4 · pp. 17–18, Figure 5 · p. 18
  • Real HOPE backgrounds reduce DINOv2-giant mean-prototype accuracy by 43.3 points and MetaCLIP-H by 13.3; patch-best-five fusion shifts from +2.9 on white to −2.1 on real.The paired protocols score the same 915 HOPE query boxes; background, neighboring objects, and occluders are not separated.Source: Table 8 · p. 17, §4.4 · p. 17, Figure 5 · p. 18

4.5. A second frozen backbone

  • The second-backbone audit uses MetaCLIP-H projected image features while keeping the saved real, random, and permuted prototype matrices unchanged.The supplied passages describe this evaluation design but do not provide its results.Source: §4.5 · p. 18, Table 9 · p. 19
  • Real prototypes improved MetaCLIP-H fusion over either control by at most one point, and every random or permuted HOPE fused run exceeded every real run.The HOPE fused deficit persisted when matching head seed 0; the DINOv2-giant HOPE head advantage was not reproduced on MetaCLIP-H.Source: §4.5 · p. 20
  • Matched logistic regression matched or exceeded the geometry head on all three datasets, while the HOPE softmax head exceeded it by 5.2 points.MetaCLIP-H image-only accuracy on real-background HOPE queries was 0.745, versus 0.774 for the DINOv2-giant patch matcher.Source: §4.5 · p. 20

4.6. Reconstruction source, descriptor, image encoder and rendered pixels

  • Reconstruction sources agree within 1.6 points at the head and 0.5 points fused; PointNet++ and frozen OpenShape stay within descriptor seed variability, and random or permuted prototypes within 0.5 points of real fused T-LESS accuracy.Source: §4.6 · p. 21

4.7. Direct shape matching

  • In the post hoc pilot, geometry matching with ground-truth masks, 24 PCA initializations, and ICP achieved object-balanced two-way accuracy of 0.486–0.503, versus 0.671 for raw-crop appearance matching.Pose-assisted arms initialized both candidates from the true identity’s pose and reached 0.774–0.833 accuracy.Source: §4.7 · p. 21

4.8. External semantic-prototype experiment

  • On CIFAR-100, HPN semantic assignments exceeded derangements by 7.1 test-accuracy points (95% t-interval [+6.4,+7.8]); all 25 permutation cells were positive.The training-set difference was +3.3 points, so the arms differed in fit as well as generalization.Source: Table 12 · p. 22, §4.8 · p. 23
  • The HPN experiment compared prototype assignments only and included no supervised classifier without prototypes.Source: §4.8 · p. 23

5. Discussion

The controls show that gains over image-prototype matching alone do not identify a contribution from scanned shape. Class assignment can affect head learning, but its value to the audited recognizer must be assessed at the fused output.

5. Discussion

  • Gains over image-prototype matching alone do not identify a contribution from scanned shape.The controls are needed to distinguish matching gains from effects attributable to the scan-derived reference.Source: §5 · p. 23
  • Random and permuted prototype matrices retain most of the fused benefit.The HOPE head-level residual does not establish a corresponding fused-output benefit.Source: §5 · p. 23
  • Class assignment can affect head learning, but its value to the audited recognizer must be assessed at the fused output.Under paired HOPE query protocols, the head contribution reverses, tying its value to appearance representation and query processing.Source: §5 · p. 23
  • The audit compares real matrices with generated and exactly permuted matrices, geometry-free classifiers, a second backbone, and stronger appearance matching.It also pairs query protocols image by image and reports paired or matrix-level differences with intervals.Source: §5 · pp. 23–24
  • Open questions include whether scanned geometry helps when queries have 3D measurements and estimated poses, and whether HOPE’s head advantage appears on MetaCLIP-H with more training seeds.The audit also leaves open whether the 1.7–2.7-point real-versus-random head difference in patch fusion holds across several seeds per random matrix.Source: §5 · p. 24
  • The evidence covers 74 objects in three domains, two image backbones for the controls, and a closed-set few-shot setting.Segmented onboarding images and real-background queries create a domain gap whose contribution was not separately measured.Source: §5 · p. 24

6. Conclusion

The conclusion frames the audit as separating scan-derived prototype contributions from label supervision, appearance strength, and query protocol. Across the tested datasets and backbones, real prototypes provide at most a one-point fused-accuracy advantage over either control, while a head-level advantage is conditional.

6. Conclusion

  • Across three datasets and two backbones, real prototypes’ largest fused-accuracy advantage over either control is one percentage point.The conclusion also notes larger differences favor controls, by up to 2.8 points in arm means.Source: §6 · p. 24
  • On HOPE with DINOv2-giant, the head-level advantage over exact permutations does not improve audited fusion.The conclusion distinguishes this conditional head result from fusion and reports that it is not observed on MetaCLIP-H.Source: §6 · p. 25
  • On CIFAR-100, the published class assignment is seven points above derangements of the same prototypes under one training budget.The comparison applies the assignment intervention to a published semantic-prototype recognizer.Source: §6 · p. 25
  • Matched supervision and paired query protocols provide comparisons for interpreting the audit’s outcomes.The audit tests whether gains depend on the original shape-to-class assignment and improve on stronger appearance rules.Source: §6 · p. 25

CRediT authorship contribution statement

The contribution statement assigns conceptualization, methods, software, analysis, investigation, data curation, writing, visualization, supervision, and funding roles across the listed authors.

CRediT authorship contribution statement

  • Chenxi Tao is credited with conceptualization, methodology, software, formal analysis, investigation, data curation, drafting and revising, and visualization.The statement also credits Hong-In Won with funding acquisition and writing–review and editing.Source: CRediT authorship contribution statement · p. 25
  • Seung-Kyum Choi is credited with supervision, funding acquisition, conceptualization, and writing–review and editing.The statement lists Hong-In Won for funding acquisition and writing–review and editing.Source: CRediT authorship contribution statement · p. 25

Declaration of competing interest

The authors report no known competing financial interests or personal relationships that could have influenced the reported work.

Declaration of competing interest

Data availability

T-LESS and HOPE are available through the BOP benchmark, and CIFAR-100 is available from its cited website. The reproducibility package is available from the corresponding author on request.

Data availability

  • T-LESS and HOPE are available through the BOP benchmark, while CIFAR-100 is available from its cited website.Source: Data availability · p. 25
  • The reproducibility package includes source-linked numerical records, control-generation and run scripts, permutation matrices and indices, matched splits, and per-query predictions.The package and FGIP crops, reconstructions, prototype matrices, and analysis code are available from the corresponding author on request.Source: Data availability · pp. 25–26

Funding

The study received support from two Korean government-affiliated programs, and the funders had no involvement in the research or publication decision.

Funding

  • Funding came through the Global Industrial Technology Cooperation Center Program and the Manufacturing AI Key-Tech Program.The first was supervised by KIAT; the second by KITECH.Source: Funding · p. 26
  • The Global Industrial Technology Cooperation Center Program support was provided through MOTIE.Source: Funding · p. 26
  • The funders had no role in study design, data collection, analysis, interpretation, writing, or the decision to submit.Source: Funding · p. 26

Appendix A. Prototype-matrix statistics and fusion weight

Appendix A examines real, random, and permuted prototype matrices and fusion weights; Appendix B probes reconstruction sources, descriptors, image encoders, and rendered pixels.

Appendix A. Prototype-matrix statistics and fusion weight

  • Real matrices have mean absolute row cosine/effective rank of (0.249, 16.4), (0.335, 14.0), and (0.415, 10.0) on T-LESS, HOPE, and FGIP.Random-matrix averages are (0.311, 14.0) for the 28-class sets and (0.313, 10.8) for FGIP; permutations preserve the real-matrix statistics.Source: Appendix A · p. 26
  • At λ = 1, real fusion scores 0.598/0.438/0.891 versus permuted 0.594/0.435/0.889 on T-LESS/HOPE/FGIP; the score still includes the image branch.The fusion-weight grid is scored on test labels as a diagnostic.Source: Figure A.1 · p. 27, Appendix A · p. 27
  • The HOPE head classifies all 560 held-out frames correctly, reaching the onboarding-calibration grid’s upper bound.Source: Appendix A · p. 27, Figure A.1 · p. 27

Appendix B. Reconstruction source, descriptor, image encoder and rendered pixels

  • The fusion spread is 0.5 points, compared with 1.6 points at the head.Source: Appendix B · p. 28
  • On T-LESS, DINOv2-giant, SigLIP2-g, and OpenCLIP-bigG do not track image-only strength.The head alone trails the image prototype for OpenCLIP-H, OpenCLIP-bigG, and SigLIP-so400m; AIMv2 is omitted after chance-level accuracy in this pipeline.Source: Appendix B · p. 29
  • Using frozen DINOv2-giant on the same 6,395 queries, replacement-run mean image prototypes from 3DGS renders score 0.392, while addition-run prototypes from crops and renders score 0.554.Source: Appendix B · p. 29
  • The CAD-lighting probe’s largest reported change is 2.5 points, from 0.947 to 0.922 with ten views under extreme illumination.The probe uses 28 objects and 24 rendered views per object, with matched, shifted, or extreme illumination and 1, 5, or 10 support views.Source: Appendix B · p. 29

Appendix C. Direct shape matching: depth, pose and silhouette diagnostics

Appendix C tests direct depth, pose, and silhouette matching; it reports both performance comparisons and limits on what the diagnostics isolate.

Appendix C. Direct shape matching: depth, pose and silhouette diagnostics

  • The measured-depth pilot records 225 correct queries for the scan and 224 for CAD, while the CAD-box control scores 0.644 with ground-truth pose.It covers 273 development queries across six T-LESS pairs; 15 unusable-depth queries count as wrong.Source: Appendix C · p. 29, Table C.1 · p. 30
  • Under identity placement, the first-flip median rotation is 3°, and 20 of 87 usable queries are incorrect; shared-pose placement has 16 errors.The analysis perturbs only the true candidate and records the first error along the tested translation and rotation grids.Source: Appendix C · p. 30
  • The depth diagnostic does not separate pose sensitivity, visibility, initial errors, or scorer behavior.Among incorrect identity-placement CAD cases, median valid-depth-pixel count is 5.5, versus a first quartile of 18 among correct cases.Source: Appendix C · pp. 30–31
  • On the confusable FGIP screw pair under white cutouts, silhouette overlap gets 176/215 queries correct versus 167 for appearance matching; overall, it gets 1,294/1,927 versus 1,769.Source: Appendix C · p. 31

cess

  • The authors report using OpenAI Codex and Anthropic Claude to organize, draft, and edit the manuscript, then reviewing and editing the content.Source: cess · p. 31
Loading 2609.04381v2…