PaperlayerView exact revision on arXiv

Source-linked AI summary

ShelfChange3D: Object-Level 3D Change Detection for Retail Shelf Monitoring

Lingyi Zhou, Yunke Wang, Mengyu Zheng, Wenbo Wang, Zijian Wang, Chang Xu

arXiv:2610.01283v1cs.CV

TL;DR

Existing out-of-stock methods mainly operate in image space and lack metric 3D localization of changed products for robotic systems. ShelfChange3D provides 145K synthetic and 5K real-world paired RGB-D observations, while ChangeBox predicts object-level 3D change boxes and refines them using depth and a gravity prior. ChangeBox detects shelf changes across viewpoints and transfers synthetically pretrained features to real data; on the real test set, refinement raises mAP0.25 from 92.9 to 97.9 after fine-tuning.

  • Problem

    Existing out-of-stock methods mainly operate in image space and lack metric 3D localization of changed products for robotic systems.

    Source: §I · p. 1

  • Method

    ShelfChange3D provides 145K synthetic and 5K real-world paired RGB-D observations, while ChangeBox predicts object-level 3D change boxes and refines them using depth and a gravity prior.

    Source: §I · p. 1

  • Results

    ChangeBox detects shelf changes across viewpoints and transfers synthetically pretrained features to real data; on the real test set, refinement raises mAP0.25 from 92.9 to 97.9 after fine-tuning.

    Source: §VI · p. 7, C. Sim2Real on ShelfChange3D-5K · p. 7

  • Takeaways & Limitations

    ShelfChange3D supports benchmarking and real-world adaptation for object-level 3D shelf change detection.

    Source: §VI · p. 7

    Future work will extend the framework to more diverse retail environments and more complex forms of shelf change.

    Source: §VI · p. 7

Abstract

from arXiv · show

Reliable shelf monitoring is an important capability for retail automation, yet existing out-of-stock detection methods mainly operate in image space and lack metric 3D localization for downstream robotic systems. We formulate shelf monitoring as object-level 3D change detection: given two RGB-D observations captured at different times, the goal is to identify changed products and localize each change with a 3D bounding box. To support this task, we introduce ShelfChange3D, comprising 145K synthetic and 5K real-world paired RGB-D observations with object-level 3D change annotations. We further propose ChangeBox, an end-to-end framework that jointly reasons over paired observations and predicts object-level 3D change boxes. To improve localization accuracy, we introduce a geometry-based refinement stage that exploits depth and gravity prior to estimate relative pose and refine predicted boxes. Experiments show that ChangeBox outperforms existing change detection baselines, with further gains from refinement and effective transfer from synthetic to real-world observations.

I. INTRODUCTION

Shelf monitoring needs to move beyond image-space out-of-stock detection to locate changed products as metric 3D boxes from observations captured over time. ShelfChange3D and ChangeBox address this gap with paired RGB-D data, cross-time reasoning, and geometry-based refinement.

I. INTRODUCTION

  • Existing vision-based out-of-stock methods and retail datasets mainly use image-space outputs, while product-level changes lack metric 3D position and extent for downstream robotic perception and action.Prior 3D change detection generally lacks 3D bounding boxes, while standard 3D object detection operates on a single observation and misses products already removed.Source: §I · p. 1
  • The task requires cross-time reasoning and object-level localization because a removed product may be absent from the current observation.The target is to identify what changed across two shelf observations and localize each changed product in 3D.Source: §I · p. 1
  • ShelfChange3D provides paired RGB-D observations with object-level 3D change annotations, and ChangeBox jointly reasons across observations to predict product-level 3D change boxes.The benchmark contains 145K synthetic and 5K real-world observations.Source: §I · p. 1
  • Precise 3D localization remains challenging, with box-center estimation the dominant error source; geometry-based refinement uses depth and a gravity prior to estimate relative pose and improve boxes.Experiments also examine viewpoint variation and synthetic-to-real transfer.Source: §I · p. 1

II. RELATED WORK

  • Prior retail benchmarks mainly provide 2D annotations, and existing 3D change approaches differ from this work’s direct product-level 3D change detection from paired RGB-D observations.DenseChangeCap detects added, removed, and moved objects as 3D boxes from registered point-cloud scans.Source: §II · p. 2

A. Synthetic Dataset

ShelfChange3D’s synthetic data are generated in simulation, where product poses are known and shelf states can be rendered from multiple viewpoints. Paired observations from simulated trajectories are labeled with removed and added products and their 3D boxes.

A. Synthetic Dataset

  • Simulation provides known product poses and supports rendering the same shelf state from multiple viewpoints, avoiding costly manual staging and 3D box annotation for each change.The acquisition setup emulates a robot revisiting shelves from stopping points with imperfect pose repeatability.Source: A. Synthetic Dataset · p. 2
  • Shelf layouts sample SKU groups with column-and-row grids, while simulated trajectories apply removals and misplaced-product additions.The example shows the sampled grid layout and, after remove-add steps, vacated cells and misplaced products.Source: 2) Generation Process · p. 2, Figure 2 · p. 3
  • Nine nominal viewpoints around each shelf layer are perturbed at trajectory and frame levels to model variation in camera pose.The figure distinguishes nominal poses, trajectory-level perturbation ranges, realized poses, and per-frame revisit jitter.Source: 2) Generation Process · p. 2, Figure 3 · p. 3
  • Observation pairs combine an earlier and a later state from the same camera group, potentially viewed from different viewpoints.Pairs are formed offline from recorded trajectories.Source: 3) Pair construction · p. 3
  • Pair labels include products removed or added between states, with each product’s 3D box transformed into the later camera frame; unchanged products are unlabelled.Viewpoint pairs are categorized as near-same, horizontal, vertical, or diagonal according to their nominal cameras.Source: 3) Pair construction · p. 3

B. Real-World Dataset

ShelfChange3D-5K provides real-world shelf observations annotated for product removals, while ChangeBox predicts product-level 3D changes from paired RGB-D observations. Its detector combines image features and metric depth, and its predictions can be refined using estimated relative camera pose.

B. Real-World Dataset

  • ShelfChange3D-5K covers a controlled setting with a fixed camera and product removals, using a semi-automatic pipeline to annotate removed products with 3D boxes.The pipeline combines image differencing, promptable segmentation, 3D reconstruction, and manual review.Source: B. Real-World Dataset · p. 3

IV. METHOD

  • ChangeBox encodes both shelf observations with a shared image backbone, appends depth to image tokens, and predicts changed-object boxes with a query-based decoder.For removed products, the refinement network uses points from the earlier observation transformed into the current frame.Source: §IV · p. 4

A. ChangeBox Detector

  • Each feature-grid cell receives its median depth, concatenated with its image feature so each token carries patch appearance and metric depth.The paired-frame tokens are joined for decoding, and each candidate change receives a region feature.Source: 3) Depth Dimension · p. 4, 4) Change Query Decoder · p. 4
  • The detector predicts removed-or-added class probabilities and oriented boxes in the gravity-aligned camera frame of the current observation.Each box includes a 3D center, size, and yaw about the vertical axis.Source: 5) Detection Head · p. 4

B. Gravity-prior Extrinsics Estimation

The gravity-prior estimator recovers relative pose from two depth maps for the refinement stage, which requires pose estimates. It uses gravity alignment, candidate generation and selection, then a forward-backward consistency check that can reject registration.

B. Gravity-prior Extrinsics Estimation

  • The refinement stage estimates relative pose from the two depth maps; known gravity directions in both frames make the estimation possible.The detector itself does not require extrinsics.Source: B. Gravity-prior Extrinsics Estimation · p. 4
  • Gravity alignment reduces the relative rotation to a search over one angle about the vertical axis, alongside a 3D translation.Each depth map is back-projected to a point cloud and rotated so gravity is vertical.Source: 1) Gravity Alignment · p. 4
  • For nearly same-view revisits, the estimator first tests identity initialization with point-to-plane ICP and evaluates the result using depth agreement.If agreement is high, this solution proceeds directly to the forward-backward check.Source: 2) Identity Short-cut · pp. 4–5
  • Otherwise, it sweeps candidate yaw angles, estimates translations from occupancy-grid and height-histogram correlations, and ranks refined candidates by symmetric depth agreement.A final backward ICP check accepts a pose only if the residual rotation and translation are below thresholds; otherwise refinement is skipped for removed boxes.Source: 3) Candidate Generation · p. 5, 4) Refinement and Selection · p. 5, 5) Forward-backward Check · p. 5

C. Local Box Refinement

ChangeBox's localization errors are primarily center shifts rather than incorrect box sizes, motivating a local depth-based refinement of predicted boxes.

C. Local Box Refinement

  • Replacing predicted centers with ground truth raises mAP0.7 from 14.8 to 94.0, while replacing sizes has little effect, indicating shifted rather than mis-sized boxes.The method therefore refines each predicted box using enclosed depth points.Source: C. Local Box Refinement · p. 5, Table I · p. 5
  • For added boxes, refinement collects points from current depth; for removed boxes, it uses earlier depth transformed into the current frame.The predicted box is slightly enlarged before points are collected and transformed into box coordinates.Source: 1) Point Collection · p. 5
  • Each collected point is represented by its normalized position, distances to box faces, depth cues, and the predicted box score, class, and log size.The point features include depth normal, depth-gradient magnitude, and local valid-depth ratio.Source: 1) Point Collection · p. 5
  • A point-wise MLP and max-and-mean pooling feed regression heads that correct center and log-scale size while leaving orientation unchanged.The refined center is c + R∆c and size is d ⊙e∆logd.Source: 2) Refinement Head · p. 5

D. Loss Function

The loss section describes assignment-based supervision for change boxes and a separately trained refinement network. It specifies focal classification loss and a symmetry-aware orientation term, but the supplied passages do not expose the full loss expressions.

D. Loss Function

  • Hungarian assignment matches ground-truth change boxes to predictions before computing the detection loss.A box is represented by change type and box parameters.Source: 1) 3D change detection Loss · p. 5
  • The classification component uses focal loss over change types, and the orientation term compares against the closest of the product's equivalent yaw angles.The closest-angle rule accounts for a product's symmetry order.Source: 1) 3D change detection Loss · p. 5
  • The refinement network is trained separately, with its center and size targets defined as corrections mapping the input box to ground truth.The supplied passage names the corrections as ∆c∗ and ∆d∗.Source: 1) 3D change detection Loss · p. 5

V. EXPERIMENTS

The experiments compare ChangeBox configurations with a 3D change-detection baseline and evaluate representation, resolution, depth, and viewpoint robustness. DINOv3-S with refinement achieves the strongest reported results, while explicit depth and the refinement stage improve localization.

A. Experimental Setup

  • The evaluation reports 3D mAP and mAR at IoU thresholds 0.25, 0.5, and 0.7, averaged across removed and added objects where both are present.It also reports mean center error in centimeters and retains at most 20 detections per observation pair.Source: A. Experimental Setup · p. 5

B. Experiments on the ShelfChange3D-145K

  • DINOv3-S with refinement achieves 93.0, 81.7, and 48.3 mAP at IoU thresholds 0.25, 0.5, and 0.7, respectively.ChangeBox substantially outperforms DenseChangeCap, and refinement improves all three backbones, especially at stricter IoU thresholds and in center error.Source: B. Experiments on the ShelfChange3D-145K · p. 6, Table II · p. 6
  • Stronger visual backbones and 640 resolution consistently improve performance, with DINOv3-S at 640 achieving the best overall results.Increasing resolution from 320 to 640 improves all three backbones, particularly at stricter IoU thresholds.Source: B. Experiments on the ShelfChange3D-145K · p. 6, Table III · p. 6
  • Adding explicit depth raises DINOv3-S mAP0.5 from 45.1 to 57.5 and reduces center error from 1.28 cm to 1.14 cm.Depth improves performance across all three backbones, with the largest gains generally at stricter IoU thresholds.Source: B. Experiments on the ShelfChange3D-145K · p. 6, Table IV · p. 6
  • Performance remains strong up to about 40° of relative camera rotation, but declines at larger rotations, especially for mAP0.5 and center accuracy.The <5° bin is not the best-performing bin despite containing most training pairs.Source: B. Experiments on the ShelfChange3D-145K · p. 6
  • The extrinsics estimator solves 90% overall, from 100% for near-same views to 62% for diagonal pairs; the consistency check lowers wrong accepts from about 10% to 3%.The forward-backward check uses τR = 1° and τt = 1 cm.Source: B. Experiments on the ShelfChange3D-145K · p. 6

C. Sim2Real on ShelfChange3D-5K

On real shelves, synthetic pretraining transfers effectively, and fine-tuning with refinement further improves localization. The conclusion presents ShelfChange3D and ChangeBox for object-level 3D change detection, while identifying broader retail environments and more complex shelf changes as future work.

C. Sim2Real on ShelfChange3D-5K

  • Fine-tuning the synthetically pretrained model reaches 92.9 mAP0.25 before refinement; refinement raises it to 97.9 and reduces center error.Synthetic-only training transfers effectively to the real test set in a zero-shot setting.Source: C. Sim2Real on ShelfChange3D-5K · pp. 6–7

VI. CONCLUSION

  • ShelfChange3D pairs synthetic and real-world datasets with ChangeBox, which predicts product-level 3D changes by jointly reasoning across observations.The conclusion describes the framework as detecting shelf changes across different viewpoints.Source: §VI · p. 7
  • Future work will extend the framework to more diverse retail environments and more complex forms of shelf change.Source: §VI · p. 7

APPENDIX

The appendix reports a robot restocking demonstration, additional training and qualitative results, comparisons with existing datasets, and details of ShelfChange3D-145K construction and annotation.

A. Restocking robot experiments

  • A dual-arm humanoid robot uses each removed product’s predicted 3D-box center as an inverse-kinematics placement target to restock the shelf.The demonstration applies the method to locate removed products and restore the shelf to its fully stocked state.Source: A. Restocking robot experiments · p. 10, Figure 9 · p. 11

B. Additional experiments

  • With DINOv3-S, Frozen→FT reaches 87.0 mAP0.25, 57.5 mAP0.5, and 1.14 cm center error; mAP0.25 is 76.2 with FT and 77.0 with a frozen backbone.The Frozen→FT schedule also performs best across ResNet-50 and ConvNeXt-T.Source: Table VI · p. 10, 1) Training Strategy · p. 10

C. Dataset Novelty

  • ShelfChange3D-145K pairs object-level 3D change boxes with single-frame RGB-D observations, unlike prior datasets offering only 2D changes, non-box 3D labels, scan pairs, or single images.DyS2Change provides 3D boxes for point-cloud scan pairs; retail datasets listed provide single images with 2D product boxes or shelf-gap points.Source: C. Dataset Novelty · p. 10, Table VII · p. 10

D. ShelfChange3D-145K dataset Details

  • The simulated assets include 67 grouped SKUs with product categories, six shape classes, and vertical rotational-symmetry labels, arranged in perturbed grids within shelf-layer areas.Shelf boards are split into usable areas with local x-width, y-depth, and z-up frames; groups, columns, and rows are recorded for products.Source: 1) Assets · p. 10, 2) Shelf Layout Generation · p. 10
  • Simulated cameras emulate an Intel RealSense D435i, with sphere-based viewpoints and stored relative pose and full 3D camera-orientation difference; changes include removals and misplaced additions.Camera viewpoints use azimuth and elevation, with trajectory-level offsets and per-frame jitter; the orientation difference includes roll.Source: 3) Camera Configuration · pp. 10–11, 4) Change Generation · p. 11, Figure 11 · p. 12

B. Additional experiments

  • ChangeBox’s predicted removed-object boxes align closely with ground truth across two real-world examples despite clutter and partial occlusions.The examples illustrate localization of multiple product removals in ShelfChange3D-5K.Source: 2) Qualitative Results on… · p. 10, Figure 10 · p. 12

D. ShelfChange3D-145K dataset Details

  • Annotations use the current t1 camera frame, enabling changed-object localization relative to the robot’s current viewpoint without a persistent global scene map.Each annotation also includes mask-tight and projected 2D boxes, occlusion and truncation in both frames, and SKU attributes.Source: 5) Coordinate System and Format · p. 12

E. ShelfChange3D-5K Dataset Details

The real-world test examples vary in the number and spatial arrangement of removed products. The method implementation weights classification, center, size, and orientation terms differently in its detection loss.

E. ShelfChange3D-5K Dataset Details

F. Implementation Details of the Method

  • The detection loss weights classification by 2, center error by 4, and size and orientation errors by 1 each.The focal loss uses γ=2 and α=0.25.Source: 1) ChangeBox Detector · p. 12

1) ChangeBox Detector:

This section describes ChangeBox training schedules and local box refinement, with annotation and test-split examples and a reimplemented point-cloud baseline.

1) ChangeBox Detector:

  • The figures show a ShelfChange3D-145K annotation example and ShelfChange3D-5K test-split examples, with t0 on the left and t1 on the right.Source: Figure 12 · p. 12, Figure 13 · p. 13
  • ChangeBox compares frozen, full-fine-tuning, and frozen→FT schedules, using AdamW, 0.01 weight decay, batches of 24 pairs, and bf16 mixed precision.The default frozen→FT schedule unfreezes the backbone after 48 epochs, then trains for 8 more epochs and decays the learning rate for 12 epochs.Source: 1) ChangeBox Detector · pp. 12–13

2) Local Box Refinement:

  • Refinement samples up to 1,024 depth points per predicted box, leaves boxes with fewer than 30 points unchanged, and trains on detector errors and perturbed ground-truth boxes.Detector-error boxes form 60% of training boxes; synthetic perturbations form 40%.Source: 2) Local Box Refinement · p. 13
  • The refinement network trains with simulated D435i depth corruption and is then used unchanged across other backbones and real-world data.Corruptions include Gaussian noise, edge fattening at depth discontinuities, and random depth holes.Source: 2) Local Box Refinement · p. 13
  • The reimplemented DenseChangeCap baseline merges four before/after and difference point clouds, samples 20,000 points, and uses PointNet++ with a 3DETR decoder.It is trained from scratch on the authors’ data with the same budget as ChangeBox.Source: 3) DenseChangeCap Baseline · p. 13
Loading 2610.01283v1…