PaperlayerView exact revision on arXiv

Source-linked AI summary

Know-Your-Scene (KYS)-SLAM: Hierarchical Semantic-Motion Priors for Feature Matching in Stereo Visual SLAM

Preeti Chatterjee, Jin Lu, Jin Sun, Suchendra M. Bhandarkar

arXiv:2609.27509v1cs.CVcs.RO

TL;DR

Feature-based stereo SLAM can misassociate semantically ambiguous regions, same-class instances, and independently moving objects, while binary rejection can reduce correspondence density. KYS-SLAM modulates matching costs with semantic, panoptic, and zero-shot motion priors; with one fixed configuration, it reduces ATE across outdoor, indoor, and synthetic evaluations.

  • Problem

    Feature-based SLAM relies on appearance descriptors and predominantly static-scene assumptions, leaving data association vulnerable to semantic ambiguity, instance confusion, and independently moving objects.

    Source: §1 · p. 1

  • Method

    KYS-SLAM integrates semantic, panoptic, and zero-shot motion priors into stereo matching as hierarchical cost modulation rather than binary feature rejection.

    Source: §1 · pp. 1–2

  • Results

    17.4% KITTI and 27.7% EuRoC average per-sequence ATE reductions across 21 sequences without regression, using coefficients fixed across datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Contextual evidence can be used to penalize implausible correspondences while preserving geometric support for bundle adjustment.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Stereo visual SLAM systems built on local descriptors suffer from semantic ambiguity, instance-level confusion, and independently moving objects, each corrupting data association and accumulating as trajectory drift. Prevailing semantic and dynamic SLAM methods address this through binary feature rejection, sacrificing correspondence density for outlier suppression. We contend that contextual implausibility is better expressed as a graded quantity than an exclusion criterion. We present Know-Your-Scene (KYS)-SLAM, a modular extension of ORB-SLAM3 that supplants feature rejection with continuous correspondence modulation. The contribution is the reframing of contextual evidence as correspondence cost, applied within feature matching and leaving the geometric backend unmodified. Each keypoint is augmented with semantic, panoptic, and motion priors fused through a hierarchical compatibility formulation, in which semantic class and instance identity enforce structural plausibility while a zero-shot motion score down-weights features on independently moving objects. That score comes from a training-free module fitting a depth-aware ego-motion model to background optical flow and classifying panoptic segments via self-calibrating, coverage-aware thresholds, so only segments with sufficient motion evidence are penalized and static structure is left unpenalized. Penalizing correspondences rather than discarding them preserves the geometric support bundle adjustment depends on. Under one fixed configuration, no coefficient retuned per sequence or dataset, KYS-SLAM reduces per-sequence ATE RMSE by 17.4% on outdoor KITTI and 27.7% on indoor EuRoC across 21 stereo sequences with no regressions, and by 6.6% on dynamic subsets of KITTI Tracking and 17.8%, up to 31.2%, on Virtual KITTI 2 -- cross-domain transfer across outdoor driving, indoor flight, and synthetic imagery under one set of constants.

1. Introduction

Feature-based SLAM is vulnerable to semantic ambiguity, same-class instance confusion, and independently moving objects. KYS-SLAM instead modulates correspondence costs with contextual priors and reports gains across outdoor and indoor stereo datasets under one fixed configuration.

1. Introduction

  • Appearance-based matching in predominantly static-scene systems is vulnerable to semantic ambiguity, same-class instance confusion, and independently moving entities, which degrade association and propagate trajectory drift.Source: §1 · p. 1
  • Prior semantic and dynamic methods generally filter or suppress observations, reducing correspondence density and sometimes discarding static, geometrically informative features.The authors argue contextual priors should modulate correspondence cost rather than determine whether a correspondence exists.Source: §1 · p. 1
  • KYS-SLAM extends ORB-SLAM3 by combining semantic, panoptic, and motion priors through hierarchical correspondence compatibility.Semantic class, temporally consistent instance identity, and depth-compensated scale rescale descriptor distance; a zero-shot, depth-aware motion module penalizes only segments with sufficient motion evidence.Source: §1 · pp. 1–2
  • KYS-SLAM reduces average per-sequence ATE by 17.4% on KITTI and 27.7% on EuRoC, without regression across 21 sequences.It also reports reductions of 6.6% on dynamic KITTI Tracking subsets and 17.8%, up to 31.2%, on Virtual KITTI 2, with coefficients fixed across datasets.Source: §1 · p. 2

2. Related Work

Prior optimization-centric systems rely on geometric estimation, while semantic and dynamic SLAM commonly suppress observations judged unreliable. KYS-SLAM uses semantic, structural, and motion evidence as continuous correspondence priors in sparse feature matching.

Optimization-Centric and Learned Visual SLAM

  • Optimization-centric systems such as ORB-SLAM, VINS-Fusion, and SVO use geometric or factor optimization but do not use semantic labels, instance identities, and motion priors as graded matching-cost priors.Source: §2 · p. 2

Semantic and Dynamic SLAM

  • Semantic and panoptic SLAM methods commonly mask or remove observations judged dynamic or unreliable, whereas KYS-SLAM preserves geometric support while penalizing implausible associations.Examples include DynaSLAM, DS-SLAM, RSO-SLAM, Panoptic-SLAM, and PSMD-SLAM.Source: §2 · pp. 2–3

Optical Flow and Motion Modeling

  • Optical-flow methods use motion reasoning for segmentation, object-motion estimation, or feature filtering; KYS-SLAM turns motion evidence into a continuous prior within sparse feature matching.Related approaches include FlowFusion, VDO-SLAM, and Chen et al..Source: §2 · pp. 2–3

3. Know-Your-Scene (KYS)-SLAM

KYS-SLAM extends ORB-SLAM3 by modulating stereo correspondence costs with semantic, panoptic, and motion priors rather than rejecting contextual mismatches. The geometric backend remains unchanged, while keypoint priors inform matching throughout tracking, mapping, and loop closing.

3.1. From Semantic Labels to Correspondence Cost

  • At α=0.1, ATE reaches 10.40 m on Seq. 02 versus 4.66 m at α=0.7; class costs retain close-class boundary matches while descriptor distance remains dominant.α=0.7 is a representative value in a stable range, and coefficients are fixed across all 21 sequences.Source: §3.1 · p. 3

3.2. Panoptic Structural Backbone

  • Panoptic matching adds temporally tracked instance identity and depth-compensated segment scale, using three compatibility tiers to gently down-weight implausible matches rather than reject them.The tiers distinguish same-instance, cross-instance same-class, and cross-class pairs; area discrepancy helps when labels or identities differ across views.Source: §3.2 · pp. 3–4

3.3. Zero-Shot Motion Detection

  • A stuff-only depth-aware flow fit yields residuals; local background-drift correction and coverage-aware dual thresholds gate a bounded motion score, leaving unflagged segments unpenalized.The model is fit on stuff pixels, and weak background coverage raises detection thresholds.Source: §3.3 · p. 5

3.4. Unified Motion-Panoptic Matching

  • Stereo candidates are rejected above μ>8 and temporal matching above μ>20; surviving pairs combine structural scaling with tier-specific motion penalties in one distance, leaving ORB-SLAM3 optimization unchanged.The two unilateral gates reflect different error sensitivities in triangulation and temporal tracking.Source: §3.4 · p. 6

4. Experiments and Results

KYS-SLAM is evaluated as a fixed-configuration stereo extension on KITTI and EuRoC, with reported KITTI gains and added computational costs. On EuRoC, the panoptic backbone carries the gain while the motion module stays largely inactive.

Setup

  • The evaluation uses all 11 KITTI and 10 EuRoC sequences, excludes EuRoC V203, averages ATE RMSE over three runs, and fixes parameters across both datasets.Baseline results come from peer-reviewed publications under matched stereo evaluation; ORB-SLAM3 is the only system rerun on the authors’ hardware.Source: §4 · p. 6

Main Results

  • 17.4%: the full system reduces per-sequence ATE over ORB-SLAM3 on KITTI, while the panoptic backbone reduces it by 16.6%.The reduction reaches 22.9% on the dynamic subset.Source: §4 · p. 6, Table 1 · p. 7
  • On EuRoC, where independent motion is scarce, the backbone carries the gain while the motion module stays largely inactive.Table 2 reports EuRoC MAV ATE RMSE, with lower values better.Source: §4 · p. 6, Table 2 · p. 7

Runtime and system cost

  • With priors resident, tracking runs at 32.6–35.7 ms/frame (28–31 Hz); the system is not optimized for real-time operation as a whole.Matching modulation adds 6.8–8.2 ms/frame, about 0.4% of pipeline cost; prior generation uses off-the-shelf perception models.Source: §4 · p. 7

5. Analysis: Ablation and Behavior

The analysis tests modulation, robustness to perception errors, and behavior across static and dynamic scenes. The motion module helps on dynamic benchmarks, but detection remains constrained by scene residual variability and can regress in some settings.

Behaviour on static scenes

  • On EuRoC, the motion module raises the mean ATE reduction over the panoptic backbone from 27.4% to 27.7%, while never regressing below ORB-SLAM3.The passage attributes feature survival under the stereo gate to culling that scales with scene difficulty, not object motion.Source: §5 · p. 8

Effect of the motion module

  • 6.6% on KITTI Tracking and 17.8% on Virtual KITTI 2: mean per-sequence ATE falls, reaching 31.2% on Virtual KITTI 2 Sc. 01.For Sc. 18/0018, the gain holds under synthetic and real ground truth at 13.5% and 11.8%, respectively.Source: §5 · p. 8, Table 5 · p. 8

Ablation and behavior

  • Disabling continuous modulation while retaining hard gates degrades ATE by 11.7%, 4.1%, and 23.1% on Seqs. 03, 09, and 10.This comparison indicates the contribution is not gating alone.Source: §5 · p. 8

Robustness to perception error

  • Tracking survives every tested corruption level because descriptor distance remains primary and priors only rescale it within bounded factors; incorrect priors reweight correspondences rather than remove them.With zero-mean flow noise scaled by local magnitude, accuracy is unchanged at 2.7 px on both sequences.Source: §5 · p. 8

Detection limits

  • Distant slow movers can remain within background residual variability, while small distant segments can trigger radial tests on static structure; lowering thresholds does not correct either error.The reported regressions against the backbone on Seqs. 00 and 01 occur with sparse motion or large depths and sparse near-field structure.Source: §5 · p. 8

6. Conclusions and Future Work

KYS-SLAM reframes semantic, panoptic, and motion evidence as a modulated descriptor distance rather than a rejection rule. Under one fixed configuration, it reports ATE reductions across outdoor, indoor, and dynamic-scene benchmarks, and identifies online priors, learned features, and end-to-end deployment as future work.

Conclusion

  • KYS-SLAM extends ORB-SLAM3 stereo matching by fusing semantic, panoptic, and zero-shot motion priors into one modulated descriptor distance.It rescales correspondences by contextual plausibility rather than rejecting them.Source: §6 · p. 8
  • 17.4% on KITTI and 27.7% on EuRoC: per-sequence ATE reductions across 21 stereo sequences, with no regressions under one fixed configuration.On dynamic subsets, reductions are 6.6% on KITTI Tracking and 17.8%—up to 31.2%—on Virtual KITTI 2.Source: §6 · p. 8

Future work

  • Future work targets online priors, learned features, and end-to-end deployment.Source: §6 · p. 8

A. Coefficient sensitivity

The coefficient sweeps show a stable geometry-dominant range for semantic weighting and comparatively shallow sensitivity to panoptic tier weights. The same-instance discount also matters, with both tested endpoints degrading accuracy across all three sequences.

Coefficient sensitivity

  • At α = 0.1, Seq. 02 reaches 10.40 m; accuracy is stable in the geometry-dominant range α ≥ 0.6, and the reported α = 0.7 lies within it.Seq. 08 is marginally better at α = 0.8 (3.40 against 3.42 m), while Seq. 07 is insensitive across the geometry-dominant range.Source: A. Coefficient sensitivity · p. 11, Table S1 · p. 11

Panoptic tiers

  • Across the panoptic tier-weight sweep, ATE varies by at most 27.5% on Seq. 10 and under 15% on Seqs. 03 and 09.The reported setting is best on Seqs. 03 and 10 and tied with (0.60, 0.25, 0.15) on Seq. 09; the caption identifies it as the reported configuration, not each sequence’s best.Source: A. Coefficient sensitivity · p. 11, Table S2 · p. 11

Same-instance discount

  • Both γ sweep endpoints degrade accuracy on all three sequences: γ = 0.60 costs 10.0–20.9%, while γ = 1.00 costs 2.4–30.8%.The reported γ = 0.75 is best on Seqs. 09 and 10 and tied with γ = 0.70 on Seq. 03; γ = 1.00 removes the same-instance preference.Source: A. Coefficient sensitivity · p. 11, Table S3 · p. 11

B. Motion sensitivity

Motion sensitivity tests show that continuous modulation contributes beyond hard motion gates, while excessive sensitivity over-penalizes usable correspondences.

B. Motion sensitivity

  • Accuracy is flat from 0.5× to 2× sensitivity, but degrades at 4× on Seq. 09 and 10.The reported 1× setting is best on Seq. 03 and 10; 2× is marginally better on Seq. 09 (1.67 against 1.70 m) and marginally worse on Seq. 10 (0.92 against 0.91 m).Source: B. Motion sensitivity · p. 11
  • Disabling continuous modulation while retaining hard gates worsens ATE by 11.7%, 4.1% and 23.1% on Seq. 03, 09 and 10.This isolates the graded motion term's contribution from gating.Source: B. Motion sensitivity · p. 11, Table S4 · p. 12
  • Gating without modulation is no better than omitting the motion module, and only their combination improves on the panoptic backbone.Against the backbone's 1.14, 1.75 and 0.99 m results, gating alone is 0.9% and 1.1% worse on Seq. 03 and 09 and 13.1% worse on Seq. 10; hard gates remain for cases where descriptor similarity is no longer meaningful.Source: B. Motion sensitivity · pp. 11–12, Table S4 · p. 12

C. Runtime and system cost

The supplementary evaluation distinguishes the online matching cost from offline perception, tests backend-time confounding, and examines how motion gating behaves across scene conditions.

C. Runtime and system cost

  • Concurrent disparity and two optical-flow passes take 840 ms/frame rather than 1016 ms in sequence, a 1.21× gain.Optical flow and panoptic segmentation run on both stereo images, while disparity is computed once; prior generation also adds 23.4 ms of CPU-side fusion.Source: C. Runtime and system cost · p. 12, Table S5 · p. 12
  • With priors resident, KYS-SLAM tracking runs at 32.6–35.7 ms/frame (28–31 Hz); including deserialisation, it runs at 53.2–55.4 ms/frame (18–19 Hz), above KITTI's 10 Hz capture rate.KYS-specific costs total 49.2–51.5 ms/frame, or 2.8–2.9% of the pipeline; modulation itself is 0.38–0.46%. Peak host memory rises 0.22–0.52 GB, 32–43% above stock peak RSS. The perception front-end is precomputed offline.Source: Table S6 · p. 12, C. Runtime and system cost · p. 12

D. Backend optimisation budget

  • An injected 50 ms/frame delay gives mapping 94–99% of that budget as extra optimisation time, yet changes ATE by at most 0.02 m.The delay raises wall-clock by 37–43% and yields no measurable benefit, so the modulation's 6.8–8.2 ms/frame cost cannot explain the accuracy gain.Source: D. Backend optimisation budget · pp. 12–13, Table S7 · p. 13

E. Feature survival under the stereo gate

  • Feature survival falls to ≈73% on V103, where rapid rotation and motion blur coincide, yet tracking continues and accuracy improves over ORB-SLAM3 (0.050 against 0.061 m).The gate is unilateral and bounded, and keypoints without valid panoptic labels bypass modulation rather than being discarded.Source: E. Feature survival under the stereo… · p. 13, Figure S1 · p. 13

F. Motion-score distribution

  • The bounded motion-score distribution places both the stereo gate (µi=8, retaining 92.3%) and extreme gate (µi=20, retaining 97.0%) on its plateau.The gates therefore transfer across sequences without per-scene adjustment; the constant and gate positions amount to one degree of freedom.Source: F. Motion-score distribution · p. 13, Figure S2 · p. 13

G. Robustness to perception error

Perception perturbations test robustness to segmentation and flow error, while qualitative examples illustrate how the motion model distinguishes moving objects from static structure.

G. Robustness to perception error

  • Misclassified textured, proximal objects attract 1.6–1.9× more ORB keypoints than their pixel-area share; on Seq. 06 at 60% corruption, exposure reaches 13.1% at the 95th percentile and 43.7% in the most affected frame.The injected segmentation errors are nested reassignment of thing segments to another observed class, with geometry, disparity and flow held fixed; the clean condition already includes the segmenter's errors.Source: G. Robustness to perception error · pp. 13–14
  • On Seq. 10, ATE stays below the ORB-SLAM3 baseline (1.32 m) at every tested segmentation and flow corruption level; Seq. 06 matches its 0.92 m baseline at 10% segmentation corruption and concedes above it thereafter.For flow noise, Seq. 06 remains at or below its baseline at 2.7 px and concedes at most 0.09 m above it; tracking is never lost. Flow noise is per-pixel independent, so this tests estimator precision rather than structured failures.Source: G. Robustness to perception error · p. 14

H. Qualitative motion classification

  • The motion model derives residual flow by subtracting depth-based predicted ego-motion flow from observed flow.The resulting per-segment classification flags moving vehicles and pedestrians while leaving parked vehicles unpenalized within the same class.Source: Figure S3 · p. 14, H. Qualitative motion classification · p. 14
  • Static regions and independently moving regions are shown separately, with static structure remaining unpenalized even when it shares a class with flagged movers.The per-segment decision determines what is penalized, while the bounded score determines how much.Source: Figure S4 · p. 14, H. Qualitative motion classification · p. 14

I. Detection limits

Motion detection can miss small or distant movers and can falsely flag parked vehicles, but bounded correspondence modulation limits the consequences in the tested cases.

I. Detection limits

  • Small, distant movers can be missed when their projected motion approaches the residual noise floor, especially when segmentation splits the object's motion evidence.The missed motorcyclist is split into rider and vehicle segments, neither of which aggregates the full evidence.Source: §I · p. 14, Figure S5 · p. 15
  • Small segments can produce false positives because their mean has greater sampling error against a full-image background threshold; two parked vehicles were flagged by the radial test alone.Their segment sizes were 907 and 693 px, versus 3079–4380 px for correctly classified vehicles, while the background threshold was unusually tight on the straight trajectory.Source: §I · p. 15, Figure S6 · p. 16
  • Lowering thresholds cannot fix the missed-mover case without flagging static vehicles first; bounded modulation attenuates weak-evidence false positives rather than discarding their correspondences.Across the tested label, flow, rotation, blur, and small- or distant-object failure cases, accuracy degrades gradually and tracking survives.Source: §I · p. 15

J. DROID-SLAM baseline protocol

  • The reported stereo EuRoC comparison uses DROID-SLAM* retrained from scratch for stereo by DVI-SLAM, rather than the original network trained on synthetic monocular video.The original stereo results were described as recovering camera trajectory up to scale; the retrained baseline is reported under controlled settings, unchanged from its published table.Source: J. DROID-SLAM baseline protocol · p. 15

Related papers

Loading 2609.27509v1…