Source-linked AI summary
DetM-Tracker: Deterministic and Memory-Efficient VIS for Autonomous Industrial Inspection
You-Rui Lin, Jen-Chieh Wu, Tzu-Ti Wei, Jen-Jee Chen, Yu-Chee Tseng
TL;DR
Industrial inspection requires deterministic, memory-efficient video instance segmentation, but dense temporal memory and fixed-capacity decoders scale poorly for constrained edge platforms. DetM-Tracker combines compact scalar memory, adaptive instance channels, and entropy-gated refinement. The paper reports improved accuracy over evaluated baselines alongside lower latency and VRAM use, and tests generalization across multiple cabinet configurations.
Problem
Dense temporal memory grows with spatial resolution and sequence length, while fixed-capacity decoders do not exploit industrial-scene sparsity, creating a memory wall for resource-constrained edge deployment.
Source: §1 · p. 1Method
DetM-Tracker combines four-scalar identity descriptors, query capacity adapted to object count, and entropy-gated feature refinement focused on uncertain regions.
Source: §1 · p. 2, §3.3 · p. 4, §3.4 · p. 5Results
DetM-Tracker outperforms evaluated baselines across all accuracy metrics; it achieves 15.2 ms latency, a 20× speedup over Cutie, and 557 MB VRAM, a 6× reduction versus dense-feature baselines.
Source: §5 · p. 7Takeaways & Limitations
The reported efficiency and accuracy support DetM-Tracker’s use for high-cardinality segmentation on low-power edge hardware in autonomous industrial inspection.
Source: §5 · p. 7Environmental generalization was validated across multiple cabinet configurations with at least two distinct arrangements.
Source: B Limitations and Discussion · p. 13
Abstract
from arXiv · showhide
Autonomous industrial inspection requires strictly deterministic real-time inference under tight memory and computational constraints. However, existing Video Instance Segmentation (VIS) models suffer from prohibitive scaling: dense temporal memory grows with spatial resolution and sequence length, while fixed-capacity decoders fail to exploit the inherent sparsity of industrial scenes, creating a practical memory wall on resource-constrained edge platforms. We present DetM-Tracker, a streamlined VIS framework for autonomous industrial inspection, where target objects often exhibit highly similar appearance under constrained "micro" fields of view. DetM-Tracker eliminates architectural redundancy through three synergistic components: (1) Compact Scalar Memory, which condenses high-dimensional instance features into four-scalar descriptors, reducing temporal memory complexity from $O(T \cdot N \cdot R)$ to $O(T \cdot \hat{N})$. This mechanism effectively compresses frame-level representations into a bank of active instances, where $R$ denotes resolution, $N$ and $\hat{N}$ represent global and window-specific instance counts, respectively and $T$ the temporal window size; and (2) Dynamic Instance Channels, which adaptively allocate query capacity according to the actual number of visible objects; and (3) Entropy-Gated Skip Connection, which concentrates computational budget on high-uncertainty spatial regions to refine object boundaries in a single forward pass without auxiliary refinement heads. Our approach effectively bridges the gap between modern VIS models and the practical demands of industrial edge deployment, reducing inference latency by approximately 20$\times$ and VRAM consumption by 6$\times$ compared to state-of-the-art dense-feature baselines.
1 Introduction
Industrial inspection demands deterministic, memory-efficient long-horizon video inference, yet industrial objects can be visually similar, densely arranged, and only partially visible. DetM-Tracker addresses these constraints with sparse memory, object-count-adaptive queries, and uncertainty-focused refinement.
1 Introduction
- Industrial video inference needs deterministic latency and minimal memory, while dense temporal feature banks grow with resolution and sequence length.The paper identifies this growth as a practical memory wall for resource-constrained edge devices.Source: §1 · p. 1
- Industrial scenes pose an Identity Dilemma: similar-looking objects appear densely in regular patterns, while close-range views show only localized, partial regions.Ethernet sockets on server racks are the paper’s tracking case; reliable association depends more on stable spatial topology than ambiguous appearance cues.Source: §1 · pp. 1–2
- DetM-Tracker is proposed as a streamlined VIS framework that replaces heavy refinement with uncertainty-aware sparsity.Source: §1 · p. 2
- Compact Scalar Memory distills identities into sparse descriptors, making long-term tracking complexity resolution-invariant.The paper contrasts this with dense temporal feature banks whose footprint scales with spatial resolution [3].Source: §1 · p. 2
- Dynamic Instance Channels scale the query bank with the actual object count, avoiding proportional computational inflation in high-cardinality arrays [4, 5].The approach leverages fixed spatial layouts.Source: §1 · p. 2
- Entropy-Gated Skip Connection concentrates computation on high-uncertainty regions, filtering noise from confident interiors while sharpening mask boundaries [6].It replaces uniform feature fusion with uncertainty-aware allocation.Source: §1 · p. 2
2 Related Works
Prior work offers strong segmentation, video-memory, query-based, and boundary-refinement approaches, but the paper identifies deployment and scaling mismatches for industrial inspection. DetM-Tracker combines sparse temporal storage, adaptive query capacity, and selective single-pass refinement to address them.
Promptable Foundation Models
- SAM’s heavy encoders and iterative mask decoders create a computational bottleneck in high-throughput industrial pipelines where interactive prompting is infeasible.DetM-Tracker instead uses initial prompts as identity anchors for autonomous long-term propagation.Source: §2 · p. 2
Video Instance Segmentation (VIS)
- SAM 2’s dense spatial-temporal attention incurs O(T · H · W) complexity that can exhaust edge memory, motivating resolution-decoupled Compact Scalar Memory [2].The paper presents this memory abstraction as supporting stable identity maintenance when visually identical objects preclude appearance-based matching.Source: §2 · p. 2
Query-based Paradigms
- Fixed-capacity query banks can waste computation in sparse scenes or fail on high-cardinality arrays, whereas Dynamic Instance Channels size active queries from reference prompts [4] [7].Binding identities to initial boxes eliminates bipartite matching and scales computational costs with object count.Source: §2 · p. 2
Boundary Refinement
- Iterative point sampling multiplies latency, while uniform skip connections can introduce neighboring-instance noise; Entropy-Gated Skip Connection selectively refines ambiguous boundaries in one pass [6] [8] [9] [10] [11] [12].The paper describes the mechanism as parameter-free and deterministic in latency.Source: §2 · p. 2
Summary
- The framework’s synthesis replaces dense pixel storage, fixed-capacity decoders, and iterative refinement with sparse memory, adaptive channels, and single-pass gating.The paper states that this integration targets the memory wall and visual ambiguity of high-density identical instances through stable spatial topology and entropy-gated refinement.Source: §2 · p. 2
3 Method
DetM-Tracker separates persistent object identity from per-frame spatial and temporal reasoning. It initializes object queries once, then uses compact memory, bidirectional mask decoding, and entropy-gated high-resolution cues to produce masks.
3.1 Reference Encoding
- Reference geometry and image features are fused once per sequence into persistent Global Object Queries that anchor object identities while reducing per-frame computation.The Object Decoder uses geometric embeddings as queries and visual features as keys and values.Source: §3.1 · p. 3, Figure 2 · p. 4
3.2 Streaming Frame Encoding
- Each streaming frame is encoded into spatial image tokens that provide context for temporal memory injection and mask decoding.The token count N' is determined by the frame dimensions and patch size.Source: §3.2 · p. 4
3.3 Compact Scalar Memory
- Compact Scalar Memory stores each instance as four values—normalized centroid, relative area, and temporal decay—and stacks descriptors from a sliding window into historical memory.A memory encoder models trajectory dynamics and inter-instance relationships; current-frame tokens attend over the resulting sparse memory tokens.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
3.4 Bidirectional Mask Decoder
- The Bidirectional Mask Decoder mutually refines persistent object queries and temporal spatial features, then uses cross-modal affinity to generate instance masks in one pass.This decouples tracking capacity from network dimensions and supports high-cardinality segmentation.Source: §3.4 · p. 5
3.5 Entropy-Gated Skip Connection
- Entropy-Gated Skip Connection injects high-resolution evidence into uncertain regions, refining ambiguous contours while preserving confident interiors and avoiding added learnable parameters.The fused features pass through a lightweight segmentation head and are upsampled to the original resolution to produce final masks.Source: §3.5 · pp. 5–6, Figure 4 · p. 6
4 Training
Training follows a three-stage curriculum, progressing from synthetic geometric supervision through temporal consistency learning to real-world domain adaptation.
Stage 1: Geometric Initialization
- Stage 1 uses 972.9k synthetic frames to learn geometric segmentation and anchor identity prototypes without temporal context.The Global Cache learns reference prototypes while minimizing spatial loss.Source: Figure 5 · p. 6, §4 · p. 7
Stage 2: Temporal Consistency Learning
- Stage 2 trains on consecutive frames with a sliding temporal window to support motion-aware representations and long-term identity persistence.Historical instance states supervise Compact Scalar Memory under camera-induced displacement and temporary visual drift.Source: §4 · p. 7
Stage 3: Sim-to-Real Domain Adaptation
- Stage 3 fine-tunes on Real-Unplugged data for environment alignment and Real-Plugged data for complex occlusions from dense cabling.These smaller real-world splits target clutter and complex routing in operational server cabinets.Source: Figure 5 · p. 6, §4 · p. 7
5 Experiments
DetM-Tracker is evaluated against segmentation and tracking baselines, with reported advantages in accuracy, efficiency, occlusion robustness, and long-term stability.
Quantitative Evaluation of Segmentation Performance
- DetM-Tracker significantly outperforms SAM 2.1, SAM 3/3.1, Cutie, and LiVOS across all reported accuracy metrics.Table 1 compares segmentation capability against foundation models, fine-tuned variants, and VOS baselines.Source: §5 · p. 7, Table 1 · p. 7
- 15.2 ms latency is a 20× speedup over Cutie; VRAM use is 557 MB, a 6× reduction versus dense-feature baselines.The paper reports these as deployment advantages alongside segmentation performance.Source: §5 · p. 7
Detection Performance and Instance Robustness
- DetM-Tracker outperforms foundation models in complex industrial environments, where SAM 3.1’s recall and AP50 collapse under heavy occlusion.The text attributes generic frameworks’ fragmented masks to lack of domain-specific temporal modeling and topological continuity.Source: §5 · p. 7, Table 2 · p. 8
- Increasing SAM 2.1 prompt density or model parameters across 1P–5P yields only marginal gains on dense, repetitive cabling.The text also reports superior AP50 and J &F scores for DetM-Tracker, associated with identity persistence and mask proposals for maintenance tasks.Source: §5 · p. 8
Qualitative Analysis of Segmentation Fidelity
- Across environmental complexities, DetM-Tracker maintains sharp boundaries and topological fidelity while baselines fragment or produce incomplete masks under occlusion.Figure 6 highlights severe-occlusion “Hard” cases and low-contrast wiring.Source: §5 · p. 8, Figure 6 · p. 9
Long-Term Temporal Stability
- Over sequences exceeding 300 frames, DetM-Tracker maintains near-optimal precision while SAM 2.1 and LiVOS drift after the first 50 frames.SAM 3.1 shows volatile fluctuations, which the text associates with insufficient temporal memory for thin objects under motion.Source: §5 · p. 8, Figure 7 · p. 9
Ablation of the Gated Entropy Module
- Entropy-Gated yields a 14.4% mIoU boost on synthesis data, from 0.7130 to 0.8160, and achieves 0.6954 mIoU on real-world datasets.The text reports the real-world score as the module’s highest mIoU, outperforming traditional skip-connections.Source: §5 · p. 8
Robustness Across Varied Cabinet Environments
- DetM-Tracker maintains steady performance in dense eight-object occlusion scenarios, achieving 0.6929 mIoU in both Env1 and aggregate evaluations.The tested cabinet layouts contain real inspection-run non-idealities, including abrupt camera motion, blur, defocus, and viewpoint variation.Source: §5 · pp. 8–9
6 Conclusion
The conclusion presents DetM-Tracker as a high-fidelity industrial VIS framework, while identifying higher-resolution processing and encoder changes as future work.
6 Conclusion
- DetM-Tracker combines Compact Scalar Memory and Entropy-Gated refinement to mitigate mask bleeding and temporal drift in long industrial sequences.The paper reports boundary precision and identity continuity in complex high-density port arrays.Source: §6 · p. 9
- Future work targets higher frame resolution for ultra-fine structures and evaluates replacing the ViT-Encoder with a Swin Transformer.The proposed transition is intended to use shifted windowing for more efficient multi-scale feature extraction.Source: §6 · pp. 9–10
A.1 Hardware Infrastructure
- Training and inference ran on a high-performance cluster; hardware specifications are listed in Table A.1.The supplied text does not state the hardware specifications themselves.Source: §A.1 · p. 12, Table 5 · p. 12
A.2 Experimental Environment and Hyperparameters
The implementation uses PyTorch and a multi-stage training curriculum for convergence and temporal stability.
A.2 Experimental Environment and Hyperparameters
- The PyTorch implementation uses multi-stage training; the paper points to Table 6 for configurations and hyperparameters.The text says this regime enables the ViT-based backbone to capture industrial spatial-temporal patterns.Source: §A.2 · p. 12, Table 6 · p. 12
A.3 Evaluation Protocol and Interface Standardization
The comparison uses each baseline’s native input interface rather than imposing a unified input, aiming to represent each architecture’s peak performance. DetM-Tracker runs without prompt frames, while baselines use their specified prompt or mask initialization.
A.3 Evaluation Protocol and Interface Standardization
- Each baseline is evaluated through its native input interface, rather than a restrictive unified input, to represent its “optimal state.”The stated goal is to compare specialized industrial frameworks with foundation models’ general-purpose zero-shot capabilities.Source: §A.3 · p. 12
- DetM-Tracker operates autonomously without prompt frames (N=1).Source: §A.3 · p. 12
- SAM 2/3 use point or polygon clicks and text prompts, respectively, while Cutie and LiVOS use mask-based initialization.Source: §A.3 · p. 12
A.4 Multi-Strategy Benchmarking
The evaluation defines three prompt-frame sampling regimes to establish model performance envelopes. Primary results use Max-coverage for SAM 2, Cutie, and LiVOS, while Tracker and SAM 3 use single-run configurations.
A.4 Multi-Strategy Benchmarking
- The three sampling regimes are Random (lower bound), First-frame (VOS standard), and Max-coverage (upper bound).Source: §A.4 · p. 12
- Primary results report Max-coverage for SAM 2, Cutie, and LiVOS, but single-run configurations for Tracker and SAM 3.The authors describe Max-coverage as a user-realistic upper bound; per-strategy envelopes are in the supplementary material.Source: §A.4 · p. 12
- Table 7 identifies mIoU as the reported metric and N as the number of runs.Source: Table 7 · p. 13
A.5 Technical Considerations on Evaluation Rigor
The protocol fixes input resolution at 480 × 272 to align with real-time industrial latency constraints. It also sets a Hungarian-matching threshold of 0.1 to prioritize detector recall in high-density scenes.
A.5 Technical Considerations on Evaluation Rigor
- Input resolution is fixed at 480 × 272 to align evaluation with real-time industrial pipeline latency constraints.Source: §A.5 · p. 13
- Although SAM 2/3 favor 1024² inputs, evaluation uses the lower fixed resolution.Source: §A.5 · p. 13
- Hungarian matching uses a 0.1 threshold to prioritize detector recall where thin structures yield minimal spatial overlap.Source: §A.5 · p. 13
B Limitations and Discussion
The discussion identifies data scarcity, environmental generalization, and edge throughput as practical considerations. The authors curated a cable-pattern dataset, validated across cabinet configurations, and report approximately 60 ms per frame on an NVIDIA GB10.
B Limitations and Discussion
- Large open-source industrial datasets are scarce because infrastructure layouts are proprietary, and labeling thin structures in high-resolution video is resource-intensive.The authors address this by curating a high-fidelity dataset focused on intricate cable patterns.Source: B Limitations and Discussion · p. 13
- Validation covered multiple cabinet configurations with at least two distinct arrangements, alongside real-world downstream task demonstrations.The authors say these results indicate invariant geometric features rather than environment-specific patterns.Source: B Limitations and Discussion · p. 13
- On NVIDIA GB10 edge hardware, throughput is about 25% of benchmark results and latency is approximately 60 ms per frame.The authors state that 8 FPS, equivalent to a 125 ms latency budget, is sufficient for effective real-time inspection; reported latency remains within that threshold.Source: B Limitations and Discussion · p. 13