PaperlayerView exact revision on arXiv

Source-linked AI summary

TM-APR: Thermal Temporal-Memory Localization via Analytic Online Adaptation

Yanshuo Bai, Kanji Tanaka

arXiv:2609.26766v1cs.RO

TL;DR

Thermal VPR faces environmental dependence and dynamic non-linear shifts, while standard ACIL's linear assumptions leave it vulnerable to extreme fluctuations. TM-APR embeds control-theoretic extensions in ACIL's analytic update loop; across three benchmark datasets, the complete system is reported to outperform existing state-of-the-art methods.

  • Problem

    Thermal VPR faces environmental dependence and dynamic non-linear shifts, and standard ACIL's linear updates are vulnerable to extreme thermal fluctuations.

    Source: §I · p. 1

  • Method

    TM-APR maps ACIL's Woodbury-based updates to state estimation and embeds U-ACIL, GMM-ACIL, and H∞-ACIL into the analytic update loop.

    Source: §I · p. 1

  • Results

    Across three benchmark datasets, TM-APR is reported to consistently outperform existing state-of-the-art methods.

    Source: §VII · p. 8

  • Takeaways & Limitations

    TM-APR couples continuous APR with selective reference-place corrections using analytic adaptation and confidence-aware fusion for smooth, drift-free trajectory estimates.

    Source: §VII · p. 8

    Standard ACIL's linear assumptions leave it vulnerable to extreme non-linear thermal fluctuations that degrade latent geometry and reintroduce retrieval ambiguity.

    Source: §I · p. 1

Abstract

from arXiv · show

Thermal Visual Place Recognition (Thermal VPR) maps camera observations to metric poses within a mapped environment, serving as a prerequisite for autonomous navigation. However, thermal VPR suffers from severe environmental dependence, heavy online retraining overheads, and an inability to model dynamic non-linear shifts, causing existing frameworks to fail during online deployment. To achieve robust domain-invariant place recognition, we bridge Analytic Class-Incremental Learning (ACIL) with domain-invariant VPR for the first time, revealing that its gradient-free matrix updates construct a surprisingly strong baseline that outperforms conventional fine-tuning. Nevertheless, standard ACIL exhibits a critical vulnerability to extreme non-linear thermal fluctuations due to its structural linear assumptions. To overcome this limitation, we exploit a novel algebraic equivalence between ACIL and modern control theory, proposing a framework which embeds Unscented propagation (U-ACIL), Gaussian Mixture partitioning (GMM-ACIL), and minimax $H_\infty$ optimization ($H_\infty$-ACIL) directly into the update loop. Our formulation guarantees exact closed-form matrix updates within $\mathcal{O}(1)$ computational complexity, bypassing backpropagation to ensure that the online update latency ($\Delta t_{\mathrm{learn}}$) remains strictly bounded below the sensor acquisition interval ($\Delta t_{\mathrm{acquire}}$), thereby eliminating trajectory jumps in real-time SLAM pipelines.

I. INTRODUCTION

Thermal VPR is vulnerable to environmental and non-linear thermal shifts that can produce false-positive retrievals and trajectory jumps. The paper introduces analytic, control-theoretic extensions to ACIL for online thermal localization, targeting robust adaptation without backpropagation.

I. INTRODUCTION

  • Thermal VPR faces environmental dependence, costly online retraining, and dynamic non-linear shifts that can distort representations and cause false-positive retrievals and trajectory jumps.Existing frameworks rely largely on offline batch training or fine-tuning and fail to adapt to dynamic non-linear shifts during online deployment.Source: §I · p. 1
  • ACIL provides a gradient-free, closed-form analytic baseline for thermal VPR that outperforms conventional gradient-based fine-tuning and state-of-the-art baselines.The paper presents this as the first bridge between ACIL and domain-invariant VPR.Source: §I · p. 1
  • Standard ACIL remains vulnerable to extreme non-linear thermal fluctuations because its linear updates can degrade latent geometry and reintroduce retrieval ambiguity.These shifts violate ACIL's structural assumptions in dynamic environments.Source: §I · p. 1
  • TM-APR maps ACIL's Woodbury-based updates to state estimation and incorporates U-ACIL, GMM-ACIL, and H∞-ACIL to address local non-linearities, multimodal variation, and worst-case noise.The extensions are embedded directly in the analytic update loop.Source: §I · p. 1
  • The formulation uses exact closed-form O(1) matrix updates without backpropagation, with online adaptation latency bounded below the sensor acquisition interval.The authors state that this fixed-time update property eliminates trajectory jumps during continuous environmental transitions in real-time SLAM.Source: §I · p. 1

II. RELATED WORKS

  • The paper validates the complete localization pipeline on MS2, STherO Valley, and a physical robot dataset, reporting retrieval accuracy, trajectory continuity, and real-time efficiency.The related-work section frames this contribution against thermal localization and analytic-learning methods.Source: §I · p. 2, §II · p. 2

A. Thermal Visual Place Recognition and Localization

Prior thermal localization methods use varied visual, thermal, and analytic-learning approaches, while static thermal representations remain vulnerable to dynamic variation. TM-APR connects ACIL's analytic updates to state estimation and keeps fixed reference memory separate from the online adaptation state.

A. Thermal Visual Place Recognition and Localization

  • Thermal localization has explored uncertainty modeling, temporal and geometric methods, attention, transformers, thermal correction, thermal-inertial alignment, and stereo tracking.ThermalLoc combines EfficientNet and ViT, and AnyThermal distills visual foundation models.Source: A. Thermal Visual Place Recognition… · p. 2

B. Analytic Incremental Learning in Machine Learning

  • ACIL uses gradient-free closed-form classifier updates, with extensions spanning kernels, dual-stream compensation, time series, class-imbalanced driving, and federated learning.The cited methods are described as achieving exact joint-learning equivalence with zero backpropagation and O(1) complexity.Source: B. Analytic Incremental Learning in… · p. 2

C. Motivation and Key Insight

  • Standard ACIL's linear ridge regression over frozen features struggles with non-linear feature overlaps, multimodal shifts, and thermal noise.The paper links this degradation under thermal shifts to state estimation under non-Gaussian noise and introduces U-, GMM-, and H∞-ACIL while retaining O(1) streaming adaptation.Source: C. Motivation and Key Insight · p. 2

A. Problem Formulation and Information Flow

  • TM-APR estimates planar position from each incoming thermal image and maintains fixed coordinate-bearing reference memory separately from its adapting classifier state.The adaptation state includes classifier weights, sufficient statistics, and class-conditional models; new observations can refine decision boundaries without expanding the reference footprint.Source: A. Problem Formulation and Information… · p. 2

B. Thermal Absolute Pose Regression

The APR branch estimates pose from frozen thermal features, while the place-retrieval branch supports independent reference-based corrections. Online analytic adaptation discretizes spatial classes and uses global class ranking to constrain local reference search.

B. Thermal Absolute Pose Regression

  • The APR branch uses a frozen AnyThermal ViT-B/14 encoder and trainable Transformer and regression heads to predict planar coordinates and heading.The heading is represented as a unit-normalized two-component vector, avoiding angle wrap-around discontinuities; the backbone remains frozen during training.Source: B. Thermal Absolute Pose Regression · p. 2

C. Analytic Place-Context Adaptation

  • TM-APR separates the spatial APR head from SALAD-based retrieval so cached descriptors can be evaluated independently of pose inference.The analytic place-context adaptation uses reference-anchored grid cells for spatial classes and maintains K, Q, and W for closed-form updates; global class ranking constrains local cosine search within fixed reference memory.Source: B. Thermal Absolute Pose Regression · pp. 2–3, C. Analytic Place-Context Adaptation · p. 3

D. Complementary U, G, and H Components

The U, G, and H extensions add complementary mechanisms for thermal feature sensitivity, multimodal variation, and disturbance attenuation within analytic adaptation. They target selective, noise-resilient classification without heavy geometric verification or exhaustive frame storage.

D. Complementary U, G, and H Components

  • U-ACIL evaluates prediction logits over deterministic symmetric sigma points along the primary feature-variation direction to model sensitivity under thermal noise.Perturbed features are integrated into the ACIL correlation matrix with covariance weights during memory accumulation.Source: D. Complementary U, G, and H Components · p. 3
  • GMM-ACIL incrementally models each spatial class with diagonal Gaussian mixtures, using likelihoods and candidate-local grouping to distinguish spatial modes from isolated high-similarity false positives.Running means and diagonal covariances capture multimodal feature distributions and complement global descriptor similarity and raw class predictions.Source: D. Complementary U, G, and H Components · p. 4
  • H∞-ACIL scales incoming updates with residual attenuation to bound worst-case disturbance effects and mitigate unbounded noise accumulation and state saturation.The update weighting uses target and predicted responses, a residual-energy scale, and an attenuation floor; candidate updates also require joint local-support and score-margin criteria.Source: D. Complementary U, G, and H Components · p. 4

E. From a Retrieved Place to a Position Observation

Retrieved references provide gated spatial observations, while Confidence-Aware Position Fusion (CAPF) combines APR estimates with accepted loop corrections in a stable temporal filter.

E. From a Retrieved Place to a Position Observation

  • A retrieved reference supplies a spatial observation and appearance score, but multi-stage gates check memory validity, detection confidence, appearance similarity, and coordinate bounds before correction.A residual-distance metric bounds spatial consistency; a local continuity check also tests the candidate against the most recent accepted position reference.Source: E. From a Retrieved Place to a… · p. 4
  • The continuity check scales allowable displacement with the APR-estimated trajectory interval, rejecting spurious jumps even when descriptor similarity is exceptionally high.This adaptive spatial validation keeps topological corrections aligned with physical vehicle dynamics.Source: E. From a Retrieved Place to a… · p. 4
  • Attenuation maps geometric-descriptor consistency into the state estimator’s covariance scheme, while candidate selection varies by dataset to optimize retrieval throughput.STherO uses top-eight filtering; MS2 and the physical robot dataset use direct top-candidate assignment, without computationally heavy top-K averaging.Source: E. From a Retrieved Place to a… · p. 4

F. Confidence-Aware Position Fusion

  • CAPF tracks planar position and velocity, and applies measurement updates for either observation type.A high-rate APR position update uses covariance Ra = raI2; an eligible loop candidate supplies a secondary reference correction with dynamic observation covariance.Source: F. Confidence-Aware Position Fusion · p. 4
  • The Joseph-form covariance update preserves numerical stability and positive semi-definiteness; without valid reference corrections, the filter propagates motion using high-frequency APR predictions.Integrating APR preconditioning and temporal filtering blends continuous regression with discrete topological corrections in real time.Source: F. Confidence-Aware Position Fusion · p. 4

G. Deterministic Fixed-Time Online Adaptation

The online adaptation is designed for fixed-time execution, using closed-form updates and constant-complexity operations rather than backpropagation.

G. Deterministic Fixed-Time Online Adaptation

  • Online adaptation has strict O(1) fixed-time execution, with learning latency much smaller than the sensor acquisition interval.Woodbury updates use fixed-size matrices; U-ACIL evaluates three sigma points, GMM-ACIL maintains statistics over M ≪ C mixtures, H∞-ACIL applies algebraic clipping, and CAPF uses a 4×4 filter.Source: G. Deterministic Fixed-Time Online… · p. 4

IV. EXPERIMENTAL PROTOCOL

The protocol evaluates localization across three thermal settings with separated sequence roles and fixed reference memory. It specifies the model setup, trajectory alignment, and framewise error measures.

A. Datasets and Sequence Roles

  • Real-UGV-Campus, MS2, and STherO Valley represent indoor, large-scale outdoor, and dynamic valley scenarios, respectively.MS2 sequences 1–3 provide 61,788 APR training frames and sequence 4 provides 10,131 validation frames; the other datasets have 1,874 and 4,166 test frames, respectively.Source: A. Datasets and Sequence Roles · p. 5
  • Training images optimize regression weights, while loop corrections use only designated reference descriptors and matching is restricted to initial references.The specified adaptation order uses sequence 3 (L) before sequence 4 (L) on MS2 and sequence 5 before sequence 6 on Real-UGV-Campus.Source: A. Datasets and Sequence Roles · p. 5

B. Implementation and Evaluation Scope

  • The APR model uses frozen AnyThermal features with lightweight trainable pose layers, and backend fusion parameters can be tuned without reoptimizing the visual backbone.Hyperparameters were tuned by coarse grid search on independent validation sets; the encoder yields 768-dimensional tokens processed by a two-layer Transformer and a 768-to-512 MLP.Source: B. Implementation and Evaluation Scope · p. 5
  • Trajectory inputs are planar-aligned to ground truth before fusion, and tracking performance is evaluated using framewise errors, mean error, and median error.The alignment removes static coordinate offsets to isolate estimation errors; the three metrics balance localized outlier penalty with nominal trajectory precision.Source: B. Implementation and Evaluation Scope · p. 5

C. Loop-Module and Complete-System Comparisons

The evaluation covers loop-candidate filtering, complete-system configurations, and localization accuracy across three datasets. TM-APR achieves the lowest reported RMSE on all three, with sub-meter error on Real-UGV-Campus and a large reduction on MS2.

C. Loop-Module and Complete-System Comparisons

  • The real-world visualization shows a UGV trajectory through indoor corridors with texture-scarce hallways and varying thermal signatures from windows and lighting.Table I identifies sequence roles and states that reference memory remains fixed during full-system evaluation.Source: Figure 3 · p. 5, Table I · p. 5
  • The STherO loop-module ablation tests candidate detection with a 10 m spatial radius, 0.82 feature-similarity threshold, and 120-frame minimum temporal separation.Its eight-way ablation evaluates complementary U-, G-, and H-extension impacts across 125 spatial place classes; global AnyThermal-VPR scores provide context.Source: C. Loop-Module and Complete-System… · p. 5
  • The MS2 complete-system ablation compares four progressive TM-APR configurations, from the full context-gated pipeline to variants omitting loop corrections or using unguided retrieval.Because existing baselines are tightly coupled to their original features, the protocol compares the standalone AnyThermal backbone directly against the full proposed framework.Source: C. Loop-Module and Complete-System… · p. 5

A. Localization Accuracy Across Three Datasets

  • 0.96 m RMSE on Real-UGV-Campus and 13.53 m vs. 28.45 m on MS2; TM-APR achieves the lowest RMSE across all three datasets.The reported MS2 comparison corresponds to over 52% RMSE reduction; Table II reports planar localization errors in meters, with lower values better.Source: Table II · p. 6, A. Localization Accuracy Across Three… · p. 6

B. Qualitative Trajectory Comparison

TM-APR’s full estimation pipeline preserves trajectory continuity and accuracy across the evaluated environments. Back-end fusion consistently reduces localization RMSE relative to raw APR predictions.

B. Qualitative Trajectory Comparison

  • TM-APR tracks tight turns and complex loops in Real-UGV-Campus, preserves smooth topologically correct trajectories on MS2, and avoids catastrophic divergence in STherO Valley.The reported comparisons describe qualitative trajectory reconstructions against baseline methods.Source: B. Qualitative Trajectory Comparison · p. 6

C. Effect of Fusion Relative to Its APR Input

  • 41.6%, 29.1%, and 9.7% RMSE reductions on Real-UGV-Campus, MS2, and STherO Valley show consistent fusion gains.Table III compares raw APR predictions with fused outputs on identical evaluation streams.Source: Table III · p. 6, C. Effect of Fusion Relative to Its… · p. 6
  • In homogeneous STherO Valley thermal scenes, confidence-aware fusion selectively uses high-quality retrievals to guide the regressor and prevent error accumulation.The authors attribute the performance improvement across datasets to the framework’s versatility.Source: C. Effect of Fusion Relative to Its… · p. 6

D. Loop-Closure Component Ablation

On STherO Valley, the full TM-APR loop-closure system improves over baseline ACIL, while qualitative MS2 examples show more accurate retrieval under thermal ambiguity. Spatial feature responses concentrate on persistent geometry.

D. Loop-Closure Component Ablation

  • Adding U, H, and G individually raises STherO Valley F1 from 0.6474 to 0.6525, 0.6617, and 0.6721, respectively.The standalone VPR evaluation excludes back-end CAPF state estimation; the modules add O(1) matrix operations during analytic updates.Source: D. Loop-Closure Component Ablation · p. 6
  • TM-APR achieves Precision 0.5419 and F1-score 0.6845, a +0.0371 absolute F1 improvement over baseline ACIL.Table IV reports the STherO Valley loop-closure ablation; the full system includes U, G, and H∞.Source: D. Loop-Closure Component Ablation · p. 6

E. Qualitative Loop-Closure Example

  • Fig. 6 shows high spatial feature activations on static geometry and lower responses to transient thermal noise, supporting geometrically consistent Top-1 retrievals.The displayed rows pair query thermal images, edge responses, spatial response maps, and retrieved Top-1 images; the response maps are non-probabilistic.Source: Figure 6 · p. 7, E. Qualitative Loop-Closure Example · p. 7
  • On MS2, AnyThermal-VPR retrieves a false match 43.94 m away, whereas TM-APR identifies the reference location 1.57 m away.The comparison attributes the correct retrieval to preserving fine-grained structural cues such as building facades, roadside poles, and road geometry.Source: E. Qualitative Loop-Closure Example · p. 7, D. Loop-Closure Component Ablation · p. 7

F. Complete-System Ablation

The MS2 ablation shows progressively lower RMSE as topological retrieval and then the full context-gated pipeline are added. TM-APR combines online analytic adaptation with temporal filtering and selective reference corrections, with reported real-time efficiency and evaluations across three benchmarks.

G. Computational Efficiency

  • 44.237 ms/frame (22.61 Hz, P95: 47.539 ms) is the complete online cycle with ACIL update; ACIL inference and one-sample update take 5.410 and 2.779 ms, respectively.Measurements used an NVIDIA GeForce GTX 1080 GPU, FP32, batch size 1, and 224 × 224 TIR images; fixed feature dimensions and class counts yielded runtime invariant to frame count.Source: G. Computational Efficiency · p. 7

A. Decoupled Reference Memory and Online Adaptation

  • Decoupling reference coordinate memory M0 from adaptation state At lets closed-form updates refine local decision boundaries without changing the fixed coordinate footprint or corrupting the reference database.The analytical updates have O(1) computational complexity.Source: A. Decoupled Reference Memory and… · p. 8
  • When unmapped regions yield no valid topological reference candidates, spatial gating rejects invalid or low-confidence corrections and falls back to APR updates and motion propagation.The stated outcome is trajectory continuity without state divergence.Source: A. Decoupled Reference Memory and… · p. 8

B. Robustness and Real-Time Operational Efficiency

  • Multi-tier gating combines innovation bounds, local spatial continuity, and H∞ residual attenuation, while closed-form updates and analytic context modeling replace backpropagation and exhaustive geometric verification.The authors frame this design for resource-constrained autonomous platforms such as UGVs in challenging thermal environments.Source: B. Robustness and Real-Time… · p. 8

VII. CONCLUSION

  • Across three benchmark datasets, TM-APR’s quantitative and qualitative evaluations consistently outperform existing state-of-the-art methods.The conclusion presents the framework as a robust and efficient solution for thermal-visual navigation.Source: §VII · p. 8
Loading 2609.26766v1…