Source-linked AI summary
When to Adapt: Multi-Signal Domain Shift Detection for Efficient Training-Free Adaptation in Open-Vocabulary Segmentation
Michele Antonazzi, Alejandra C. Hernandez, José Araujo, Olov Andersson, Patric Jensfelt
TL;DR
Per-frame adapter updates add runtime costs, and sequential robotic data has been less explored. The paper combines complementary signals and temporal coherence to trigger training-free adaptation when needed; it reports maintained accuracy with fewer adaptations and improved inference speed.
Problem
Per-frame adapter retrieval and model updates add runtime and energy costs, while sequential robot data has received limited evaluation.
Source: §I · pp. 1–2Method
The approach combines three complementary shift signals and uses temporal coherence to trigger training-free adaptation only when a domain shift is detected.
Source: §I · p. 2Results
3.3× FPS while maintaining accuracy; compared with SemLA, the method has a 1.4 mIoU-point drop and triggers adaptation only 0.8% of the time.
Source: B. Results on training-free continual… · p. 6, §VI · p. 8Takeaways & Limitations
The authors conclude that the approach makes training-free adaptation practical for long-term robotic deployment.
Source: §I · p. 2
Abstract
from arXiv · showhide
Robust and reliable perception is essential for autonomous robots operating in real-world environments, particularly in long-term missions where environmental conditions may change significantly over time. Although recent advances in Visual Foundation Models (VFMs) have improved open-vocabulary semantic segmentation, these models can still suffer from domain shift, which can significantly degrade performance if they are not adapted to the current environment. Training-free domain adaptation is a relevant paradigm for adaptation, consisting of adjusting the model online using lightweight adapters. Recent approaches apply this on a per-frame basis, which is impractical for deployments on resource-constrained robotic hardware. To tackle this, we propose a multi-signal domain shift detection method for training-free continual test-time adaptation (TF-CTTA) in open-vocabulary segmentation. Our method leverages temporal coherence across consecutive frames by monitoring and combining complementary aspects of domain shift (visual change, adapter mismatch, and semantic drift) to trigger adaptation only when needed. We validate our approach on a benchmark including indoor and outdoor environments and using real robotic data. We demonstrate that our approach maintains segmentation accuracy while substantially reducing adaptations, making training-free adaptation practical and feasible for long-term, real-world robotic deployments.
I. INTRODUCTION
Robots need reliable perception across changing environments, but domain shift can degrade segmentation and per-frame adaptation adds deployment costs. The paper proposes combining domain-shift signals to trigger adaptation only when needed, and evaluates the approach on robotics-focused benchmarks and real-world sequences.
I. INTRODUCTION
- Changing scene types, appearances, lighting, or layouts can reduce VFM accuracy under domain shift, making adaptation relevant during deployment.The motivation includes long-term robotic missions across indoor and outdoor settings, new buildings, and varying illumination and weather.Source: §I · p. 1
- Per-frame adapter retrieval and model updates increase energy consumption and runtime overhead, while sequential robot data has received limited evaluation.The paper identifies these as practical limitations of SemLA in robotic deployment.Source: §I · pp. 1–2
- The method exploits temporal coherence to trigger adaptation only when a domain shift is detected, reducing adaptation events by two orders of magnitude while maintaining segmentation accuracy.The paper positions this as making training-free adaptation practical for long-term robotic deployment.Source: §I · p. 2
- The detector combines three complementary signals to reduce per-frame adaptation overhead while maintaining semantic segmentation quality.The signals monitor visual change, adapter mismatch, and semantic drift, and are fused hierarchically to trigger adaptation when multiple signals detect a shift.Source: §I · p. 2
- The authors propose a robotics-focused continual test-time adaptation benchmark that captures alternating and temporally correlated domain shifts.They also report evaluation on real-world robotic sequences across indoor and outdoor environments and benchmark inference on NVIDIA Jetson hardware.Source: §I · p. 2
II. RELATED WORK
Prior CTTA methods can impose computational and memory demands, while training-free adaptation avoids gradient-based updates by selecting and merging pretrained adapters. The paper positions its detector as a way to address the runtime limits of per-frame adaptation and the saturation of single-signal shift detection.
Continual test-time adaptation
- CTTA adapts online without source data, but teacher-student and other backpropagation-based approaches can burden robotic deployment with inference-speed and memory costs.The cited approaches may require backpropagation and weight sharing for EMA accumulation between large architectures.Source: §II · p. 2
Training-free continual test-time adaptation
- SemLA selects and merges relevant LoRA-based adapters using CLIP embeddings, but updating the model every frame compromises real-time inference.Training-free continual adaptation combines pretrained models relevant to the current domain without additional training.Source: §II · p. 2
Domain shift detection
- Single-indicator detectors such as entropy and feature statistics can saturate quickly and miss some types of shift in long-term robotic sequences.The paper contrasts these with its use of multiple complementary signals for different shift types.Source: §II · p. 3
Positioning our work
- The paper brings training-free continual adaptation to real-world robotic image streams by using temporal coherence to trigger adaptation only when a shift is detected.The stated aim is improved efficiency while maintaining segmentation accuracy.Source: §II · p. 3
III. PRELIMINARIES
The paper formulates open-vocabulary segmentation and training-free continual adaptation for robots operating across target environments. It then describes SemLA’s adapter library and its image-conditioned retrieval and merging process.
A. Problem formulation
- Robots segment RGB image sequences across target environments using deployment-time object categories that may differ from the source-training categories.The source model maps images to dense masks, with pixels assigned to object categories.Source: A. Problem formulation · p. 3
- TF-CTTA adapts a source-pretrained model online without target labels or backpropagation-based fine-tuning.The model evolves over a stream of target images to improve performance in domains encountered during deployment.Source: A. Problem formulation · p. 3
B. Training-free adaptation via adapters merging
- SemLA retrieves the top-K adapters whose CLIP centroids are closest to each test-image embedding, weights them by normalized distances, and merges them into the pretrained model before inference.Each dataset descriptor is the average of its CLIP image features, and each adapter is a low-rank adapter trained on that dataset.Source: B. Training-free adaptation via… · p. 3
IV. MULTI-SIGNAL DOMAIN SHIFT DETECTION
The detector monitors complementary aspects of domain shift and fuses their signals hierarchically to avoid costly per-frame adaptation. Its visual-change signal tracks temporal trends in CLIP-feature proximity and uses the trend to identify a changing domain.
IV. MULTI-SIGNAL DOMAIN SHIFT DETECTION
- The detector monitors visual change, whether active adapters still match the domain, and semantic drift, then fuses the signals to trigger adaptation when substantial evidence accumulates.This design addresses per-frame model updates in robotic settings where perceptions within an environment are highly correlated.Source: §IV · p. 3
A. Visual change signal (S1)
- The visual-change signal computes CLIP embeddings for incoming frames and measures their proximity to an exponentially averaged feature memory.The memory uses an EMA with α = 10^-2, and proximity is measured as one minus cosine similarity.Source: A. Visual change signal (S1) · p. 4
- An increasing trend in the proximity signal suggests domain change; the method smooths the signal, estimates its derivative, and accumulates derivatives with an EMA.The method marks visual change when the derivative is substantially above a positive derivative memory, then waits until that memory becomes negative.Source: A. Visual change signal (S1) · p. 4
B. Adapter mismatch signal (S2)
S2 measures whether the active adapters still match the current domain, while S3 tracks semantic change relative to contexts associated with prior adaptation. Together, these signals distinguish adapter mismatch from changes in visual embeddings alone.
B. Adapter mismatch signal (S2)
- S2 compares the current domain’s mean CLIP representation over the last w frames with the top-K loaded adapters using the lowest inverse cosine distance.Its score is normalized by the value recorded at the first frame after adaptation, making τ2 independent of absolute embedding distances.Source: B. Adapter mismatch signal (S2) · p. 4
C. Semantic drift dignal (S3)
- S3 captures semantic change that image-embedding signals may miss, asking whether predicted scene content has diverged from contexts associated with active adapters.A semantic descriptor combines normalized pixel frequencies for predicted object classes with their precomputed CLIP text embeddings.Source: C. Semantic drift dignal (S3) · p. 4
- S3 uses a memory of semantic contexts from prior adaptation events, with embedding drift measuring distance to relevant past contexts and class novelty detecting previously absent categories.The memory stores semantic descriptors and frequently predicted classes; J-nearest neighbors help prevent saturation in long sequences.Source: C. Semantic drift dignal (S3) · p. 5
D. Hierarchical Fusion
The hierarchical fusion combines three shift signals to trigger adaptation. Its logic treats abrupt visual change differently from cases without such a change.
D. Hierarchical Fusion
- When S1 indicates an abrupt visual change, adaptation is triggered if either S2 or S3 also exceeds its threshold; when S1 is silent, both must exceed theirs.The signals are combined through a hierarchical logic.Source: D. Hierarchical Fusion · p. 5
- S1 is binary and highly sensitive, so a minor CLIP-embedding variation can activate it; the hierarchy therefore requires another signal to confirm when S1 fires.A person approaching the camera is given as an example of a minor variation.Source: D. Hierarchical Fusion · p. 5
- After a trigger, the method retrieves the top-K adapters using mean CLIP embeddings over the last w frames rather than SemLA’s single-frame descriptor.The authors describe the multi-frame descriptor as more robust.Source: D. Hierarchical Fusion · p. 5
A. Experimental setup and benchmark design
The benchmark combines indoor and outdoor datasets and tests adaptation through sequences of domains, including leave-one-out evaluation on unseen scenes. It measures segmentation accuracy and adaptation frequency against several triggering strategies.
A. Experimental setup and benchmark design
- The benchmark combines ScanNet and ScanNet++ indoors with ACDC and MUSES outdoors, covering distinct environments, weather, and illumination conditions.ScanNet and ScanNet++ use 707 and 906 environment domains respectively; ACDC has five weather or illumination domains, while MUSES has eight.Source: A. Experimental setup and benchmark… · p. 5
- Evaluation uses 16 randomly chosen domains in alternating sequences, discards the corresponding trained adapter to test unseen scenes, and repeats across 50 sampled sequences.The label space combines the object categories from all four datasets.Source: A. Experimental setup and benchmark… · p. 5
- The setup measures segmentation accuracy with mIoU and efficiency by adaptation frequency per domain, comparing zero-shot, entropy, DSS, periodic, and per-frame SemLA strategies.Periodic adaptation is set to every 120 frames; DSS uses batches of 50 frames, while SemLA adapts every frame.Source: A. Experimental setup and benchmark… · pp. 5–6
B. Results on training-free continual adaptation
The proposed method keeps segmentation accuracy close to per-frame adaptation while triggering far fewer adaptations. It also performs well across benchmark splits and captures shifts that entropy and feature-statistic baselines miss.
B. Results on training-free continual adaptation
- A 1.4 mIoU-point drop from SemLA accompanies adaptation on only 0.8% of the time.SemLA performs per-frame adaptation and represents a performance plateau.Source: B. Results on training-free continual… · p. 6
- The method achieves the best segmentation accuracy in 13 of 16 splits; Periodic leads by 0.3–0.5 mIoU in EI3, EI10, and EI15.Periodic's adaptation frequency was set to mimic the proposed method's efficiency on this benchmark.Source: B. Results on training-free continual… · p. 6
- +4.4 mIoU over zero-shot and a 0.4 mIoU drop from per-frame adaptation are reported in out-of-distribution camera tests, with 3.2 versus 960 adaptations.The tests use DSLR data for ScanNet++ and outdoor ACDC scenes.Source: Table II · p. 6, B. Results on training-free continual… · pp. 6–7
- Entropy saturates after a few early triggers, while DSS detects major indoor–outdoor shifts but misses smaller within-environment shifts that the proposed method captures.The authors attribute entropy's saturation to the absence of online fine-tuning, which would otherwise progressively increase model confidence on current data.Source: B. Results on training-free continual… · p. 6
- Across 2,735 Odin frames, 12 of 42 S1 activations were confirmed adaptations, with the domain described as consistent between triggers.The sequence covers indoor and outdoor environments on a university campus.Source: Figure 3 · p. 7
C. Signal analysis
The signal analysis shows that combining visual change, adapter mismatch, and semantic drift balances segmentation accuracy against adaptation frequency. Removing a signal can either miss gradual drift or trigger excessively.
C. Signal analysis
- NA = 4.1 and mIoU 44.7 give the full system the best accuracy-to-adaptation trade-off, within 1.3 mIoU of per-frame adaptation (46.0).It triggers 3.5× fewer adaptations than S2 + S3.Source: C. Signal analysis · p. 7
- S1 + S2 has NA = 1 and mIoU 42.9, while S2 + S3 reaches mIoU 45.5 but over-triggers at NA = 14.4 without visual change.The authors report that S1 + S2 misses gradual semantic drift, while adding visual change helps limit triggers.Source: C. Signal analysis · p. 7
- Alone, S1 and S3 trigger frequently (NA = 10.8 and 12.4), whereas S2 barely triggers (NA = 1) as adapter mismatch builds gradually.S1 + S3 yields NA = 3.1 and mIoU 43.5; the passage says its improvement is marginal.Source: C. Signal analysis · p. 7
D. Real-World Deployment
On Jetson AGX Orin, adaptation is the deployment bottleneck; in a real-world Odin sequence, 12 adaptations added 7 seconds of overhead versus 25 minutes for per-frame adaptation.
D. Real-World Deployment
- Fig. 4 compares segmentation accuracy with adaptation count across 50 sequences of eight domains, reporting mIoU and NA for each signal combination.Source: Figure 4 · p. 7
- Adaptation averaged 550 ms at full power and 700 ms at low power; per-frame adaptation reduced throughput by 1.6–3.3×.With the small model, triggered adaptation reached 4.2 FPS at full power, 3.2× the per-frame throughput.Source: D. Real-World Deployment · p. 7, Table III · p. 8
- 12 adaptations added only 7 seconds of overhead on the Jetson, compared with 25 minutes under per-frame adaptation.The Odin sequence comprised 2,735 processed frames; 71% of S1 activations were rejected as false positives during stable periods.Source: D. Real-World Deployment · p. 8
VI. CONCLUSIONS
- 3.3× FPS while maintaining accuracy is the paper’s reported efficiency gain from triggered adaptation.The pipeline profiles on a Jetson AGX Orin; the conclusion reports the gain without specifying a model size or power mode.Source: §VI · p. 8
- Future work will investigate adapter indexing and merging strategies to improve segmentation accuracy and further reduce adaptation cost.Source: §VI · p. 8