PaperlayerView exact revision on arXiv

Source-linked AI summary

GAFT: Geo-Anchored Fine-Tuning for Hazard Identification from Rare Failures

Yanran Xu, Chuanhang Qiu, Yue Wang, Wenbo Wu, Zhaoxing Li

arXiv:2608.30858v1cs.ROcs.CV

TL;DR

Off-road hazards are difficult to identify because failures are rare and labels do not reveal which visual cues indicate physically dangerous structures. GAFT adapts a vision foundation model using a geometry-derived attention prior while preserving pretrained representations, improving repeated-LOSO hazard identification over frozen DINOv2 and supervised PEFT baselines.

  • Problem

    Rare failure labels associate frames with outcomes but do not identify the visual evidence responsible for hazardous interactions, encouraging scenario-specific cues.

  • Method

    GAFT aligns spatial attention rollout with a proximity-weighted geometric prior during LoRA adaptation and uses a representation anchor to limit drift from pretrained features.

  • Results

    GAFT improves repeated-LOSO mean F2 from 0.0607 for frozen DINOv2 to 0.3757 across ten independently initialized adaptations.

  • Takeaways & Limitations

    Geo-anchored adaptation can improve RGB-only hazard identification when hazard labels are scarce and adaptation remains anchored to pretrained features.

  • Takeaways & Limitations

    The benchmark covers two verified forest hazard morphologies and evaluates offline RGB identification rather than arbitrary environments or closed-loop navigation outcomes.

Abstract

from arXiv · show

Off-road navigation can fail when physical structures induce irrecoverable states such as high-centering or entrapment, requiring human interventions. Identifying these structures is crucial, yet challenging. Such failure events are rare and costly to collect, resulting in limited training data. Moreover, the collected data associate frames with outcomes, but do not indicate the visual cues responsible for the failure. Learning directly from these data can therefore exploit scenario-specific visual cues, leading to poor generalization. We propose \textbf{Geo-Anchored Fine-Tuning (GAFT)}, a parameter-efficient method that adapts a vision foundation model with a geometry-derived prior. It guides LoRA adaptation by aligning a spatial attention-rollout map with the geometry prior, while preserving pretrained representations. On an intervention-verified forest hazard benchmark, across ten independently trained adaptations, GAFT consistently outperforms frozen DINOv2 and supervised PEFT baselines, improving the repeated leave-one-scenario-out mean $F_2$ from 0.0607 to 0.3757 with statistical significance under paired analysis. Within these independently trained models, the best-performing GAFT model achieves a repeated-LOSO $F_2$ of 0.570. Code and benchmark: https://github.com/Xu-Yanran/geo_anchored_fine_tuning

1 Introduction

Hazard identification requires reasoning about how structures interact physically with a robot, but rare, outcome-only failures provide too little evidence about the responsible visual cues. GAFT addresses this gap by geometry-guided, parameter-efficient adaptation of a vision foundation model and improves cross-scenario performance.

  • Hazards are structures whose geometry can drive a ground robot into irrecoverable states such as high-centering, entrapment, or sensor damage.
  • Rare, costly hazard events provide coarse outcome labels without identifying the image regions responsible for hazardous interactions.
  • Generic visual pretraining captures broad structure but does not specifically prioritize features predicting physical consequences of robot–terrain interactions.
  • GAFT uses RGB–depth data to build a proximity-weighted geometric prior, aligns attention rollout with it, and anchors representations during LoRA adaptation.
  • GAFT consistently outperforms frozen DINOv2 and supervised PEFT baselines on an intervention-verified benchmark, with statistical significance under paired analysis.

2 Related Work

Related work combines geometry, visual features, and robot-derived supervision for traversability, but sparse hazard labels remain insufficient to identify interaction-relevant structures. GAFT builds on foundation models and geometry-guided adaptation to address this limitation.

  • 2 Related Work: Geometry-based navigation captures slope, clearance, and structural properties, while vision-based methods learn terrain properties from visual or semantic features.
  • 2 Related Work: Robot motion and proprioceptive measurements provide platform-specific supervision for visual predictions of traction, mobility, or collision outcomes.
  • 2.2 Learning from Sparse Failure Data: Sparse binary hazard labels associate frames with failures but do not identify relevant image regions, allowing texture, illumination, or background correlations to transfer poorly.
  • 2.2 Learning from Sparse Failure Data: Few-shot and meta-learning methods assume related tasks during meta-training, whereas hazardous structures vary with pose, terrain, and platform capability.
  • 2 Related Work: Pretrained vision foundation models and parameter-efficient methods such as LoRA, visual prompt tuning, and AdaptFormer reduce the need for task-specific training from scratch.
  • 2 Related Work: Geometry-guided adaptation uses geometric information for distinct purposes, including multi-view consistency, 3D awareness, and related foundation-model adaptation objectives.

3 Methodology

GAFT uses unlabeled RGB–depth traversal data to guide parameter-efficient vision foundation model adaptation with a proximity-weighted geometric prior, then trains a linear hazard probe on scarce labeled RGB frames. The method aligns spatial attention with geometry while preserving pretrained representations and deploys using RGB only.

  • 3.1 Problem Formulation and Overview: GAFT uses two complementary datasets: temporally matched RGB–depth traversal pairs without hazard annotations and a much smaller labeled hazard set.The traversal data guide adaptation, while the hazard set supports the downstream classifier.
  • 3.2 Proximity-Weighted Geometric Prior: Stage 1 constructs a proximity-weighted geometric prior from depth relative to an estimated ground plane and uses it to guide LoRA adaptation of DINOv2-ViT-S/14.All original backbone parameters remain frozen during this adaptation.
  • 3.4 Hazard Probe and RGB-Only Inference: Stage 2 freezes the adapted backbone, pools final patch tokens with rollout weights, and trains a linear hazard probe on labeled RGB frames.Deployment uses RGB only, and the benchmark’s red boxes are visual annotations rather than training supervision.
  • 3.2 Proximity-Weighted Geometric Prior: The geometric prior emphasizes valid pixels within a selected height band above the ground plane, weighting nearer structures more strongly while suppressing distant or out-of-band pixels.The prior is bounded in [0, 1] by binary masks and exponential distance decay.
  • 3.3 Geo-Anchored VFM Adaptation: GAFT aligns CLS-to-patch attention rollout with the geometric prior using residual-adjusted, normalized spatial maps upsampled to the prior resolution.The geometric loss constrains relative spatial distribution rather than absolute rollout magnitude.
  • 3.3 Geo-Anchored VFM Adaptation: A KL representation anchor penalizes divergence between adapted and frozen DINOv2 CLS embeddings while allowing LoRA to adjust spatial routing.This separates representation preservation from geometric alignment.

4 Experimental Evaluation

The evaluation uses intervention-verified forest hazards and repeated two-direction LOSO testing to measure cross-scenario generalization. GAFT outperforms frozen DINOv2 and supervised PEFT controls, with performance depending on geometric anchoring and adaptation capacity.

  • Datasets and Protocol: 124 positive frames from 40 intervention-verified approach runs define hazards as observed robot–terrain failures requiring human intervention.The benchmark includes fallen logs and woody protrusions across two deliberately different scenarios.
  • Datasets and Protocol: GAFT and baselines are evaluated with 50 repeated two-direction LOSO tests per fixed Stage 1 adapter, using grouped five-fold CV for threshold selection.Test sets contain original frames only, while augmentation is restricted to positive training frames.
  • Datasets and Protocol: F2 weights recall four times more than precision, reflecting the higher cost of missed hazards than false alarms.A stratified-random predictor has expected F2 ≈0.042 at the observed 1:23 positive-to-negative ratio.
  • Main Results: 0.3757 mean F2 is achieved by GAFT across ten independently initialized Stage 1 adapters, compared with 0.0607 for frozen DINOv2.GAFT QKV reaches 0.2695, while supervised PEFT controls use the same backbone and evaluation protocol but omit geo-anchored adaptation.
  • Main Results: Sup-LoRA QKV+MLP reaches F2 = 0.0138, while VPT and AdaptFormer reach 0.0123 and 0.0282 under the matched repeated-LOSO protocol.The supervised methods fit their source scenarios but transfer poorly to held-out scenarios.
  • Main Results: +0.3150 [0.2350, 0.3904] is GAFT QKV+MLP’s paired ΔF2 against frozen DINOv2, while its advantage over GAFT QKV is not robust.The matched-capacity Sup-LoRA QKV+MLP comparison is +0.3619 [0.2865, 0.4284].
  • Main Results: Stage 1 adaptation, rather than repeated Stage 2 evaluation, introduces most GAFT variation across independently trained adapters.The averaged Stage 2 standard deviations are 0.0575 for GAFT QKV and 0.0536 for GAFT QKV+MLP.
  • Ablation Studies: 0.2587 ± 0.1535 with βkl = 0.05 exceeds 0.0596 ± 0.0624 with βkl = 0.001 in the QKV-only RANSAC ablation.Weak anchoring is seed-sensitive and does not consistently exceed the frozen baseline.

5 Discussion

GAFT improves cross-scenario hazard identification over supervised PEFT in a data-scarce setting, but its benefits depend on adaptation choices and remain bounded by the benchmark and offline evaluation scope.

  • Cross-scenario transfer: Supervised PEFT fits the source scenario but transfers poorly to the held-out scenario, whereas geo-anchored adaptation improves repeated-LOSO transfer.This contrast supports geo-anchored adaptation as an inductive bias rather than attributing the result to matched adapter capacity or hazard-label supervision alone.
  • Ablations: Weak KL anchoring and very low LoRA rank produce weak or seed-sensitive results, making GAFT’s benefit conditional on adaptation settings.The difference between GAFT QKV and QKV+MLP is not robust under paired analysis.
  • Scope: The evidence is limited to offline RGB hazard identification under rare, intervention-verified failures involving two verified forest hazard morphologies.Repeated LOSO evaluations measure resampling and probe-fitting variation, not independent environments or robustness across arbitrary deployment conditions.
  • Interpretation and future work: GAFT’s attention rollout is a differentiable spatial-routing interface, while its proximity-weighted geometric proxy is not a pixel-accurate hazard mask.Closed-loop effects on immobilization, recovery frequency, and mission time remain future work.

6 Conclusion

GAFT uses a proximity-weighted geometric prior to adapt an RGB vision foundation model before scarce hazard-label supervision. Across independently initialized adaptations and repeated LOSO evaluation, it improves RGB-only hazard identification over frozen DINOv2 while supervised PEFT controls transfer unreliably.

  • Method: GAFT introduces a two-stage framework that uses a proximity-weighted geometric prior before training a hazard classifier from scarce interaction labels.The prior guides adaptation of an RGB vision foundation model before hazard-label supervision.
  • Results: 0.0607 to 0.3757 mean F2: GAFT improves frozen DINOv2 across ten independently initialized adaptations under repeated LOSO evaluation.The reported comparison concerns RGB-only hazard identification with scarce hazard labels.
  • Conclusion: Matched supervised PEFT controls fit their source scenario but fail to transfer reliably to the held-out one.The conclusion supports a narrow scope: geo-anchored adaptation can improve RGB-only hazard identification when labels are scarce and pretrained features remain anchored.
Loading 2608.30858v1…