PaperlayerView exact revision on arXiv

Source-linked AI summary

MANTLE: A Framework for Adaptive In-Situ Planetary Perception Using a Modular Uplink Principle

Pranav Durai, Gary Doran

arXiv:2608.28724v1cs.CV

TL;DR

Mars exploration needs perception systems that support both scientific terrain interpretation and safe navigation across orbital and surface imagery. MANTLE addresses this with a shared frozen DINOv2 backbone and lightweight task-specific heads, achieving strong classification and segmentation results while enabling later capabilities to be uplinked without retraining the backbone.

  • Problem

    Mars missions need onboard perception for scientific terrain understanding and hazard assessment, while existing systems treat capabilities as separate models or fixed deployments.

  • Method

    MANTLE uses a shared frozen DINOv2 backbone with task-specific heads trained on HiRISE landform and MSL boulder datasets.

  • Results

    MANTLE achieved 92.56% test accuracy for seven-class terrain classification and 0.753 validation IoU for boulder segmentation, with 0.735 IoU on a temporally held-out test set.

  • Takeaways & Limitations

    The Modular Uplink Principle allows new perception capabilities to be trained as lightweight heads and added against a persistent frozen backbone.

Abstract

from arXiv · show

Planetary surface exploration missions rely increasingly on autonomous robotic platforms capable of interpreting complex terrain to ensure safe navigation, enable targeted science, and improve operational efficiency, as demonstrated across past Mars missions from Viking through Perseverance. Among the key perception capabilities, landform classification provides contextual information for landing site selection and scientific analysis, while boulder segmentation supports hazard assessment and path planning. This paper presents MANTLE, a multi-task adaptive network for terrain and landform extraction. The model uses a shared DINOv2 backbone for high-level feature extraction with task-specific heads: a classification head for large-scale landform classification, and a segmentation head for pixel-wise boulder localization, each trained on curated datasets built respectively from HiRISE orbital imagery and MSL surface-level imagery. The classification head achieved a test accuracy of 92.56% across seven Martian terrain classes, while the segmentation head achieved a validation IoU of 0.753 and showed strong cross-sol generalization on a held-out test set from previously unseen rover traverses. A key advantage of MANTLE is its modular, extensible design, formalized here as the Modular Uplink Principle: only a shared, frozen backbone needs to remain onboard, while subsequent perception capabilities are trained on Earth as lightweight task-specific heads and uplinked without retraining the full model. This work demonstrates two such high-impact capabilities, terrain classification and boulder segmentation, as an initial realization of a framework built to support many more over a mission's lifetime. With this foundation, future explorers need not arrive on Mars fully formed, but can continue to learn, adapt, and grow more capable with every uplink.

I. INTRODUCTION

Mars exploration has progressed toward increasingly autonomous robotic perception, creating demand for onboard systems that support both scientific interpretation and hazard assessment. MANTLE addresses this need with shared features and lightweight task-specific heads spanning orbital and ground-level imagery.

  • Successive Mars missions have advanced visual perception, navigation, and hazard avoidance toward kilometer-scale autonomous traverses and geological investigations.
  • Perseverance’s combined orbital and surface instruments highlight the need for onboard terrain understanding, science-target detection, and hazard localization.
  • MANTLE uses a shared frozen backbone with lightweight heads for terrain classification and boulder segmentation across orbital and ground-level views.
  • A shared backbone amortizes onboard memory and flight-qualification effort, allowing future capabilities to be added as independently trained heads without modifying it.
  • The work contributes a balanced seven-class HiRISE dataset, an 840-sol MSL boulder dataset, the MANTLE architecture, the Modular Uplink Principle, and public data and code.

II. BACKGROUND

Prior Mars-perception research has developed separate methods for orbital landform analysis, hazard detection, and ground-level terrain understanding. The background motivates MANTLE as a unified, uplink-compatible framework across these vantage points.

  • Planetary computer vision supports rover navigation, hazard finding, and identification of scientifically valuable terrain.
  • Prior work spans terrain segmentation, impact detection, image retrieval, rock measurement, and learned boulder segmentation across planetary imagery.
  • AI4MARS provides large-scale rover terrain labels, but its crowdsourced polygon boundaries are coarse for small, irregular boulders.
  • SegMarsViT demonstrates efficient onboard segmentation, while prior systems generally target fixed capabilities or separate orbital and ground-level settings.
  • The paper identifies a gap in unifying orbital-scale analysis and ground-level terrain understanding within one shared backbone with extensible perception capabilities.

III. METHODOLOGY

MANTLE targets two complementary Mars-perception tasks: scientific terrain classification and boulder hazard detection. It uses a shared transformer-based backbone and task-specific processing supported by curated orbital and ground-level datasets.

  • MANTLE addresses scientifically valuable terrain identification and hazardous-boulder detection for mission planning, site selection, and rover safety.
  • The architecture combines a shared frozen backbone with lightweight task-specific heads for terrain classification and boulder hazard detection.
  • MSL Boulder Dataset image–mask pairs span different sols and capture variation in boulder sizes, densities, and terrain contexts.
  • The two tasks use high-resolution datasets curated from HiRISE orbital imagery and MSL ground-level imagery.

A. High-Resolution HiRISE Landform Dataset

The high-resolution HiRISE Landform Dataset was constructed to address limitations in earlier data, especially class imbalance, intraclass variability, and low spatial resolution. It uses balanced 1024 × 1024 pixel patches across seven Martian terrain classes.

  • The earlier HiRISENet dataset had severe class imbalance, substantial intraclass variability, and low-resolution 227 × 227 tiles from approximately 10 m/pixel imagery.
  • The new dataset uses raw HiRISE imagery at 0.5 m/pixel and extracts 1024 × 1024 pixel patches across seven terrain classes.
  • The seven classes are crater, dark dune, slope streak, bright dune, impact ejecta, Swiss cheese, and spider.
  • Black-pixel regions were mitigated through threshold-based replacement before patch preparation and resizing.
  • Class balance was fixed at Nc = 350 samples per class, with augmentation for under-represented classes and pruning for over-represented classes.

B. High-Resolution MSL Boulder Dataset

The MSL boulder dataset combines filtered Mastcam imagery with pixel-accurate masks and sol-level partitioning to support rover hazard perception. MANTLE uses this dataset alongside a shared frozen backbone for ground-level segmentation.

  • Boulder detection supports hazard avoidance, path planning, wheel-entrapment prevention, and geological analysis.
  • 27,122 raw Mastcam PNG images were filtered for boulder-positive content, quality, and resolution before mask generation.The pipeline used zero-shot VLM filtering, spectral and resolution checks, and subsequent processing for dataset construction.
  • SAM 2 produced pixel-accurate boulder masks with human-in-the-loop correction.
  • 6,232 image–mask pairs from 840 unique sols form the final training-and-validation dataset.The corpus was partitioned by sol into 4,892 training and 1,340 validation images to prevent within-sol leakage.
  • The boulder segmentation head operates on MSL ground-level imagery within MANTLE’s shared-backbone architecture.The backbone is kept frozen to support adding independently trained task-specific heads.

1) Backbone Feature Extractor:

MANTLE processes orbital and ground-level imagery independently through one frozen DINOv2 backbone, producing spatial features for classification and segmentation. The backbone’s activations organize task-relevant structures across both modalities without Mars-specific fine-tuning.

  • MANTLE sends two distinct input modalities independently through the same frozen backbone.
  • 224 × 224 HiRISE inputs yield 256 patch tokens, while 784 × 784 MSL inputs yield 3,136 tokens on a 56 × 56 grid.The resulting spatial features support the classification and segmentation heads at their respective input resolutions.
  • The shared backbone’s activation patterns isolate craters in orbital scenes and boulder outlines and contact edges in surface scenes.These representations provide the task-relevant spatial organization consumed by the two heads.
  • The classification head applies a two-layer ReLU network to the 384-dimensional CLS token and outputs probabilities over seven terrain classes.
  • The segmentation head transforms a 768 × 56 × 56 feature grid through attention and decoder stages into a full-resolution binary boulder mask.

4) Training Objectives:

MANTLE trains its classification and segmentation heads independently with separate losses and optimizers. The segmentation objective combines region, overlap, boundary, and annealed auxiliary supervision terms.

  • The classification and segmentation heads are trained independently, each with its own loss function and optimizer.
  • The classification head uses standard cross-entropy loss.
  • The segmentation head combines BCE, Dice, and boundary-aware losses with annealed auxiliary supervision at two decoder stages.
  • α = 1.2 weights the positive class, while the Laplacian-based boundary term emphasizes mask edges.
  • λaux(t) decays linearly from 0.3 to 0 during the first half of training.

IV. EXPERIMENTAL EVALUATION AND RESULTS

MANTLE was evaluated on held-out terrain-classification and cross-sol boulder-segmentation data. The classifier achieved high overall accuracy, while qualitative segmentation results showed strong large-boulder predictions alongside missed small rocks.

  • A. Terrain Classification Evaluation: 92.56% accuracy was achieved on the 336-image held-out test set for seven Martian terrain classes.The test images were selected to avoid spatial proximity to training tiles and reduce train/test leakage.
  • A. Terrain Classification Evaluation: The classifier generalized strongly across most terrain types, with near-perfect accuracy on bright dune and Swiss cheese.
  • B. Boulder Segmentation Evaluation: Figure 7 compares source MSL Mastcam images with boulder-probability heatmaps across the held-out Sol ≥2305 test set.Warmer colors indicate higher predicted boulder likelihood.
  • B. Boulder Segmentation Evaluation: The qualitative segmentation examples show strong detection of large boulders in both dense and sparse arrangements.
  • B. Boulder Segmentation Evaluation: Missed small rocks and pebbles are the dominant qualitative failure mode and have negligible impact on rover navigation.
  • A. Terrain Classification Evaluation: Impact ejecta, spider, and crater produced the most frequent confusions, while impact ejecta had 0.72 recall.These classes share morphological signatures such as radial patterns and shadowing effects.

B. Boulder Segmentation Evaluation

MANTLE’s boulder-segmentation head generalized from its training sols to previously unseen rover traverses, with strongest performance on larger traversal-relevant boulders.

  • 0.753 validation IoU was achieved on the Sol 0017–2302 split using the frozen DINOv2 ViT-S/14 backbone.The segmentation head was trained on the MSL Boulder Dataset.
  • 0.735 test IoU on 53 images from Sol ≥2305 represented a 0.018 decrease from validation.The held-out images covered different terrain morphology, dust accumulation, and lighting conditions.
  • 62.3% of held-out images achieved an IoU of 0.70 or higher.This corresponded to 33 of 53 test images.
  • The strongest performance concentrated on larger boulders that constitute genuine navigation hazards, while lower-performing cases were dominated by small rocks and pebbles.The passage characterizes the smaller rocks as posing negligible rover-traversal risk.
  • Figure 7 supplements the quantitative evaluation with per-pixel prediction-confidence heatmaps over representative Sol ≥2305 test images.

V. MODULAR UPLINK PRINCIPLE

The Modular Uplink Principle addresses the difficulty of updating fixed rover perception systems by retaining one frozen backbone and delivering new capabilities as lightweight heads.

  • New capabilities follow a three-step process: Earth-based head training, DSN uplink and relay, then onboard execution as a plug-in beside existing heads.The process uses a copy of the frozen, flight-qualified backbone during Earth-based training.
  • Existing rover perception capabilities are difficult to expand after landing because uplink bandwidth and communication windows are limited, while large neural models are impractical to transmit.The paper connects these constraints with missions rarely expanding perception systems after landing.
  • Only the shared backbone remains persistent onboard, while compact heads implement future tasks without modifying the backbone or flight-qualified software.The principle supports tasks including classification, segmentation, novelty detection, and hazard mapping.
  • Task-specific heads introduce only 10^5–10^6 parameters versus 2.2×10^7 for the backbone.This is roughly 1.5–2.3 orders of magnitude fewer parameters.
  • Head payloads are approximately 0.1–2.9 MB, making incremental capability updates practical under deep-space communication constraints.The passage notes that quantization can reduce payloads further.

C. Use-Cases

MANTLE’s modular heads support post-landing expansion, adaptive hazard and science targeting, and sample-caching or follow-on mission tasks without retraining or modifying the onboard model.

  • Use-Cases: Post-landing capability expansion can add heads for mineral phase detection, dust devil identification, or frost mapping.These specialized capabilities are delivered by uplinking new heads without updating the backbone.
  • Use-Cases: New segmentation or detection heads can address newly identified hazards and science targets during a mission.Examples include fresh boulder fields, outcrop boundaries, hydrated mineral signatures, and lithologic contacts.
  • Use-Cases: Additional heads can detect cached sample tubes, rendezvous markers, or engineered artifacts needed for Mars Sample Return.
  • Use-Cases: MANTLE combines a balanced seven-class HiRISE landform dataset with an 840-sol pixel-accurate MSL boulder dataset.The framework uses a shared, frozen DINOv2 backbone with lightweight task-specific heads.
  • Use-Cases: The classification head achieved 92.56% test accuracy, while segmentation reached 0.753 validation IoU and 0.735 on a temporally held-out test set.These results support the framework’s two demonstrated perception capabilities.
  • Use-Cases: End-to-end uplink validation remains a next step, including INT8 quantization, FP32 comparison, and flight-like hardware benchmarking.The paper identifies Snapdragon-class processors as an example target.
Loading 2608.28724v1…