PaperlayerView exact revision on arXiv

Source-linked AI summary

TriCCOT: Tri-part Convolutional Conformal Transformer for Onboard Space Object Detection

Adrien Dorise, Marjorie Bellizzi, Julia Cohen, Stéphane May

arXiv:2609.08659v1cs.CVcs.AI

TL;DR

Onboard object detection must handle raw, degraded imagery under strict hardware constraints. TriCCOT combines convolutional proposals, conformal box adjustment, and FPGA-friendly Aper-GATES classification; it remains competitive on clean imagery, performs best among evaluated FPGA-compatible methods on noisy imagery, and deploys on a Versal VCK190 without DPU modification.

  • Problem

    Onboard detection must operate on uncorrected, degraded imagery while transformer attention remains difficult to deploy on constrained accelerators.

  • Method

    TriCCOT sequentially combines a convolutional RPN, conformal prediction for proposal adjustment, and Aper-GATES crop classification for embedded FPGA inference.

  • Results

    TriCCOT achieves competitive clean-image detection, outperforms evaluated convolutional and FPGA-compatible baselines under raw degradations, and enables CNN-Transformer inference without modifying the accelerator architecture.

  • Takeaways & Limitations

    Aper-GATES provides a practical trade-off between predictive performance and deployability by enabling convolutional and attention-based architectures within one DPU.

  • Takeaways & Limitations

    Sequential training can propagate errors between the RPN, conformal predictor, and classifier because the full pipeline is not optimized end-to-end.

Abstract

from arXiv · show

Onboard object detection in Earth observation is constrained by limited computational resources and the absence of fully corrected imagery. While convolutional detectors are hardware-efficient, they often struggle to extract robust representations from raw and noisy data. Conversely, transformer-based models provide stronger global reasoning capabilities but remain difficult to deploy on FPGA accelerators due to quadratic attention complexity and non-compatible operations. We introduce TriCCOT, a tri-part architecture for robust and deployable onboard object detection. TriCCOT combines a convolutional region proposal network, a conformal prediction stage, and Aper-GATES, our hardware-friendly attention-based classifier. The region proposal network generates candidate bounding boxes, which are subsequently enlarged via conformal prediction, providing a distribution-free probabilistic coverage guarantee. The resulting crops are processed by Aper-GATES, which reformulates self-attention through convolutional projections, global channel statistics, and hardware-friendly gating operations, avoiding standard transformer operations that are poorly suited to CNN-oriented accelerators. Experiments on the DIOR and VDVRaw datasets demonstrate competitive detection performance and improved robustness to spatial blur and signal-dependent noise when compared to FPGA-compatible architectures. Finally, we report full deployment on a Xilinx Versal VCK190 FPGA without modifying the underlying DPU architecture, enabling unified CNN-Transformer inference for spaceborne embedded applications.

1 Introduction

Onboard Earth observation seeks to process imagery closer to acquisition, but raw, noisy data and constrained hardware make robust object detection difficult. TriCCOT addresses this by combining attention-based representations with deployment-oriented reformulation.

  • Onboard processing reduces downlink bandwidth and latency by moving tasks such as object detection directly onto satellites.
  • Using raw imagery can eliminate onboard restoration stages, but remains uncommon because raw data are challenging for detection models.
  • Transformer attention offers strong representations but is difficult to deploy onboard because softmax, large matrix multiplications, and quadratic token complexity strain constrained hardware.
  • TriCCOT introduces Aper-GATES, an FPGA-compatible attention formulation, together with a convolutional, conformal, and attention-based detection architecture for raw imagery.

2 Related Work

Related work spans convolutional and transformer-based remote-sensing detectors, onboard learning from raw imagery, hardware-oriented hybrid models, and conformal prediction. TriCCOT positions conformal prediction as a structural part of detection rather than only a calibration tool.

  • Object detection in remote sensing: Remote-sensing detectors include convolutional two-stage and one-stage frameworks, often using multi-scale pyramids, while transformers provide strong performance at higher computational cost.
  • Onboard deployments for remote sensing applications: Onboard remote-sensing inference aims to reduce transmission overhead and support real-time responses, including models trained on raw or minimally restored imagery.
  • Embedded object detection: Standard self-attention scales quadratically with token number and uses softmax, while linear approximations still require global matrix operations that limit accelerator compatibility.
  • Conformal prediction: TriCCOT uses conformal prediction to enlarge proposals before classification, targeting contextual completeness and mitigation of proposal-generation bias.

3 Combining convolution, conformal and attention

TriCCOT combines sequential proposal generation, conformal box adjustment, and crop classification with Aper-GATES. Object-centric crops reduce irrelevant content before attention-based classification.

  • TriCCOT comprises a convolutional RPN, conformal adjustment of predicted boxes, and Aper-GATES crop classification in three sequential stages.
  • Processing object-centric crops reduces irrelevant image content and enables a compact classifier.
  • The RPN predicts boxes, the conformal predictor refines them, and the resulting crops are aggregated into a 2D token space for the attention classifier.

3.2 Region Proposal Network (RPN)

The RPN uses a two-stage detection design with CSPDarknet and FPN components to predict boxes across object scales. It requires few changes for degraded raw imagery because localization is reported as relatively unaffected by such degradations.

  • The RPN follows two-stage detection by predicting bounding boxes before a second classifier assigns object classes.
  • A CSPDarknet backbone with an FPN mechanism supports the RPN’s handling of multiscale objects.
  • Few RPN modifications are required because reported raw-data degradations have little effect on bounding-box localization.

3.3 Conformal predictor

TriCCOT uses conformal prediction to enlarge RPN boxes before classification, addressing missing object content and limited context while providing probabilistic coverage under conformal assumptions.

  • Conformal prediction relaxes RPN box constraints because imperfect proposals can crop or shift objects and tight annotations provide little context.
  • The predictor adjusts each bounding-box coordinate independently using calibrated corrections that may differ in magnitude and sign.
  • Under conformal coverage assumptions, adjusted boxes entirely contain a fraction 1−α of ground-truth objects.
  • The resulting crops add contextual information and adapt to localisation errors and directional biases with limited computational overhead.This data-driven adjustment replaces a fixed multiplicative expansion ratio.
  • The implementation uses the PUNCC library for conformal prediction.

3.4 Aper-GATES: An Aperture-Gated Attention Transformer for Embedded Systems

Aper-GATES reformulates attention for FPGA deployment by combining convolutional projections, global channel statistics, and hardware-efficient gating while preserving input-size flexibility through fixed-grid embeddings.

  • Aper-GATES motivation: Aper-GATES targets transformer deployment constraints by replacing costly softmax-based attention with hardware-efficient gating for global information flow.The design is motivated by transformers’ global context and FPGA limitations from softmax and quadratic attention complexity.
  • Multi-resolution patch embedding: Its multi-resolution embedding bank selects a convolutional branch for each admissible crop size and maps all inputs to a fixed G×G grid.Selection is performed by Boolean conditioning on the CPU; the implementation uses G = 8.
  • Multi-resolution patch embedding: Metadata encoding preserves awareness of the original object scale after inputs are projected into the fixed feature grid.The metadata is fused through a two-layer convolutional network and additive residual fusion.
  • Gated convolutional attention: Aper-GATES replaces quadratic token-to-token affinity with a global second-order statistic derived from element-wise K⊙V products and spatial pooling.The pooled descriptor summarizes global feature correlations rather than individual token dependencies.
  • Gated convolutional attention: Grouped 1×1 convolutions compute head-wise Q, K, and V projections while preserving the two-dimensional spatial structure.
  • Gated convolutional attention: The gating function multiplicatively modulates queries using transformed global context, prioritising relevant features and suppressing noise.The formulation has linear complexity O(BCHW) and uses hardware-friendly activation such as Hardsigmoid.
  • DPU-compatible activation: ConSmax-inspired gating replaces hardware-intensive softmax with depthwise affine transformation and piecewise-linear Hardsigmoid operations.The affine parameters are learned channel-wise through a depthwise 1×1 convolution.

3.5 Training and inference

TriCCOT trains three models sequentially, using RPN predictions to calibrate conformal corrections before training Aper-GATES on frozen-stage crops.

  • TriCCOT trains its three independent models sequentially, with each model learning from the previous model’s predictions.
  • The RPN is fitted first, then its separate calibration-set outputs train the conformal corrections.
  • The conformal predictor enlarges crops while avoiding overly large regions that would degrade inference time.
  • Aper-GATES is trained afterward on crops generated by the frozen RPN and conformal stages.

3.6 Datasets

The evaluation uses DIOR, degraded DIOR, and real raw VDVRaw imagery to assess detection across standard and sensor-impaired conditions.

  • Datasets: DIOR contains 23,463 optical remote-sensing images spanning 20 object classes.
  • Degradation simulation: Degraded DIOR applies spatial blur and signal-dependent noise intended to represent raw satellite imaging conditions.
  • Datasets: VDVRaw contains 282 multispectral images divided into 4 238 patches, with 3 827 annotated objects across 17 classes and multiple sensor artefacts.
  • Degradation simulation: Spatial degradation is modeled with a parametric Modulation Transfer Function combining optical blur and sensor sampling.
  • Degradation simulation: The simulated images use MTF@Nyquist values in [1%,2%] and luminance-point SNR ranges of [25dB, 35dB] and [65dB, 75dB].

4 Results

TriCCOT’s full configuration performs best in ablation and remains competitive across DIOR and VDVRaw, while preserving FPGA deployment efficiency and robustness to degraded imagery.

  • 4.1 Ablation study: The full TriCCOT implementation achieves the best performance on all ablation criteria.Removing Aper-GATES decreases mAP@50 by 0.82 percentage points, whereas removing conformal prediction decreases it by 7.39 percentage points.
  • 4.2 Detection performance: On original DIOR, TriCCOT stays within 2.5 percentage points of FPGA-compatible leaders for both mAP@50 and mAP@50:95, using 3.5M parameters and 8.2 GFLOPs.
  • 4.2 Detection performance: Across representative DIOR categories, AP@50 ranges from 43.09% for Bridge to 91.92% for Airplane.
  • 4.2 Detection performance: TriCCOT obtains the best FPGA-compatible performance on raw DIOR, reaching 74.8% mAP@50 and 42.2% mAP@50:95.
  • 4.2 Detection performance: On VDVRaw, TriCCOT again leads FPGA-compatible models with 25.3% mAP@50 and 18.0% mAP@50:95.
  • 4.3 FPGA deployment: TriCCOT is deployed on a Xilinx Versal VCK190 without altering the DPU architecture and runs at 28.2 FPS on 640x640 patches.Its total inference time is 35.5 ms, with the RPN accounting for 29.1 ms.

5 Discussion

TriCCOT prioritizes robustness to degraded imagery and FPGA deployability while combining conformal prediction with attention-based classification. Its sequential training remains a practical limitation because errors can propagate between stages.

  • Robustness: TriCCOT performs best among evaluated convolutional methods on degraded DIOR imagery with spatial blur and signal-dependent noise.It does not achieve the highest accuracy on original images, but prioritizes preserving detection quality under degradation.
  • Conformal prediction: Conformal prediction enlarges crops, improves object coverage, adds context, and provides a mathematically defined coverage guarantee without arbitrary scaling parameters.The stage also compensates for model prediction biases and helps remove labeling biases.
  • Embedded deployment: Aper-GATES reformulates self-attention convolutionally, enabling fast inference and combined CNN-transformer execution on a Versal VCK190 DPU without reconfiguration.This offers a practical trade-off between predictive performance and deployability, although it does not match large foundation transformer models.
  • Training: All TriCCOT components can be trained in 4-5 h on one NVIDIA RTX 4090, with crop-based transformer classification yielding straightforward training and fast convergence.The classifier processes primarily task-relevant object-crop information rather than full images.
  • Limitations: Sequential training prevents end-to-end optimization and allows RPN biases or errors to propagate through the conformal predictor to the classifier.Joint or cooperative training and hard-example mining are proposed as future directions.
  • Overall assessment: TriCCOT is competitive with state-of-the-art detectors on original images and performs best on noisy images among models deployable on FPGAs.The result supports lighter transformer architectures with greater edge-device deployment capability for onboard remote sensing.

6 Conclusion

TriCCOT combines convolutional proposals, conformal prediction, and Aper-GATES attention for degraded-image object detection compatible with embedded FPGA deployment. Experiments show competitive clean-image performance and stronger robustness under raw imaging degradations, while sequential training and model size remain limitations.

  • Architecture: TriCCOT combines a convolutional RPN, conformal prediction, and Aper-GATES self-attention for onboard detection under degraded imaging conditions.The design pairs convolutional localization, probabilistic crop coverage, and attention-based representations within an FPGA-compatible architecture.
  • Results: Experiments on DIOR show competitive clean-image detection and outperformance of evaluated convolutional and FPGA-compatible baselines under spatial blur and signal-dependent noise.Aper-GATES replaces standard attention operations with convolutional projections and hardware-friendly gating.
  • Limitations: Sequential training may propagate errors among the RPN, conformal predictor, and classifier, motivating joint or cooperative training through hard-example mining.The proposed future work targets better optimization of the full detection pipeline.
  • Future work: Rotated bounding boxes are proposed as future work because varying object rotations can produce substantially different bounding-box ratios.Harmonizing ratios across instances of the same class could benefit classification.
  • Future work: The CSPDarknet RPN backbone limits total model parameters and size, motivating replacement with a lighter backbone such as MobileNet.The intended benefits are a smaller TriCCOT model and faster inference.
Loading 2609.08659v1…