PaperlayerView exact revision on arXiv

Source-linked AI summary

FUSEye: Training-Light Fisheye Detection with Overlapping Views and Zero-Initialized Adapters

Wenya Su, Kai Luo, Di Wen, Ruiping Liu, Yufan Chen, Junwei Zheng, Kunyu Peng, Kailun Yang

arXiv:2610.02799v1cs.CVcs.ROeess.IV

TL;DR

Fisheye distortion and boundary compression limit transfer from COCO-pretrained detectors, while full fine-tuning requires substantial labels and compute. FUSEye adapts a frozen-backbone detector with overlapping views, residual adapters, and cross-view agreement fusion. On WoodScape, it raises YOLO26-x mAP50 from 0.1480 to 0.2661 while retaining 97.6% of full-label performance with 25% of the labels.

  • Problem

    Radial distortion and boundary compression make COCO-pretrained detectors less transferable to fisheye imagery, while full fine-tuning requires abundant labels and compute.

    Source: §I · pp. 1–2

  • Method

    FUSEye combines overlapping views, zero-initialized residual adapters, and learned cross-view agreement fusion while keeping the pretrained backbone frozen.

    Source: §I · p. 2, A. Overall Framework · p. 3, B. Overlapping Grid View Generation… · p. 3

  • Results

    0.1480 to 0.2661 mAP50: FUSEye improves YOLO26-x on WoodScape, retains 84.3% of full-fine-tuning accuracy, and retains 97.6% of full-label performance using 25% of labels.

    Source: §I · p. 2

  • Takeaways & Limitations

    FUSEye fits label- and training-compute-constrained detector adaptation settings, with deployment overhead from running the same detector over five overlapping views.

    Source: §I · p. 2

    For bus, FUSEye falls below direct transfer, whereas full fine-tuning improves performance on this challenging category.

    Source: B. Main Results · p. 6

Abstract

from arXiv · show

Fisheye cameras give mobile robots a single-sensor, low-cost view of their surroundings, yet the COCO-pretrained detectors that practitioners routinely reuse fail on them: strong radial distortion warps local image structure, while boundary compression shrinks objects to near-invisible sizes. Full fine-tuning closes much of the gap but requires abundant fisheye labels and compute. We present FUSEye, a training-light framework that turns a frozen-backbone COCO-pretrained extra-large YOLO26 detector (YOLO26-x) into a fisheye detector. FUSEye adds roughly 227k new parameters while updating the inserted modules and the pretrained detection head. It addresses the transfer gap at three causally linked levels. At the input level, overlapping grid view generation and box remapping (GridViews) enlarge compressed boundary regions. At the feature level, zero-initialized residual adapters (Z-Adapters) correct distortion-induced feature misalignment. At the decision level, learned cross-projection agreement fusion (AgreeFusion) promotes low-confidence detections only when they are supported by consistent evidence across multiple views. On the WoodScape surround-view fisheye benchmark, FUSEye raises YOLO26-x from 0.148 to 0.266 mAP50 and retains 84.3% fully fine-tuned accuracy. Moreover, randomly using only 25% of the labeled training images, FUSEye achieves 0.2597 mAP50, retaining 97.6% of its full-label performance. FUSEye also consistently improves YOLOv8-11 detectors, showing that the recipe is architecture-agnostic. Source code will be available at https://github.com/Su-wenya/FUSEye.

I. INTRODUCTION

FUSEye targets the fisheye transfer gap with a training-light framework that adapts a COCO-pretrained detector while keeping its backbone frozen. On WoodScape, it improves YOLO26-x detection and retains much of full-fine-tuning performance with fewer labels and trainable parameters.

Motivation

  • Strong radial distortion and boundary compression impede transfer from perspective-trained detectors, especially for small objects entering a robot’s field of view.The introduction notes that missing early observations reduces the time available for braking or replanning.Source: §I · pp. 1–2

Research gap

  • Prior approaches offer useful components, but their integration in frozen-backbone fisheye adaptation under limited labels and training budgets remains less explored.The stated question is whether input compression, feature misalignment, and cross-view conflicts can be addressed jointly without full-model fine-tuning.Source: §I · p. 2

Approach

  • FUSEye coordinates GridViews for compressed boundary objects, Z-Adapters for feature misalignment, and AgreeFusion for consistent cross-view detections.The three modules target input, feature, and decision levels, respectively.Source: §I · p. 2

Results

  • 0.1480 to 0.2661 mAP50: FUSEye improves YOLO26-x on WoodScape MVR, retains 84.3% of full-fine-tuning accuracy, and retains 97.6% of full-label performance using 25% of labels.Experiments also report consistent improvements on YOLOv8–11 detectors.Source: §I · p. 2

II. RELATED WORK

Related work covers fisheye detectors, distortion-aware architecture changes, parameter-efficient adaptation, and multi-view inference. FUSEye combines these lines while aiming to reuse pretrained detectors with limited additional training.

A. Fisheye and Omnidirectional Object Detection

  • Many fisheye detectors redesign or retrain complete models, while this work aims to reuse a COCO-pretrained YOLO26 detector with minimal additional training.The related-work section also surveys fisheye and omnidirectional datasets, benchmarks, and detection systems.Source: A. Fisheye and Omnidirectional Object… · p. 2

B. Distortion-Aware Convolutions and Rectification

  • Distortion-aware kernels and rectification can address fisheye structure, but cited approaches may require calibration, preprocessing, or full-model retraining.Z-Adapters are presented as a calibration-free, parameter-efficient alternative that preserves the pretrained model’s initial behavior.Source: B. Distortion-Aware Convolutions and… · pp. 2–3

C. Parameter-Efficient Fine-Tuning and Visual Adapters

  • Generic parameter-efficient adapters update few weights but are not designed for spatially non-uniform fisheye distortion; Z-Adapters learn fisheye residual corrections with the backbone frozen.The passage situates these methods alongside residual adapters, LoRA, zero-initialized adapters, and detector-specific parameter-efficient fine-tuning.Source: C. Parameter-Efficient Fine-Tuning and… · p. 3

D. Multi-View Inference and Detection-Level Fusion

FUSEye uses shared-detector views and detection-level fusion to handle small objects and redundant predictions. Its learned agreement scorer promotes cross-view-supported candidates while retaining the original confidence threshold for isolated detections.

D. Multi-View Inference and Detection-Level Fusion

  • GridViews supplies calibration-free overlapping views, and AgreeFusion re-scores per-cluster agreement to promote candidates supported across views while preserving the original threshold for single-view detections.The related-work passage distinguishes this from conventional overlap suppression and confidence-weighted box fusion.Source: D. Multi-View Inference and… · p. 3

A. Overall Framework

  • Each resized view passes through the shared COCO-pretrained detector; detections are remapped to full-frame coordinates, clustered by class and IoU, re-scored, fused, and deduplicated with NMS.AgreeFusion operates at detection level, while the overall pipeline reuses the same detector across views.Source: A. Overall Framework · p. 3

B. Overlapping Grid View Generation and Box Remapping

GridViews generates overlapping global and local views to enlarge peripheral objects at network input scale. It remaps per-view boxes to shared full-frame coordinates before agreement fusion.

View generation

  • Five views comprise the full image and four overlapping corner crops, increasing the input-scale size of small objects while retaining global context.Adjacent crops overlap so the same object can appear in multiple views for later agreement assessment.Source: B. Overlapping Grid View Generation… · p. 3

Box remapping

  • Per-view detections are remapped from crop coordinates to the original image and normalized evaluation coordinates for cross-view fusion.The crop region and view-space detection box provide the coordinates used in remapping.Source: B. Overlapping Grid View Generation… · pp. 3–4

Inference and implementation

  • All five views use the same detector; a 0.02 confidence threshold retains low-confidence candidates for fusion, and the process costs about five single-view forward passes.GridViews requires no calibration, dewarping, or trainable parameters.Source: B. Overlapping Grid View Generation… · p. 4

C. Zero-Initialized Residual Adapter

FUSEye uses zero-initialized residual adapters to learn fisheye-specific feature corrections while leaving the pretrained backbone unchanged at initialization and frozen during training.

C. Zero-Initialized Residual Adapter

  • Zero-initializing only the output projection makes each adapter an identity mapping initially, preserving the pretrained detector’s output while allowing the projection to receive gradients.After the first optimizer step, gradients can flow to the inner convolutions and the adapter learns a residual correction.Source: C. Zero-Initialized Residual Adapter · p. 4
  • Each residual branch reduces channels with a 1 × 1 convolution, applies batch normalization and SiLU, performs 3 × 3 depthwise spatial modeling, then restores the channels.The depthwise convolution uses 9C/r parameters; affine-free batch-normalization layers add no parameters.Source: C. Zero-Initialized Residual Adapter · p. 4
  • Three adapters are inserted after P4, P5, and C2PSA; together they add 222,480 parameters to YOLO26x.The pretrained backbone remains frozen while the adapters learn corrections for distortion-induced feature misalignment.Source: C. Zero-Initialized Residual Adapter · p. 4

D. Cross-Projection Agreement Fusion

AgreeFusion pools remapped detections across views, scores their agreement, and promotes detections with consistent cross-view support while retaining the baseline threshold for isolated detections.

D. Cross-Projection Agreement Fusion

  • AgreeFusion keeps the baseline confidence threshold for isolated detections and uses a learned cluster score to promote detections supported by other views.An early feature-level fusion gate degraded detection quality and was excluded from the final design.Source: D. Cross-Projection Agreement Fusion · p. 4
  • For each class, remapped detections are clustered around the highest-confidence unassigned box using an IoU threshold of 0.5, with support counted across distinct views.Multiple detections from one view may join a cluster, but do not count as multiple-view support.Source: D. Cross-Projection Agreement Fusion · pp. 4–5, Figure 3 · p. 5
  • Each cluster’s 12-dimensional feature vector includes view-match rate, confidence statistics, noisy-OR score, pairwise IoU, normalized displacement and box size, and a class indicator.The class indicator lets the scorer learn class-specific acceptance policies.Source: D. Cross-Projection Agreement Fusion · p. 5
  • The learned scorer re-scores clusters; when displacement is at most 0.6, boxes are consolidated with confidence-weighted WBF, otherwise the highest-confidence box is kept.Class-wise NMS at IoU 0.5 removes remaining duplicates.Source: D. Cross-Projection Agreement Fusion · p. 5

A. Experimental Setup

Experiments train on three WoodScape views and evaluate on a held-out fourth view, using COCO-pretrained detectors and a specified frozen-backbone FUSEye setup.

A. Experimental Setup

  • Training uses WoodScape front, rear, and mirror-left views, while the mirror-right validation split is held out for evaluation; filtering leaves 6,082 annotated images.The reported classes are car, person, and bus; AP50 averages their AP at IoU 0.50, while AP50:95 averages across ten IoU thresholds.Source: A. Experimental Setup · p. 5
  • Unless stated otherwise, FUSEye uses COCO-pretrained YOLO26-x at 640 × 640 with a frozen backbone, and trains its head and adapters for eight epochs.The detection head, adapters, and scorer together contain approximately 6.67 million trainable parameters, including approximately 227 k introduced by FUSEye.Source: A. Experimental Setup · pp. 5–6
  • Table 1 reports direct-transfer results for COCO-pretrained detectors on WoodScape MVR, with results expressed as percentages.The table caption describes backbone blocks and identifies best and second-best distinct results by bolding and underlining.Source: Table 1 · p. 6

B. Main Results

FUSEye improves YOLO26-x over direct transfer while recovering part of the full fine-tuning gains; ablations show complementary effects from GridViews and Z-Adapters, with AgreeFusion’s gains depending on multi-view candidates.

B. Main Results

  • Across tested models, scaling benefits the lower-end variants more, while YOLO26-x leads AP50:95, car, and person and is 0.03 AP50 points below the best family result.Full fine-tuning raises YOLO26-x from 14.80 to 31.56 AP50.Source: B. Main Results · p. 6
  • FUSEye improves YOLO26-x by 11.81 AP50 points and 6.30 AP50:95 points over direct transfer, recovering 70.5% and 53.1% of the respective full fine-tuning gains.It outperforms full fine-tuning on person but remains substantially lower on car; bus also remains a limitation.Source: B. Main Results · p. 6, Table 2 · p. 6

C. Ablation Study

  • GridViews alone yields only a marginal overall AP50:95 change, while Z-Adapters provide a larger overall gain driven mainly by car; combining them gives the best person score without fusion.The combination leaves car slightly below Z-Adapters alone, reflecting a person–car tradeoff.Source: C. Ablation Study · pp. 6–7, Table 3 · p. 7
  • AgreeFusion improves both overall metrics and car and person when added after GridViews and Z-Adapters, but adds no change to Z-Adapters without GridViews.The latter configuration has no cross-view candidates; the table does not isolate the candidate-level mechanism.Source: C. Ablation Study · p. 7, Table 3 · p. 7

D. Training-Set Size

With 25% of the training images, FUSEye retains 97.6% of the full-data result on both overall metrics on average; it also improves AP50 across four additional YOLO variants, though bus scores decrease.

D. Training-Set Size

  • The three 25% training subsets retain 97.6% of the full-data result on both overall metrics on average, but all remain slightly below the full-data result.Without full-data variability estimates, the comparison does not support a significance claim; the 25% subsets contain 1,520 images each.Source: D. Training-Set Size · p. 7

E. Cross-Model Evaluation

  • FUSEye improves AP50 by 5.80–8.29 points for every additional detector, spanning YOLOv8-x, YOLOv10-x, YOLO11-x, and YOLOv9-e.Bus decreases by 0.25–0.75 points in every pair.Source: E. Cross-Model Evaluation · p. 7, Table 5 · p. 7
  • Across the four additional detector families, car gains are 4.80–5.84 points, while person gains vary from 2.99 to 5.55 points.The consistent improvements span different backbone blocks, but are not class-uniform.Source: E. Cross-Model Evaluation · p. 7

F. Qualitative Results

FUSEye recovers peripheral detections missed or weakly detected by the zero-shot baseline, although agreement does not ensure correctness. The conclusion reports gains over direct transfer, including with fewer labels and across additional YOLO detectors.

F. Qualitative Results

  • Qualitative examples show FUSEye recovering peripheral objects missed or weakly detected by the zero-shot baseline, while false positives and localization errors show that agreement does not ensure correctness.Figure 5 columns show ground truth, the zero-shot COCO baseline, and complete FUSEye, with insets enlarging representative detections.Source: F. Qualitative Results · p. 7, Figure 5 · p. 8

V. CONCLUSION

  • FUSEye raises YOLO26-x from 14.80 to 26.61 AP50 and from 10.96 to 17.26 AP50:95, recovering 70.5% and 53.1% of the full fine-tuning gains.It adds approximately 227k parameters; with 25% of the training images, it retains 97.6% of full-data AP50 and AP50:95.Source: §V · pp. 7–8
  • Figure 4 compares C2PSA activation maps for direct transfer and FUSEye, with warmer colors indicating stronger responses.The figure caption identifies the maps and color meaning but does not state an outcome comparison.Source: Figure 4 · p. 8
  • FUSEye improves AP50 by 5.80–8.29 points in four additional YOLO detectors.The conclusion attributes the gains to complementary effects from GridViews, Z-Adapters, and AgreeFusion.Source: §V · p. 8
Loading 2610.02799v1…