PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain shift-robust object detection with GenAI image editing

Isabel D. Stein, Thijs A. Eker, Sebastiaan P. Snel, Ella P. Fokkinga, Klamer Schutte, Luca Ambrogioni, Friso G. Heslinga

arXiv:2609.02299v1cs.CV

TL;DR

Object detectors degrade under domain shifts, while collecting sufficient target-domain data is often infeasible, leaving out-of-domain robustness underexplored for generative image editing. This paper synthetically adds camouflage to uncamouflaged military-vehicle images using diffusion-based editing and evaluates detector performance on real camouflaged imagery, finding that realistic synthetic camouflage reduces the domain gap and that fine-tuning helps for specialized camouflage.

  • Problem

    Object detectors can rely on domain-specific visual shortcuts, and capturing sufficient real-world target-domain variation for robust detection is often infeasible.

  • Method

    The study adds synthetic camouflage to images from 15 uncamouflaged military vehicle classes using Qwen, Flux, LoRA fine-tuned Qwen, and a black-bar baseline, then trains and evaluates object detectors on real camouflaged imagery.

  • Results

    Realistic synthetic camouflage reduces the domain gap, with Qwen exceeding black-bar augmentation by 6.3 mAP on netting and 15.6 mAP on foliage; LoRA improves performance where zero-shot editing struggles with multi-spectral camouflage.

  • Takeaways & Limitations

    Synthetic augmentation is most effective when its camouflage faithfully represents the target appearance, while zero-shot editing can suffice when models already have adequate prior knowledge.

  • Takeaways & Limitations

    Prompt engineering and LoRA prompt adjustment rely on qualitative assessment and iterative experimentation, and a larger, more balanced test set is needed for robust evaluation.

Abstract

from arXiv · show

Object detectors often degrade under domain shifts such as changes in lighting, weather, or occlusion. These shifts alter object appearance and expose a reliance on visual shortcuts learned from the training distribution that do not generalize across domains. Acquiring sufficient real-world samples to capture such domain variation is particularly difficult in specialized, low-data settings. Recent advances in diffusion-based generative image editing have shown promise for improving the in-domain performance of object detectors through synthetic data augmentation. However, their potential to improve out-of-domain robustness remains largely unexplored. We hypothesize that generative image editing can simulate a controlled domain shift in training data, effectively bridging the gap between source and target domains. To test this, we studied camouflaged military vehicle detection as a challenging domain shift scenario. Detectors trained on uncamouflaged data demonstrate substantial degradation on real test imagery containing foliage, netting, and multi-spectral camouflage across 15 vehicle classes in close-up, ground-level imagery. We used two diffusion-based editing models, Qwen Image Edit 2509 and Flux.2 Dev, to synthetically add camouflage to the training data, alongside a LoRA fine-tuned version of Qwen. A non-generative black-bar occlusion baseline served as a lower bound on augmentation quality. Using a GroundingDINO detector trained on real and synthetic data, generative camouflage augmentation yielded substantial mAP improvements for foliage (+20.1) and netting (+14.4) camouflage. Generating multi-spectral camouflage proved more challenging, but LoRA fine-tuning improved performance by 4.4 mAP over the uncamouflaged baseline.

1. INTRODUCTION

Object detectors can degrade under domain shifts, while collecting enough target-domain data is often infeasible. This work investigates whether generative editing can introduce controlled target-domain appearance variations into training data for camouflaged vehicle detection.

  • Motivation: Domain shift occurs when deployment environments differ from training data, changing the visual cues available for detection.Such shifts can expose reliance on predictive shortcuts that fail to generalize across domains.
  • Motivation: Diffusion-based synthetic augmentation has improved in-domain vision performance, but its use for out-of-domain detection by editing source images remains underexplored.
  • Research question: The proposed hypothesis is that editing source images with controlled target-domain appearance variations can reduce the source-target domain gap while preserving object and scene content.
  • Application: Camouflaged military vehicle detection provides a challenging low-data test case because camouflage occludes cues, reduces object-background contrast, and is difficult to collect and annotate at scale.The dataset contains 15 uncamouflaged military vehicle classes, onto which synthetic camouflage is introduced while preserving vehicle geometry.
  • Study design: The study compares zero-shot and fine-tuned editing pipelines for foliage, netting, and multi-spectral camouflage, with a non-generative augmentation baseline.

2. RELATED WORKS

Prior augmentation methods manipulate existing image information or combine samples, while generative models can synthesize novel semantic and stylistic content. This work builds on diffusion-based augmentation and addresses domain-specific generation challenges in low-data detection.

  • Augmentation principles: Data augmentation aims to encourage domain-invariant features by exposing models to diverse transformations and reducing reliance on source-specific biases.
  • Information dropping: Information-dropping methods simulate occlusion by masking image regions, whereas camouflage reduces object discriminability through concealment without necessarily removing pixels.
  • Sample combination: MixUp, LossMix, and Copy-Paste increase diversity by combining samples or relocating objects, helping weaken spurious object-context associations.
  • Low-data domains: Specialized military vehicle datasets are often small, sensitive, and difficult to expand, limiting conventional augmentation to recombinations of existing visual information.Generative models are motivated as a way to synthesize semantically novel imagery beyond that information.
  • Generative augmentation: Diffusion augmentation has shown benefits across multiple low-data domains, but off-the-shelf models may miss domain-specific geometry, scale, viewpoint, or appearance.LoRA is presented as an efficient fine-tuning approach for specialized applications.
  • Related domain adaptation: DODA bridges domain gaps using layout-conditioned diffusion data guided by unlabeled target-domain reference images, whereas this work focuses on editing source-domain images without that stated requirement.

LoRA

The pipeline optionally uses LoRA fine-tuning to improve synthetic camouflage generation before detector training and evaluation on real camouflaged vehicles.

  • Generation: Generative editing introduces synthetic camouflage into uncamouflaged vehicle images using text prompts.
  • LoRA: LoRA fine-tuning is an optional support for improving generation quality and uses a separate camouflaged-image set not shown in the figure.
  • Detector training: Synthetic and real images are combined into a mixed dataset to fine-tune GroundingDINO, which is evaluated on real-world camouflaged vehicles.

3. METHODS

The experimental setup compares generative and non-generative camouflage augmentation strategies, then evaluates detectors trained on the resulting datasets against real-world camouflaged vehicle imagery.

  • Experimental design: The study constructs multiple synthetic training datasets to test whether camouflage augmentation reduces the gap between uncamouflaged and camouflaged detection.
  • Compared methods: The evaluated approaches are zero-shot Flux.2 Dev, zero-shot Qwen Image Edit 2509, LoRA fine-tuned Qwen, and a non-generative baseline.
  • Pipeline: The generative pipeline combines camouflage-specific prompt engineering, zero-shot or LoRA-fine-tuned image editing, and downstream object-detector fine-tuning.

3.1 Real dataset

The real dataset contains uncamouflaged imagery across 15 military vehicle classes, while camouflage evaluation uses a seven-class subset and three camouflage types. The close-up, ground-level test imagery keeps vehicles visible but challenges fine-grained recognition by obscuring class-specific cues.

  • The uncamouflaged dataset contains 15 military vehicle classes with 360 training, 150 validation, and 432 test images.
  • Camouflaged evaluation is restricted to seven vehicle classes because camouflage imagery is unavailable for the remaining classes.
  • The camouflaged test set covers foliage, netting, and multi-spectral camouflage using publicly available web-sourced imagery.
  • Close-up, ground-level imagery leaves vehicles generally visible, making fine-grained recognition rather than localization the primary challenge.
  • Figure 2 presents the PzH 2000 without camouflage and with foliage branches, standard netting, or multi-spectral camouflage.

3.2 Synthetic camouflage generation

The study generates synthetic camouflage by editing existing vehicle images with Qwen or Flux, and fine-tunes Qwen with LoRA using paired real and synthetic examples. Generated images receive automatic labels, quality filtering, and optional artifact-reduction post-processing.

  • Two open-weight editing models, Qwen and Flux, are evaluated for text-guided camouflage insertion while preserving image identity or scene content.
  • Prompts target foliage, netting, and multi-spectral camouflage while aligning camouflage type, placement, and coverage across models.
  • LoRA fine-tuning: LoRA fine-tuning adapts Qwen with trainable low-rank parameters while keeping the original model weights fixed.
  • LoRA fine-tuning: Separate LoRA adapters use 37 foliage pairs, 24 netting pairs, and 14 multi-spectral pairs, with foliage and netting limited to dominant subtypes.
  • Quality filtering and post-processing: GroundingDINO automatically assigns vehicle boxes and inherited source-image class labels to generated images using 15 class prompts.
  • Quality filtering and post-processing: Quality filtering removes hallucinated additions, structural deformation, incomplete rendering, asset fusion, and physically impossible camouflage placement.
  • Quality filtering and post-processing: Gaussian blurring and a Flux.1 Dev refinement pass are tested to reduce editing artifacts that could encourage detector reliance on synthetic features.

3.3 Non-generative baseline: black-bar augmentation

The non-generative baseline simulates camouflage-related occlusion by overlaying black bars on original images, providing a lower bound for augmentation quality.

  • Black bars are overlaid on original images to simulate information loss from occlusion without generative image editing.
  • The baseline varies bar widths from 15 to 30 pixels and image-area coverage from 20% to 40%.
  • Each image receives a randomly assigned width and coverage combination, with the number of bars determined by the target coverage fraction.

3.4 Experiments

Experiments fine-tune GroundingDINO detectors on mixtures of real and synthetic data, varying synthetic proportions and repeating configurations across random seeds. The evaluation measures mAP on real camouflaged imagery against an uncamouflaged-data baseline.

  • The baseline detector is fine-tuned on 360 uncamouflaged vehicle images before evaluation on camouflaged test imagery.
  • Performance is measured with mAP across IoU thresholds from 0.50 to 0.95, combining classification accuracy with localization accuracy.
  • Synthetic-data proportions range from 0% to 100% in 25% increments, using 1080 synthetic images and oversampled real images to control the ratio.
  • All configurations use comparable computational budgets, and experiments are repeated three times with different random seeds.

4. RESULTS

Synthetic camouflage augmentation improves out-of-domain detection, especially for foliage and netting, while LoRA fine-tuning specializes Qwen for multi-spectral camouflage at an overall robustness trade-off.

  • Zero-shot editing produced visually plausible foliage and netting camouflage, while multi-spectral camouflage became more convincing after LoRA fine-tuning.
  • The baseline detector’s performance fell from 86.6 to 56.8 mAP when evaluated on camouflaged vehicles.
  • 25% synthetic and 75% real training data consistently yielded the best performance across tested synthetic-data ratios.
  • Qwen zero-shot reached 69.9 mAP on overall camouflaged detection, improving by +13.1 mAP over the uncamouflaged baseline.It also exceeded the black-bar baseline by +7.3 mAP and Flux zero-shot by +8.5 mAP.
  • Uncamouflaged-test performance remained stable from 85.6 to 87.7 mAP across augmentation approaches.Neither Gaussian blurring nor a second diffusion pass consistently improved the performance gap between Flux and Qwen.
  • Generative approaches produced the largest gains for foliage and netting, whereas multi-spectral gains were more limited.LoRA increased multi-spectral performance from 70.9 to 76.5 mAP at 25% synthetic data, but overall camouflaged performance was lower than Qwen zero-shot.
  • Black-bar augmentation improved overall camouflaged detection from 56.8 to 62.6 mAP but remained below Qwen zero-shot.Qwen’s advantage was largest for foliage and netting, while multi-spectral performance was comparable between the two methods.

5. DISCUSSION

Synthetic camouflage augmentation reduces the domain gap while preserving uncamouflaged performance, but its benefits depend on camouflage fidelity and target-domain representation. Fine-tuning helps for specialized appearances, whereas evaluation and model-selection choices constrain interpretation.

  • Synthetic camouflage augmentation reduces the domain gap without reducing in-domain performance under the best-performing 25% synthetic-data configuration.Uncamouflaged imagery performance remains stable across augmentation approaches.
  • 56.8 to 62.6 mAP: black-bar occlusion improves overall camouflaged detection despite visually implausible synthetic images.The result indicates that exposure to partial occlusion contributes to augmentation gains.
  • 6.3 mAP on netting and 15.6 mAP on foliage: Qwen outperforms black-bar augmentation when synthetic camouflage reflects the target domain.For multi-spectral camouflage, zero-shot generation does not show the same advantage.
  • LoRA fine-tuning improves multi-spectral detection when zero-shot editing cannot reliably represent the specialized camouflage appearance.Fine-tuning provides no consistent benefit for foliage and netting, where zero-shot editing is already sufficiently representative.
  • Qwen generally produces larger detector gains and lower variance than Flux, apparently because it better preserves vehicle geometry and matches target-domain appearance.The comparison highlights synthetic-data quality as an important determinant of detector performance.
  • The evaluation is constrained by subjective prompt and LoRA development, non-camouflaged checkpoint selection, limited test-set diversity, and use of a single detector architecture.These limitations affect reproducibility, evaluation robustness, and transferability beyond MM GroundingDINO with a Swin-T backbone.

6. SUPPLEMENTARY - EXTENDED RESULTS

Performance generally worsens when synthetic data exceeds 25–50%, especially for Flux.2 Dev, while Qwen degrades more gradually. Post-processing and quality filtering provide limited or inconsistent benefits, with black-bar augmentation an exception on multi-spectral camouflage.

  • Synthetic data ratio: Synthetic-data proportions above 25–50% generally reduce performance, with the strongest degradation observed for Flux.2 Dev.At 100% synthetic data, Flux.2 Dev reaches 30.0 mAP on camouflaged imagery and 28.4 mAP on uncamouflaged imagery.
  • Synthetic data ratio: At 100% synthetic data, Qwen Image Edit 2509 retains 54.9 mAP, indicating a more gradual degradation than Flux.2 Dev.The passage attributes this pattern to greater consistency between Qwen-generated images and the real data distribution.
  • Synthetic data ratio: Synthetic-data overuse can cause detectors to overfit generation artifacts absent from real images, reducing transfer at test time.The reported mechanism is that learned features become tied to synthetic artifacts rather than camouflage-relevant features.
  • Multi-spectral camouflage: Black-bar augmentation was an exception on multi-spectral camouflage, peaking at 77.3 mAP with 50% synthetic data versus 75.4 mAP for the uncamouflaged baseline.The table reports performance across mixtures of real and synthetic data at 0%, 25%, 50%, 75%, and 100% synthetic.
  • Post-processing: Gaussian blurring and a second diffusion pass did not consistently improve performance, and processing reduced the reported result to 54.9 mAP in one comparison.The targeted discrepancies included sharpness, noise, texture, and color differences in Flux-generated camouflage.
  • Quality filtering: Quality filtering had no meaningful effect across generative pipelines despite flagging approximately 10% of generated images as faulty.The filtering result is reported across the evaluated generative pipelines.
Loading 2609.02299v1…