PaperlayerView exact revision on arXiv

Source-linked AI summary

In-the-Wild Camouflage Attack on Vehicle Detectors through Controllable Image Editing

Xiao Fang, Yiming Gong, Stanislav Panev, Celso de Melo, Shuowen Hu, Shayok Chakraborty, Fernando De la Torre

arXiv:2603.19456v1cs.CV

TL;DR

Vehicle camouflage attacks must deceive detectors while remaining visually convincing, but existing approaches motivate treating realistic appearance manipulation as a controllable editing problem. The paper fine-tunes ControlNet with image-level and scene-level stylization and a unified objective, reporting stronger attacks, improved structure and stealthiness, and transfer to black-box and physical settings.

  • Problem

    Vehicle camouflage attacks seek to deceive detectors through visible appearance changes while remaining stealthy to human observers.

  • Method

    The framework fine-tunes ControlNet to generate image-level and scene-level camouflages using structural, style-consistency, and adversarial objectives.

  • Results

    More than 38% AP50 decrease is achieved, alongside stronger attack effectiveness, better vehicle-structure preservation, improved stealthiness, and transfer to black-box detectors and the physical world.

  • Takeaways & Limitations

    Conditional image editing provides a framework for generating context-aware vehicle camouflage directly on real images while retaining visual coherence and adversarial effectiveness.

  • Takeaways & Limitations

    The digital image-editing pipeline does not explicitly model 3D geometry, material properties, or viewpoint variation, and its image-level strategy is less effective for ground-view images.

Abstract

from arXiv · show

Deep neural networks (DNNs) have achieved remarkable success in computer vision but remain highly vulnerable to adversarial attacks. Among them, camouflage attacks manipulate an object's visible appearance to deceive detectors while remaining stealthy to humans. In this paper, we propose a new framework that formulates vehicle camouflage attacks as a conditional image-editing problem. Specifically, we explore both image-level and scene-level camouflage generation strategies, and fine-tune a ControlNet to synthesize camouflaged vehicles directly on real images. We design a unified objective that jointly enforces vehicle structural fidelity, style consistency, and adversarial effectiveness. Extensive experiments on the COCO and LINZ datasets show that our method achieves significantly stronger attack effectiveness, leading to more than 38% AP50 decrease, while better preserving vehicle structure and improving human-perceived stealthiness compared to existing approaches. Furthermore, our framework generalizes effectively to unseen black-box detectors and exhibits promising transferability to the physical world. Project page is available at https://humansensinglab.github.io/CtrlCamo

1 Introduction

Camouflage attacks alter vehicle appearance to deceive detectors while remaining visually stealthy, and this paper formulates them as controllable conditional image editing. The proposed framework combines two context-aware stylization strategies with structural, stylistic, and adversarial objectives.

  • Motivation: Camouflage attacks manipulate visible vehicle appearance to deceive vision models while remaining stealthy to human observers.The paper treats full-object appearance manipulation as relevant to realistic vehicle-detector threat models.
  • Approach: The framework preserves vehicle structure and background, applies user-guided stealthy style edits, and reduces detector confidence.These three properties define the intended adversarially camouflaged output.
  • Approach: The method fine-tunes ControlNet and optimizes structural-preservation, style-consistency, and adversarial detection losses.At inference, camouflaged images are generated by direct sampling.
  • Stylization strategies: Image-level stylization blends vehicles with immediate surroundings, whereas scene-level stylization adapts appearance to a shared semantic concept in the scene.Scene-level stylization addresses the image-level strategy’s limitation for moving vehicles by providing location-invariant camouflage.
  • Contributions: The paper presents conditional image editing and two camouflage strategies as contributions for attacking detectors on real-world images.The strategies produce context-aware, visually coherent camouflage.
  • Results: The approach achieves strong attack effectiveness, preserves vehicle physical structure, improves stealthiness, and transfers to black-box detectors and the physical world.Experiments evaluate the method on COCO and LINZ datasets.

2 Related Work

Prior camouflage attacks vary in how much of an object they modify, from imperceptible perturbations to localized patches and full-object appearance changes. The cited literature notes that tiny pixel-level changes can have limited practical impact against robust object detectors.

  • Taxonomy: Camouflage-attack methods are grouped into imperceptible perturbations, localized patches, and full-object appearance modifications.The grouping is based on how extensively each method alters an object’s surface.
  • Imperceptible perturbations: Imperceptible attacks use small norm-constrained perturbations or diffusion-guided sampling to reduce detector confidence while keeping changes visually subtle.The passage describes both classical Gaussian-noise refinement and more recent diffusion-based approaches.
  • Imperceptible perturbations: Tiny pixel-level changes are less practically impactful against object detectors because detectors are generally more robust to them than classifiers.The cited passage frames this as a limitation of purely imperceptible attacks on detection systems.
  • Adversarial patches: Adversarial patches restrict modifications to localized regions placed on the target object.The related-work passage introduces patches as a separate surface-modification category.

3 Method

The method uses a two-stage ControlNet pipeline to generate stealthy vehicle camouflage through reference-guided conditional image editing. It combines image-level and scene-level style selection with structural, stylistic, background, adversarial, and color-consistency objectives.

  • Two-Stage Framework: The framework first fine-tunes ControlNet without detector-dependent loss, then further fine-tunes it against a known detector for white-box camouflage attacks.The first stage transfers reference style while preserving geometry; the second adds detector-targeted optimization while retaining visual realism and stylistic attributes.
  • Latent Diffusion Backbone: The diffusion backbone estimates a clean latent from a noisy latent and predicted noise, then decodes it into an image for training and attack optimization.Stable Diffusion encodes images into latent space, while a one-step estimate enables image-space losses on reconstructed outputs.
  • Style Reference Selection: The pipeline selects a reference region or synthesized exemplar to guide vehicle appearance toward surrounding context or a naturally occurring scene concept.Image-level camouflage uses immediate surroundings, whereas scene-level camouflage groups images by scene type and selects a compatible concept through MLLMs.
  • ControlNet Conditioning: ControlNet receives the vehicle luminance channel, style reference, vehicle mask, and, for image-level camouflage, an additional background image.These conditions support structure-preserving, reference-guided vehicle stylization directly on input images.
  • No-Box Attack: The No-Box objective combines structural preservation, style consistency, and background reconstruction losses, with background reconstruction enabled for image-level camouflage.The structural term preserves the vehicle’s LAB luminance structure, while style matching compares masked latent features from the generated vehicle and reference area.
  • Style Loss: Latent style matching masks vehicle and reference features before minimizing their layer-wise average-feature L1 difference, reducing interference from irrelevant regions.The method applies resized masks to latent representations because zero-valued pixels can still produce nonzero latent activations.

4 Experiments

Experiments evaluate attack effectiveness, stealthiness, structural preservation, transferability, and ablations on COCO and LINZ. The method consistently reduces detector performance while preserving vehicle structure, producing context-consistent camouflage, transferring across models and physical setups, and benefiting from background and color-consistency supervision.

  • Experimental Setup: Experiments use COCO and LINZ to assess attack effectiveness, stealthiness, structural preservation, and transferability across white-box, black-box, and physical settings.The evaluation includes Faster-RCNN and ViTDet as white-box targets, with additional black-box testing and projector-based physical experiments.
  • Comparison with State-of-the-art Methods: At least 38.9% AP50 reduction is achieved against AdvCAM and Diff-PGD across datasets, detectors, and strategies, with higher COCO SSIM and faster inference than per-image optimization methods.The reported comparison covers stylization-based attacks and notes that competing methods require per-image optimization during camouflage generation.
  • Comparison with State-of-the-art Methods: The method transfers style according to reference areas and scene context while preserving vehicle geometry, unlike square-patch baselines that produce less coherent stylization.Qualitative comparisons cover COCO and LINZ environments, including lake and parking-lot scenes, with real-image backgrounds.
  • Comparison with State-of-the-art Methods: Scene-level camouflage is preferred by humans in 85.3% of cases, compared with 11.7% for Diff-PGD and 3.0% for AdvCAM; image-level preference is 53.1%.The human study operationalizes stealthiness as perceptual alignment with surrounding context or reference objects.
  • Comparison with State-of-the-art Methods: Over 56% AP50 deduction is achieved across datasets and detectors compared with non-stylization noise and patch attacks.The comparison reports average AP50 for both non-stylization strategies.
  • Transferability: Black-box evaluation shows at least a 47.1% AP50 drop and a 54.5% classification-accuracy decrease, while physical projection produces a clear detection-confidence drop despite moderate appearance shifts.The physical experiment uses reconstructed or 3D-printed scenes, projected camouflage, and smartphone photographs.
  • Ablation Study: Background supervision improves context-consistent stylization, while color-consistency loss limits adversarial color deviations and enhances visual coherence.The ablations evaluate background reconstruction loss L_b and color-consistency loss L_c in the image-level and two-stage settings.

5 Conclusions

The paper presents a conditional image-editing framework for stealthy vehicle camouflage, combining two stylization strategies with structural fidelity and adversarial effectiveness. It also documents implementation details, supplementary evaluations, and limitations concerning physical realism and ground-view imagery.

  • 5 Conclusions: The framework uses image-level and scene-level stylization to generate stealthy vehicle camouflage while preserving structure and reducing detector performance.Image-level camouflage adapts vehicles to nearby surroundings, whereas scene-level camouflage uses semantic concepts present in the broader scene.
  • 5 Conclusions: The method fine-tunes ControlNet within a two-stage framework that jointly enforces vehicle structure, style consistency, and adversarial effectiveness.The pipeline generates camouflaged images directly and can guide camouflaging corresponding 3D real-world vehicles.
  • 5 Conclusions: Experiments on LINZ and COCO show stealthier camouflage, stronger vehicle-structure preservation, and transfer to unseen black-box detectors and the physical world.
  • 5 Conclusions: The digital image-editing formulation does not explicitly model 3D geometry, material properties, or viewpoint variation, limiting direct control over unshaded surface texture.The L channel in LAB color space is used as a coarse shading proxy, but single-view images entangle illumination, geometry, and material effects.
  • C.3 Scene-reference selection: The supplementary workflow selects scene references by predicting and refining scene categories, extracting object information, and synthesizing representative concept exemplars.The refined scene groups include Residential, Industrial, Agricultural, Highway, and Parking lot.
  • C.1 Experimental setup: The supplementary implementation evaluates SD v3.5 with ControlNet fine-tuning, 28 inference sampling steps, and experiments assigning image-level attacks to LINZ and scene-level attacks to COCO.The comparison reports AP50, SSIM, and average sampling time per image.

C.1 Experimental Setup

The experimental setup specifies training choices and implementation details for the proposed method and comparison attacks. Baselines are grouped into imperceptible perturbations, adversarial patches, and stylization attacks, with detector-specific losses used where needed.

  • Training parameters: Training settings include learning rate, iteration count, and coefficients for structure, style, background, color-consistency, and adversarial losses.The coefficients place stronger constraints on style and structure preservation than on adversarial loss.
  • Implementation: The evaluation uses SD v3.5, samples images one at a time, and tests all methods on a single RTX A6000 GPU.
  • Baseline methods: Baseline methods cover imperceptible perturbations, adversarial patches, and stylization attacks.
  • Imperceptible perturbations: TOG injects bounded RGB-space perturbations with a perturbation norm constrained to 100 and 80 optimization iterations per image.
  • Imperceptible perturbations: DiffAttack performs latent-space diffusion optimization and is adapted from classification to detection by replacing its adversarial loss with a detector-specific loss.The setup uses a 50-step diffusion process and 60 adversarial optimization steps.
  • Patches and stylization: NAP, BadPatch, AdvCAM, and Diff-PGD use shared or reference-guided patches with dataset-specific training, inversion, or optimization procedures.For AdvCAM and Diff-PGD, square references are selected from local surroundings or scene reference areas.

C.2 Extended Qualitative Results

Extended visualizations show that both camouflage strategies produce structurally faithful, scene-coherent vehicles across COCO and LINZ. Alternative scene–concept pairings retain comparable quantitative performance, although occasional style misalignment remains.

  • COCO results: COCO visualizations cover urban, rural, road, sky, and lake scenes paired with building, grass, tree, sky, and water concepts.Image-level examples use reference areas extracted from the vehicle’s immediate surroundings.
  • LINZ results: LINZ visualizations cover residential, industrial, agricultural, parking lot, and highway environments using concepts such as house, building, field, tree, and grass.Both image-level and scene-level results maintain realistic structure and visual coherence with their surroundings.
  • Cross-dataset findings: The visualizations corroborate generalization across datasets and stylization modes, consistently producing structurally faithful and stealthy camouflage.
  • Alternative pairings: Alternative scene–concept pairings achieve comparable attack effectiveness and vehicle-structure preservation on both COCO and LINZ.The alternative pairings use platform, tree, road, cloud, and beach for COCO environments, with corresponding alternatives evaluated for LINZ.
  • Alternative pairings: Scene-relevant concepts generally produce coherent and stealthy camouflage, but occasional style misalignment occurs between camouflaged vehicles and reference areas.

C.4 Robustness under defense strategies

The attacks remain effective after common denoising and smoothing defenses, although preprocessing weakens their effectiveness. The reported robustness is attributed to modifying object appearance rather than relying on fragile high-frequency perturbations.

  • Defense evaluation: Common image-preprocessing defenses evaluated on COCO include Non-local Means denoising and bilateral filtering.The evaluation distinguishes clean images, undefended adversarial images, denoised images, and smoothed images.
  • Defense evaluation: Preprocessing weakens attack effectiveness, but large AP50 drops remain across Faster R-CNN and ViTDet under both camouflage strategies.
  • Quantitative results: AP50 decreases from 85.6% to 29.8% under denoising for Faster R-CNN using the image-level strategy.
  • Quantitative results: AP50 decreases from 91.4% to 36.4% under denoising for ViTDet using the image-level strategy.
  • Quantitative results: Under smoothing, AP50 remains below 26% in all reported cases.
  • Interpretation: The authors attribute defense robustness to appearance modification, which is less dependent on fragile high-frequency perturbations.

C.5 Ablation Studies

The ablations examine the two-stage pipeline, structural preservation, style consistency, and stealthiness evaluation. Results indicate that staged training and the structure and style losses support realistic, environment-consistent camouflage.

  • Two-stage vs. One-stage: The two-stage pipeline produces more accurate vehicle stylization than a one-stage variant under the image-level setting.The comparison jointly considers structure preservation, style, adversarial, and background supervision losses.
  • Two-stage vs. One-stage: Early reconstruction failures make adversarial supervision misleading because detectors already assign low confidence to non-vehicle regions.The two-stage design first ensures reliable vehicle reconstruction before adversarial learning, reducing unstable optimization.
  • Structure preservation loss: Removing Lstruct produces heavily distorted shapes that no longer resemble the underlying vehicle.The result shows that structure preservation is needed while adapting appearance to the reference area.
  • Style loss: Without Ls, vehicles tend to receive similar, unnatural colors or textures that may draw human attention.Style loss instead guides appearance transfer from surrounding regions or scene concepts, supporting environmental consistency.
  • Human Evaluation: The human-study protocol evaluates stealthiness through preferences for stylization matching reference areas and for edited-vehicle naturalness.The supplementary evaluation compares these dimensions with state-of-the-art approaches and uses separate ablation analyses.

D.1 Comparison with State-of-the-art Methods

The human evaluation compares perceived stealthiness of the proposed image-level and scene-level camouflage strategies with state-of-the-art approaches. Participants assess how naturally edited vehicles fit their scene contexts.

  • Reference-area matching: The study evaluates stealthiness through preferences for stylization that best matches the reference areas.This comparison is described as one part of the human evaluation against state-of-the-art approaches.
  • Edited-vehicle naturalness: The study separately evaluates preferences for the naturalness of edited vehicles.This forms the second human-evaluation criterion for comparing camouflage methods.
  • Study design: The questionnaire contains 15 LINZ images and 15 COCO images, with three images from each scene type.The two strategy sections are presented in randomized order for each participant.

D.2 Comparison with Ablation Variants

The ablation study separately evaluates perceived stealthiness while accounting for differences in structure preservation and adversarial effectiveness. The complete formulation receives higher human preference than the ablation variants on both datasets.

  • Evaluation rationale: Direct comparison with prior methods is challenging because outputs differ in structure preservation and adversarial effectiveness as well as appearance style.These differences can confound human judgments of style realism.
  • Human evaluation: The complete formulation receives 71.4% preference on LINZ and 94.6% on COCO over ablation variants.Participants select the edited vehicle that appears most natural within the scene context.
  • Study design: The study presents the real image alongside three edited vehicles for each question.The question pool samples images from the datasets and asks participants to choose the most natural version.
  • Location transferability: Scene-level transferability is additionally evaluated by compositing vehicles into five backgrounds within each scene type while keeping vehicle appearance unchanged.This tests whether learned camouflage generalizes across locations within the same scene category.

E.2 Projector-based Physical Experiment

The projector-based experiment tests real-world transferability for image-level and scene-level camouflage on LINZ and COCO. Because the physical sample is small, attack success rate is used instead of AP50.

  • Physical setup: The physical setup projects images onto whiteboards or reconstructed 3D-printed scenes and captures the results with a camera.The LINZ setup uses a 3D-printed car model, while the COCO setup uses reconstructed 3D-printed scenes.
  • Evaluation protocol: The physical evaluation tests five LINZ scene-level images, four LINZ image-level images, and three COCO images for each strategy.The small number of physical test images makes AP50 estimates unreliable.
  • Evaluation protocol: Attack success rate is reported instead of AP50 because a single prediction can substantially alter the precision–recall curve with limited samples.The attack success rate depends only on detector confidence associated with the ground-truth object.
  • Detection criterion: The detection criterion requires confidence above 0.948 for LINZ or 0.806 for COCO and IoU greater than 0.5.These thresholds are selected from validation performance using the best F1 score.
  • Results: 100% attack success rate is achieved for both camouflage strategies on both datasets in the physical tests.All physical test samples are suppressed by the detector under the stated criterion.
  • Results: LINZ confidence scores remain below 0.55 for scene-level and 0.22 for image-level camouflage, while COCO scene-level vehicles are not detected.COCO image-level predictions remain below 0.71, also below the 0.806 threshold.

F Limitations

The pipeline has limitations in reproducing vehicle shading, maintaining perceptual attributes during optimization, and selecting suitable style references for ground-view images.

  • Shading and optimization: The LAB-space L channel only coarsely approximates shading, so camouflaged vehicles may not perfectly retain their original shading patterns.The L channel also carries texture information, creating interactions between structure-preservation and style losses during joint optimization.
  • Shading and optimization: Hue distributions remain strongly aligned with reference regions, but shading or saturation may differ because loss-term interactions introduce subtle appearance shifts.The method reliably transfers the intended color component even when other perceptual attributes are affected.
  • Ground-view reference selection: The image-level strategy is less effective for ground-view images because perspective distortion can make surrounding reference regions spatially distant or semantically unrelated.These regions may provide inconsistent or misleading style guidance, producing appearances that do not align with the vehicle’s immediate context.
Loading 2603.19456v1…