PaperlayerView exact revision on arXiv

Source-linked AI summary

RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

Arnold Brosch, Kira Maag

arXiv:2610.37015v1cs.CVcs.CRmath.PR

TL;DR

Semantic-segmentation backdoors can redirect victim-class pixels to a target while preserving benign predictions, and existing defenses are often adapted from classification or require retraining and model modification. RePAIR uses an unlabeled, potentially contaminated image pool and surviving internal victim semantics to localize and selectively repair affected pixels without retraining. Across eight Cityscapes attack configurations, it substantially reduces attack success while retaining segmentation utility across convolutional and transformer architectures.

  • Problem

    Existing segmentation-backdoor defenses are comparatively scarce and often adapt classification defenses or require clean data and model modification, despite the need to correct attacked pixels selectively.

    Source: §1 · p. 1

  • Method

    RePAIR is a post-hoc, label-free, attack-unaware defense that derives a victim-target relation and safety-approved reference from an unlabeled image pool, then selectively repairs pixels using internal victim evidence.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    Across eight Cityscapes attack configurations, defended ASR is 0.46%-7.86% for DeepLabV3+ and at most 0.54% for SegFormer, with measured attack success fully suppressed in six SegFormer settings.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Across convolutional and transformer architectures, surviving internal semantic evidence can support attack-unaware localization and constrained pixel-wise repair without trigger reconstruction or model modification.

    Source: §6 · pp. 9–10

    Reference-calibration budgets do not provide population-level guarantees for unseen benign inputs.

    Source: §3 · p. 6

Abstract

from arXiv · show

Despite significant advances in computer vision, semantic segmentation models remain vulnerable to backdoor attacks, posing a security risk. These attacks redirect pixels of a chosen victim class to an attacker-specified target class when a trigger is present, while preserving clean predictions. Existing defenses are largely derived from image classification, while dedicated segmentation defenses remain scarce and typically require retraining or model modification. We introduce RePAIR, a post-hoc, label-free, attack-unaware defense based on the observation that backdoor-induced shifts can be stronger in the classifier head than in internal representations, leaving recoverable victim-class semantics. RePAIR infers the victim-target relation from an unlabeled, potentially contaminated image pool, constructs a safety-approved semantic reference, and localizes suspicious pixels by comparing head predictions with internal semantic evidence. It then applies feature-space and output corrections toward the inferred victim semantics, while conservative localization and output constraints limit changes to benign predictions. We model the attack as adversarial probability transport and repair as its approximate reversal, deriving bounds and sufficient conditions for localization, residual attack success, clean-side changes, defense-induced target predictions, and attack-unaware pair identification. Experiments across segmentation architectures and attack settings show that RePAIR substantially reduces attack success while largely preserving clean performance.

1 INTRODUCTION

Semantic-segmentation backdoors can cause targeted pixel relabeling while preserving benign predictions, but existing defenses often detect attacks or require clean data and model modification. RePAIR uses surviving internal victim semantics to localize and selectively repair affected pixels, with an attack-unaware reference-building process and architecture-specific semantic evidence.

1 INTRODUCTION

  • Existing defenses often detect suspicious inputs or retrain models, motivating selective pixel repair that limits changes to unrelated regions.Segmentation backdoors can relabel a victim-class object as a target such as road while leaving the rest of the image prediction largely unaffected.Source: §1 · p. 1
  • RePAIR exploits cases where the final head shifts more strongly toward the target than internal representations, leaving victim semantics partially recoverable.The useful representation depends on architecture: DeepLabV3+ can use a pre-ASPP backbone stage, whereas SegFormer reconstructs victim evidence from multiple blocks.Source: §1 · pp. 1–2
  • RePAIR treats targeted attacks as directed probability transport from victim to target and approximates its reversal using model-derived evidence and reference geometry.The victim-target pair is treated as a hypothesis and identified without prior knowledge of either class.Source: §1 · p. 2
  • The post-hoc method selectively corrects attacked pixels without retraining or globally modifying the deployed network.Its theoretical framing connects surviving internal evidence to localization, attack-unaware pair identification, residual attack success, and clean-side safety.Source: §1 · p. 2
  • The method infers victim-to-target evidence and screens candidate clean-like images with a one-way victim-safety firewall before building its repair reference.Images with severely degraded internal victim evidence, or moderate degradation paired with an unusually strong victim-to-target shift, are rejected; the method does not pad an undersized reference set.Source: §1 · p. 2
  • The method develops architecture-specific semantic references, with required repair information concentrated in one representation for some models and distributed across layers for others.The supplied example contrasts a pre-ASPP reference for DeepLabV3+ with a learned multi-layer read-out for SegFormer.Source: §1 · p. 2

2 RELATED WORK

Prior segmentation backdoor work expanded from fixed triggers and victim-target mappings to flexible, contextual, and automatically optimized attacks. Existing defenses largely adapt classification methods, while RePAIR is positioned as post-hoc selective repair without clean labels or model modification.

Backdoors Attacks

  • Segmentation backdoors progressed from fixed triggers and predefined victim-target mappings to placement-dependent, contextual, and automatically optimized attacks.Examples include Object-Free Backdoor Attack, Contextual Segmentation, and BADSEG’s multiple attack types and conditional variants.Source: §2 · p. 3

Defense Methods

  • Many defenses detect suspicious inputs or representations, while pruning, fine-tuning, and Segment Cleanse mitigate attacks through retraining or other model modification.The described pruning and fine-tuning approaches require clean data for mitigation.Source: §2 · p. 3
  • RePAIR differs by performing post-hoc selective pixel repair without retraining, ground-truth labels, or an external clean reference set.Unlike detection-based methods, it repairs affected predictions; unlike ABL, it does not operate during training.Source: §2 · p. 3

3 THEORETICAL FRAMEWORK

The framework models a targeted segmentation backdoor as directed victim-to-target probability transport, with surviving internal victim evidence providing a signal for localization and repair. It gives conditions and bounds for localization, residual attack success, pair identification, and clean-side changes, while distinguishing this analytical model from the implemented approximate reversal.

3 THEORETICAL FRAMEWORK

  • A targeted backdoor is modeled as spatially selective probability transfer from victim class v to target class t, potentially leaving internal victim evidence despite a target prediction.The model describes a dominant directed failure, not a globally one-dimensional poisoned network or an observed clean distribution.Source: §3 · p. 4
  • RePAIR approximates reverse victim-directed transport from the discovered pair, surviving internal evidence, and safety-approved reference geometry; the transposed coupling is not itself the inverse operator.The framework makes no optimal-transport claim without a specified transport cost.Source: §3 · p. 5
  • When the released pair is correct, the localization miss rate is bounded by ηref, the fraction of successful attacks lacking direct victim evidence.Additional residual branches can only reduce the miss rate further.Source: §3 · p. 5
  • Residual attack success depends on missed attacks, repair failures on localized pixels, and new target predictions outside the original attack event.The No New Target Veto prevents unsupported target predictions outside the authorized repair support.Source: §3 · p. 5
  • The attacked pair is uniquely identifiable under a sufficient separation condition: its attack-induced advantage must exceed competing natural evidence and estimation error.This condition is not a distribution-free guarantee for the implemented multi-route selector.Source: §3 · p. 5
  • Clean prediction changes are bounded by benign-pixel repair selection and changes outside the repair mask; exact preservation occurs when both terms are zero.The reference-calibration budgets constrain the safety-approved population, not unseen benign inputs.Source: §3 · pp. 5–6

4 METHOD

RePAIR freezes the compromised model and uses an unlabeled, potentially contaminated image pool to discover a victim–target pair, build a safety-approved reference, and selectively repair localized pixels. Its pipeline combines architecture-specific internal evidence, conservative localization, feature transport, and constrained output correction.

Reverse-transport interpretation

  • The pipeline approximates reverse probability transport using the discovered semantic direction, victim-side reference geometry, and pixel- and branch-dependent repair magnitudes.It does not estimate the hidden attack operator, transported mass, or transport coupling.Source: §4 · pp. 6, 8

Architecture-specific semantic reference

  • RePAIR compares head predictions with architecture-specific internal semantic evidence: CNNs use final-backbone features, while transformers combine evidence across selected blocks.The transformer readout uses victim/target pseudo-labels from the frozen head, not semantic or attack labels.Source: §4 · p. 6

Attack-unaware pair discovery

  • RePAIR discovers a directed victim–target relation from model-derived evidence, without prior knowledge of either class, and uses that pair to guide localization and repair.CNNs use stable internal-to-head contradiction; transformer pair candidates are evaluated with image-disjoint evidence and conservative release rules.Source: §4 · pp. 6–7

Safety-approved reference state

  • The builder screens candidate images sequentially for unstable predictions and victim-class degradation, then leaves repair inactive if reliable support or semantic coverage is insufficient.Rejected images are never reintroduced, and thresholds are not relaxed; runtime references and calibrations come only from the final safety-approved population, whose true cleanliness is not guaranteed.Source: §4 · p. 7

Pair-consistent localization

  • Pair-consistent localization selects pixels using semantic contradiction and residual tests, with CNN and transformer branches applying different cues to head-target pixels.For transformers, sequential mutually exclusive stages define the union mask; probability-residual calibration uses an empirical reference-pixel budget distinct from the repair margin.Source: §4 · p. 7

Feature-first repair

  • Feature-first repair retrieves safety-approved victim references after removing the victim–target direction from the search geometry, then transports each localized feature toward its victim reference.The retrieved feature preserves its original norm and is passed through the unchanged remainder of the network; repair strength depends on the branch.Source: §4 · p. 7

Output correction

  • If feature repair leaves a target prediction, a calibrated, potentially partial output correction modifies only the victim and target logits.The correction is restricted to the discovered pair and calibrated from safety-approved victim references.Source: §4 · p. 8

Output safety

  • The No New Target Veto restores the original head prediction when repair creates the discovered target outside the authorized mask, while leaving existing target predictions unchanged.The constraint is intentionally asymmetric and applies only to creation of the discovered target.Source: §4 · p. 8

5 EXPERIMENTS

Experiments compare RePAIR with four defenses across two segmentation architectures, multiple attacks, and three datasets. On Cityscapes, RePAIR consistently reduces attack success and retains substantial benign segmentation performance; results on PASCAL VOC show a different mitigation range, while ADE20K lacks a sufficiently strong undefended backdoor for a meaningful mitigation study.

5.1 EXPERIMENTAL SETTING

  • The evaluation uses Cityscapes, PASCAL VOC 2012, and ADE20K with DeepLabV3+ and SegFormer, comparing pruning, fine-tuning, Segment Cleanse, and ABL.It tests attacks at a 10% poison rate and reports ASR, benign mIoU, and attacked mIoU.Source: §5.1 · p. 8

5.2 NUMERICAL RESULTS

  • RePAIR maintains Cityscapes mIoU-B at 74.50% for DeepLabV3+ and 72.30% for SegFormer, while achieving the highest mIoU-A among evaluated defenses.The undefended backdoored models have mIoU-B of 73.28% and 75.80%, respectively; ABL substantially degrades mIoU-B.Source: §5.2 · p. 9, Table 1 · p. 9
  • ADE20K's evaluated attacks do not create a sufficiently strong undefended backdoor effect for a meaningful mitigation study.The corresponding results are reported in Table 5.Source: §5.2 · p. 9
  • On PASCAL VOC, RePAIR reduces ASR from a range of 12.21%–99.71% across attacks to 5.41%–19.67%.The paper attributes differences from Cityscapes partly to VOC's typical single foreground class and aggregated background, which affect learned representations and defense evidence.Source: §5.2 · p. 9

6 CONCLUSION

RePAIR uses surviving internal victim-class evidence to localize and selectively repair backdoor-affected pixels without retraining. Across the evaluated Cityscapes settings, it substantially reduces attack success while retaining segmentation utility, and its results support internal semantic evidence as a basis for constrained repair.

Conclusion

  • Across eight Cityscapes attack configurations, RePAIR reduces ASR to 0.46%–7.86% for DeepLabV3+ and at most 0.54% for SegFormer, with complete measured suppression in six SegFormer settings.Average mIoU-B is 74.50% for DeepLabV3+ and 72.30% for SegFormer, versus 73.28% and 75.80% for the respective undefended models.Source: §6 · p. 9
  • The directed-transport formulation treats attacks as victim-to-target shifts, while internal representations provide model-derived evidence for selective reversal across convolutional and transformer architectures.The experiments show effective repair without explicitly reconstructing triggers or modifying the compromised model.Source: §6 · pp. 9–10
  • The paper provides an implementation covering pair selection, reference construction, localization, feature transport, and output safety across its three datasets and two architectures.Attack and defense configurations, metrics, and implementation details are described in the paper and appendix.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 THEORY: FULL STATEMENTS AND PROOFS

The theory section formalizes the local transport operator and the proof obligations underlying the paper's theoretical claims.

A.1 THEORY: FULL STATEMENTS AND PROOFS

  • The section formalizes the local transport operator's domain and feasibility conditions.Source: §A.1 · p. 13
  • It distinguishes the inverse transport operator from the transposed transport coupling.Source: §A.1 · p. 13
  • It describes the operator's operational interpretation for RePAIR and the proof obligations underlying the theoretical claims.Source: §A.1 · p. 13

A.1.1 PROBABILITY SPACE AND STANDING CONVENTIONS

The appendix defines triggered and clean evaluation distributions, attack-success measures, and conventions for unknown-pair release and localization. Its guarantees distinguish correct pair discovery from merely identifying the target, and use fail-closed behavior when no active state is released.

A.1.1 PROBABILITY SPACE AND STANDING CONVENTIONS

  • The notation excludes pixels labeled ∅ from semantic metrics and defines zero-prevalence conditionals as zero, leaving positive-mass conditionals unchanged.The conventions also make products in subsequent bounds unambiguous.Source: §A.1.1 · pp. 13–14
  • ASRbase is the probability that the undefended model predicts the target on triggered victim pixels, while ASRdef measures the same event after defense.The victim class is v and the target class is t.Source: §A.1.1 · p. 13
  • The unknown true pair (v, t) is either released as a discovered pair in an active state or inference abstains and releases an inactive, fail-closed state.Cpair requires an active release with the exact true pair; Cpair implies Ctgt, which requires only the true target.Source: §A.1.1 · p. 13
  • With no active state, fail-closed inference leaves the undefended prediction unchanged and sets localization masks to zero.Under Cpair, the semantic arm marks pixels predicted as the target that also have internal victim evidence.Source: §A.1.1 · pp. 13–14
  • Internal victim evidence is defined by architecture: a CNN head representation predicts v, while a transformer’s internal support exceeds a frozen, pair-specific clean-null threshold.The official transformer threshold is calibrated as the 0.995 empirical quantile of image-disjoint out-of-fold scores on the final safety-approved reference population.Source: §A.1.1 · p. 13

A.1.2 FIXED-SIMPLEX GEOMETRY

The fixed-simplex model represents an attack as moving probability mass from victim class v to target class t. This shift changes their pairwise margin, but crossing their boundary alone does not guarantee that t becomes the overall prediction.

A.1.2 FIXED-SIMPLEX GEOMETRY

  • The directed displacement d_vt = e_t − e_v transfers ξ probability mass from victim v to target t; for 0 ≤ ξ ≤ p*_v, the victim coordinate remains nonnegative and q stays in the simplex.Since ψ_vt(p) = p_v − p_t, this transfer gives ψ_vt(q) = ψ_vt(p*) − 2ξ.Source: §A.1.2 · p. 14
  • The signed Euclidean distance to the v–t boundary uses the victim-positive margin and the boundary normal e_v − e_t, whose norm is √2.The distance follows from the pairwise margin divided by the normal’s norm.Source: §A.1.2 · p. 14
  • The pairwise ordering q_t > q_v follows when the transported mass crosses the stated boundary threshold, with equality leaving the pair tied.This condition concerns only v and t; t must also exceed every other class coordinate to become the argmax.Source: §A.1.2 · p. 14

A.1.3 FORMAL TRANSPORT OPERATOR AND COUPLING

The formal transport operator moves a feasible amount of probability mass from the victim coordinate to the target coordinate and has an explicit inverse on its image. A transposed coupling represents reverse mass flow as a joint measure, not as that inverse operator.

A.1.3 FORMAL TRANSPORT OPERATOR AND COUPLING

  • For fixed ξ, T_ξ maps simplex points with p_v ≥ ξ to points with q_t ≥ ξ by moving ξ from v to t, and the map has a mutual inverse on those domains.The inverse subtracts ξ from q_t and adds it to q_v.Source: §A.1.3 · p. 15
  • The one-edge coupling assigns ξ to the v-to-t entry and leaves the remaining mass on the diagonal, coupling the forward distributions p and q.Transposing that joint measure gives a feasible coupling from q back to p.Source: §A.1.3 · p. 15
  • The transposed coupling is not the inverse operator, and neither coupling is claimed optimal without a specified transport cost.The transpose exchanges the coupling’s marginals; it describes reverse mass flow as a joint measure.Source: §A.1.3 · p. 15

A.1.4 INTERNAL-REFERENCE SURVIVAL AND LOCALIZATION

The localization analysis relates missed baseline attacks to the absence of internal victim evidence, under correct pair discovery and a stated mask condition. Additional localization branches may reduce misses without weakening this inclusion.

A.1.4 INTERNAL-REFERENCE SURVIVAL AND LOCALIZATION

  • Under Cpair and Equation 17, localization misses among successful baseline attacks are bounded by the corresponding cases lacking internal victim evidence.The proof uses the semantic arm, which marks target-predicted pixels when direct victim evidence is present.Source: §A.1.4 · p. 15
  • Additional residual branches may reduce localization misses beyond the direct semantic arm but cannot invalidate the inclusion.Source: §A.1.4 · p. 15

A.1.5 DIRECTED PAIR EVIDENCE AND POPULATION SEPARATION

This appendix part examines how attack-unaware evidence can distinguish directed pairs and how mapping, reference construction, and localized feature and output repair are constrained, including implementation boundaries and conditional guarantees. It also describes attack-independent experimental settings, with ADE20K as an attack-effectiveness control and VOC as a transfer setting.

A.1.5 DIRECTED PAIR EVIDENCE AND POPULATION SEPARATION

  • A sufficient scalar-separation condition makes the true pair’s empirical evidence score exceed every competitor, subject to a uniform score-difference error bound.The condition is sufficient, not necessary, and does not guarantee the complete transformer mapping selector.Source: §A.1.5 · p. 16

A.1.6 TRANSFORMER MAPPING RELEASE CONTRACT

  • The transformer builder releases a directed pair only for a viable frozen family with a uniquely accepted orientation; otherwise it returns no mapping and permits no later family fallback.This is a deterministic contract on observable builder evidence, not a guarantee that the released pair is the true attacked pair.Source: §A.1.6 · p. 17

A.1.7 REPAIR-STATE SAFETY

  • Reference safety is one-way: vetoed references cannot re-enter later repair banks, and the implementation abstains if reference-count or semantic-coverage requirements cannot be maintained.The implementation requires at least 24 safety-approved images and 75% semantic-class coverage; capacity fallback may use only the already-approved reservoir without relaxing thresholds.Source: §A.1.7 · p. 18, §A.2.3 · p. 22

A.1.8 ORTHOGONAL SEMANTIC MATCHING

  • Orthogonal retrieval is invariant to displacement along the victim–target axis and does not assume that an attack lies entirely on that axis.The residual logit substep changes only victim and target logits, uses a zero-sum correction to meet a safety-calibrated victim-over-target margin, and chooses the minimum-norm feasible adjustment; runtime may apply only a fraction of it.Source: §A.1.8 · p. 18, §A.1.9 · p. 18

A.1.10 RESIDUAL ATTACK SUCCESS

  • Residual attack success is bounded using baseline attack success, localization error, repair error, and newly created target predictions.When the discovered target is correct, the no-new-target veto confines new target predictions to the supported repair region; target-correct transformer localization within the original target-predicted region implies no newly created true-target attacks.Source: §A.1.10 · p. 19, §A.1.11 · p. 19

A.1.12 CLEAN-SIDE INTERVENTION

  • Clean prediction changes are bounded by localized clean intervention and nonlocalized spillover, and exact preservation requires both terms to be zero.The calibration budget is empirical on the safety-approved reference population, not a population-level guarantee; the theory also does not claim an oracle clean gate, genuinely clean references, or a distribution-free guarantee that the released pair is true.Source: §A.1.12 · p. 20, §A.1.13 · p. 20

A.2.1 INFORMATION BOUNDARY AND STATE CONSTRUCTION

  • RePAIR makes decisions from a frozen model and an unlabeled pool without using annotations, clean/poison provenance, attack identity, or the true pair; insufficient mapping confidence, reference capacity, or semantic coverage leaves it inactive.CNN and transformer pair evidence is architecture-specific; transformer proposals resolve the target before the victim and require separation from alternatives, while CNN localization combines semantic contradiction, geometric residuals, and probability residuals.Source: §A.2.1 · p. 21, §A.2.2 · p. 21

A.2.4 LOCALIZATION AND REPAIR

  • For CNNs, runtime localization combines direct semantic contradiction, reference-calibrated geometric residuals, and a semantic probability residual.The broader analysis treats transport as a local abstraction with feasible, not necessarily optimal, coupling, and does not assume the backdoor acts only on two probabilities or along one internal dimension.Source: §A.1.13 · p. 20, §A.2.4 · p. 22
  • RePAIR repairs localized features toward victim semantics and may apply a conservative output correction, without retraining or permanently modifying network parameters.An output veto reverts unsupported target predictions outside localized support to prevent repair from introducing them.Source: §A.2.4 · p. 22

A.3 DETAILS ON EXPERIMENTAL SETTING

  • RePAIR hyperparameters are fixed independently of attack identity and evaluation performance, with data-dependent thresholds derived from a frozen safety-approved reference using image-disjoint calibration.The settings do not use oracle victim-target pairs, provenance, attack masks, ground-truth labels, or evaluation ASR for tuning.Source: §A.3 · p. 24

A.4 ADDITIONAL EXPERIMENTAL RESULTS

  • ADE20K attacks did not establish a meaningful backdoor effect, so RePAIR was not applied and the results serve only as an attack-effectiveness control.The evaluated configurations therefore do not provide evidence of defense performance on ADE20K.Source: §A.4 · p. 25, Table 5 · p. 25
  • On VOC, all evaluated attacks use an O2B relation, while RePAIR receives no victim-target pair; the supplied tables report ASR and segmentation utility but provide no cell values here.Only states released by fail-closed construction are reported, without oracle victim or target information during construction.Source: §A.4 · pp. 25–26, Table 6 · p. 26, Table 7 · p. 26

Related papers

Loading 2610.37015v1…