PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Shortcut Learning: Texture-Penalized Prototype Networks

Akshay Anilkumar Girija, Elena Hoemann, Frank Köster, Sven Hallerbach

arXiv:2609.07504v1cs.CV

TL;DR

CNNs favor local textures over semantic geometry, while existing shape-bias approaches can trade clean accuracy or deployment efficiency. TPPN combines a texture-penalty branch with spatial prototypes; on cue-conflict data it reports lower texture bias than the ViT-B/16 benchmark.

  • Problem

    CNNs favor local, high-frequency texture over semantic geometry, while stylized training can degrade unperturbed accuracy and ViTs can impose deployment overhead.

    Source: §I · p. 1

  • Method

    TPPN combines a gradient-reversal texture-prediction branch that penalizes high-frequency features with spatial prototypes, without relying on specialized external datasets.

    Source: §I · pp. 1–2

  • Results

    TPPN records 29.73 ± 1.55% texture bias versus 39.78% for ViT-B/16, and a 40.47 ± 2.40% deception failure rate versus 54.67%.

    Source: B. Texture-Shape Disentanglement on… · p. 5

  • Takeaways & Limitations

    Within the controlled evaluation setting, TPPN offers a structural alternative to expensive stylized datasets and exhibits resilience to severe image perturbations.

    Source: §VI · p. 6

Abstract

from arXiv · show

Standard Convolutional Neural Networks (CNNs) exhibit severe performance degradation due to a strong inductive texture bias that prioritizes local, high-frequency patterns over global structural shapes. This dependency causes confident misclassifications during textural changes or environmental effects. To address this flaw, this study introduces the Texture-Penalized Prototype Network (TPPN), a novel architectural framework that shifts this inherent bias without depending on resource-intensive augmented datasets. Specifically, a Texture-Penalization Branch (TPB) imposes a penalty to suppress the extraction of local texture proxies, forcing the network backbone to discard high-frequency cues and extract purified, shape-biased representations. By evaluating similarities within a prototype-based hypersphere derived from the final convolutional features, the approach enforces strict geometric constraints, treating objects as compositions of essential parts to achieve robust classification. Evaluations on texture-shape cue-conflict datasets and synthetic noise benchmarks demonstrate the stronger shape bias of this structural disentanglement. The proposed framework reduces the inherent texture bias of a baseline ResNet-50 from 55.11% to 29.73%, surpassing the texture-suppression capabilities of an off-the-shelf Vision Transformer (ViT-B/16). Furthermore, the approach demonstrates robust generalization under cue-conflict conditions, resisting textural shortcut learning when encountering Out-of-Distribution (OOD) shapes. The model maintains stronger shape accuracy against elevated perturbations. On clean validation data, the architecture incurs a minimal drop in accuracy of 0.90 percentage points. This provides a structural, efficient solution to CNN texture bias.

I. INTRODUCTION

CNNs favor local, high-frequency texture over semantic geometry, making them vulnerable to deceptive textures and environmental shifts. TPPN combines texture suppression with spatial prototypes to encourage shape-biased representations without specialized external datasets.

I. INTRODUCTION

  • Standard CNNs prioritize local, high-frequency surface statistics over semantic geometry, leaving perception vulnerable in dynamic environments.Deceptive textural overlays or environmental shifts can trigger misclassification despite unchanged object geometry.Source: §I · p. 1
  • Stylized training can encourage shape bias but may reduce accuracy on unperturbed data, while ViTs’ computational overhead can hinder real-time edge deployment.The paper presents TPPN as an alternative that does not rely on specialized external datasets.Source: §I · p. 1
  • TPPN pairs a Texture-Penalization Branch with spatial prototypes to shift CNN representations from local texture toward global shape without specialized external datasets.The branch uses on-the-fly texture proxies and gradient reversal to penalize high-frequency feature extraction.Source: §I · pp. 1–2
  • The authors report that TPPN representations resist cue-conflict conditions and extreme perturbations while preserving clean baseline accuracy.The contribution statement describes these outcomes without supplying numerical results here.Source: §I · p. 2

II. RELATED WORK

Prior shape-bias approaches trade accuracy or deployment efficiency, motivating a CNN-compatible alternative that combines texture penalization with prototype-based reasoning. TPPN extracts projected spatial features from ResNet-50’s deepest convolutional stage, Layer 4.

II. RELATED WORK

  • Matching Transformer-like shape bias with CNN efficiency remains a challenge; TPPN repurposes gradient reversal to penalize features that identify synthetic high-frequency perturbations.Prototype-based spatial similarity complements this penalty by structuring classification around learned geometric parts.Source: §II · p. 2

III. METHODOLOGY

  • TPPN combines high-frequency texture penalization with hyperspherical structural features to decouple geometric shape from surface texture.The framework is described as a sequence of mechanisms detailed in later methodology subsections.Source: §III · p. 2

A. Deep Semantic Feature Extraction

  • The feature-extraction stage uses ResNet-50 Layer 4 and projects its feature map through a 1 × 1 convolution, batch normalization, and ReLU into a lower-dimensional spatial tensor.The projected tensor is denoted Z4, with d as its bottleneck dimension.Source: A. Deep Semantic Feature Extraction · p. 2

B. Hyperspherical Prototype Matching

TPPN compares projected spatial features with learnable class prototypes on a unit hypersphere, then pools the strongest spatial similarities to reduce background interference.

B. Hyperspherical Prototype Matching

  • Learnable class prototypes and spatial features are projected onto a unit L2-hypersphere so semantic similarity is separated from feature magnitude.Their raw cosine similarity is computed as a dot product.Source: B. Hyperspherical Prototype Matching · p. 2

C. Dynamic 50 % Spatial Pooling

  • Dynamic Top-K pooling averages the top 50% of spatial similarities rather than aggregating the entire map, which can dilute geometric signals with background pixels and spatial noise.The pooled similarity is the mean of the selected top-K values.Source: C. Dynamic 50 % Spatial Pooling · p. 2

D. Feature Purification via Texture Penalization

The Texture-Penalization Branch predicts synthetic texture proxies from raw Layer 4 features while gradient reversal penalizes the backbone for recognizing them. This bottleneck is intended to reduce reliance on texture and favor global shape.

D. Feature Purification via Texture Penalization

  • The TPB connects to unprojected Layer 4 features and predicts which of four texture proxies—clean, blur, speckle noise, or color jitter—was applied.Adaptive average pooling and a texture-head MLP process the features around the gradient reversal layer.Source: D. Feature Purification via Texture… · p. 2
  • The gradient reversal layer acts as an identity in the forward pass and reverses the gradient by −λAdv during backpropagation, while the texture head learns to identify the perturbations.The backbone is optimized to maximize texture-prediction error.Source: D. Feature Purification via Texture… · p. 2
  • Penalizing extraction of signals needed to identify noise or blur is intended to make the latent representation discard natural textures and rely on global shapes.During inference, the adapted backbone processes cue-conflict or perturbed images.Source: D. Feature Purification via Texture… · p. 3, Figure 1 · p. 3

E. Temperature Scaling and Reasoning Head

The reasoning head scales prototype similarities before converting them into shape predictions. The composite objective combines classification and contrastive learning with prototype diversity and texture suppression.

E. Temperature Scaling and Reasoning Head

  • Scaled similarities feed a bias-free linear layer that produces the final shape prediction vector.Target-prototype connections are initialized to 1.0 and non-target connections to −0.1.Source: E. Temperature Scaling and Reasoning… · p. 3

F. The Composite Objective

  • The total objective combines cross-entropy classification with a contrastive pull loss on scaled similarities.Cross-entropy is applied to zshape; the pull loss uses a dynamic margin to mask low-similarity target prototypes and encourage use of all m prototypes.Source: F. The Composite Objective · p. 3
  • A diversity loss enforces orthogonality among same-class prototypes to prevent feature collapse and promote broad spatial coverage.The diversity term is applied to the m prototypes within each class.Source: F. The Composite Objective · pp. 3–4
  • The texture-penalization loss is cross-entropy over four texture predictions, optimized through the Gradient Reversal Layer.Source: F. The Composite Objective · p. 4

IV. EXPERIMENTS

An empirical testing paradigm is established to evaluate the proposed TPPN.

  • The study establishes an empirical testing paradigm for evaluating TPPN.Source: §IV · p. 4

A. Datasets and Evaluation Metrics

The evaluation tests shape and texture dependencies using clean validation and cue-conflict data, with specialized models and pretrained benchmarks as comparisons.

A. Datasets and Evaluation Metrics

  • Training uses a balanced 10-class ImageNet subset, while clean validation and cue-conflict images test classification and shape-versus-texture behavior.The training subset contains 20,000 images; the filtered cue-conflict set contains 1,050 images, including ID-shape/OOD-texture and OOD-shape/ID-texture subsets, and is tested at four speckle-noise levels.Source: A. Datasets and Evaluation Metrics · p. 4
  • Clean-set Top-1 and Top-3 accuracy are reported alongside cue-conflict metrics for shape accuracy, texture bias, and texture deception vulnerability.Cue-conflict metrics distinguish shape recognition with ID or OOD textures and the rate of being fooled by ID texture on an OOD shape.Source: A. Datasets and Evaluation Metrics · p. 4

B. Baselines and Ablation Configurations

  • Comparisons include ResNet-50 baselines, pretrained VGG-16 and ViT-B/16, and a TPPN warmup checkpoint.Pretrained-model outputs are mapped from 1,000 ImageNet classes to 10 broad categories by taking the highest confidence score within each group; the warmup checkpoint tests prototypes trained without TPB influence.Source: B. Baselines and Ablation Configurations · p. 4

C. Implementation Details

The implementation trains TPPN in warmup and texture-penalization phases. On clean validation data, it retains accuracy close to the ResNet-50 baseline.

C. Implementation Details

  • Training uses a 10-epoch frozen-backbone warmup followed by 30 epochs of joint backbone and texture-head optimization.The penalization weight is ramped from 0.0 to 1.0; training uses differential learning rates and early stopping based on a composite validation-accuracy and texture-accuracy metric.Source: C. Implementation Details · p. 4

V. RESULTS AND DISCUSSION

  • Cue-conflict performance is reported as mean and standard deviation across five random-seed runs; baselines are generally single-run and pretrained models zero-shot.Source: §V · p. 4

A. Performance on ImageNet Validation Dataset

B. Texture-Shape Disentanglement on Cue-Conflict Data

On clean cue-conflict data, TPPN combines texture suppression with spatial prototypes to reduce texture bias and improve shape accuracy over ResNet-50. It also shows more stable texture predictions across familiar and unfamiliar shapes.

B. Texture-Shape Disentanglement on Cue-Conflict Data

C. Shape Resilience Under Texture Perturbation

As speckle noise increases, TPPN retains shape accuracy better than the compared baselines, with its largest reported advantage at the highest perturbation level.

C. Shape Resilience Under Texture Perturbation

VI. CONCLUSION AND FUTURE WORK

The conclusion presents TPPN as a structural approach to CNN texture bias, reporting texture suppression and resilience to severe perturbations in a controlled evaluation. Future validation is needed beyond the restricted class subset.

VI. CONCLUSION AND FUTURE WORK

  • TPPN uses a Texture-Penalization Branch to penalize synthetic perturbation recognition, reporting stronger texture suppression than off-the-shelf ViT baselines and resilience to severe perturbations in the controlled evaluation.The conclusion states that it reduces texture dependence by half without expensive stylized datasets.Source: §VI · p. 6
  • The evaluation uses a restricted class subset; scaling to ImageNet-1K and finer-grained hyperparameter ablations remains necessary to validate broad applicability.Source: §VI · p. 6
  • Future work will explore deeper MLP configurations for the texture head to capture complex noise patterns.Source: §VI · p. 6
Loading 2609.07504v1…