Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness

Yehonatan Elisha; Oren Barkan; Noam Koenigstein

Source-linked AI summary

Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness

Yehonatan Elisha, Oren Barkan, Noam Koenigstein

arXiv:2603.08309v2Mar 9, 2026cs.CVcs.AIcs.LG

TL;DR

ViTs often rely on spurious correlations, limiting robustness under distribution shifts, while coarse foreground supervision misses fine-grained object concepts. CFT automatically generates concept masks and aligns relevance with them during fine-tuning; across five OOD benchmarks, it consistently improves robustness and generalizes to unseen classes.

Problem
ViTs rely on spurious correlations such as background cues, and existing foreground–background regularization misses the fine-grained concepts that define objects.
Method
CFT automatically proposes class-relevant concepts, grounds them with GroundedSAM, and fine-tunes relevance maps to align with concept regions while suppressing spurious backgrounds.
Results
CFT consistently outperforms prior saliency-regularization approaches across five OOD benchmarks, with robustness improvements generalizing to classes unseen during fine-tuning.
Takeaways & Limitations
Concept-guided masks provide more effective robustness supervision than conventional segmentation maps and produce relevance maps better aligned with semantic object parts.
Takeaways & Limitations
CFT depends on GroundedSAM’s concept localization quality and adds preprocessing and relevance-map computation costs, especially for datasets with many classes.

Abstract

from arXiv · show

Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations, such as background cues, rather than semantically meaningful features. Existing regularization methods, typically relying on simple foreground-background masks, which fail to capture the fine-grained semantic concepts that define an object (e.g., ``long beak'' and ``wings'' for a ``bird''). As a result, these methods provide limited robustness to distribution shifts. To address this limitation, we introduce a novel finetuning framework that steers model reasoning toward concept-level semantics. Our approach optimizes the model's internal relevance maps to align with spatially grounded concept masks. These masks are generated automatically, without manual annotation: class-relevant concepts are first proposed using an LLM-based, label-free method, and then segmented using a VLM. The finetuning objective aligns relevance with these concept regions while simultaneously suppressing focus on spurious background areas. Notably, this process requires only a minimal set of images and uses half of the dataset classes. Extensive experiments on five out-of-distribution benchmarks demonstrate that our method improves robustness across multiple ViT-based models. Furthermore, we show that the resulting relevance maps exhibit stronger alignment with semantic object parts, offering a scalable path toward more robust and interpretable vision models. Finally, we confirm that concept-guided masks provide more effective supervision for model robustness than conventional segmentation maps, supporting our central hypothesis.

1. Introduction

ViTs can fail under distribution shifts by relying on spurious context rather than semantic object content. CFT addresses this gap by steering relevance toward fine-grained, class-relevant concepts instead of coarse foreground regions.

CFT intuition: Figure 1 shows CFT shifting ViT-B relevance from backgrounds toward meaningful parts, including bird beaks and wings or fish fins and mouths.The examples use ImageNet-A and ImageNet-R, although the model was not fine-tuned on those datasets.
Motivation: ViTs often rely on background textures and contextual cues rather than target-object semantics, causing failures on diverse out-of-distribution data.Reported failure settings include natural adversarial examples, altered viewpoints, and artistic renditions.
Motivation: Existing foreground-guidance methods require extensive retraining or annotated segmentation masks, limiting scalability for large pretrained models.These methods may improve generalization under distribution shifts but impose annotation and efficiency costs.
Motivation: Binary foreground–background masks are too coarse because they treat the object uniformly and overlook discriminative internal parts such as wings and a long beak.The introduction uses bird recognition to illustrate why semantic structure within the foreground matters.

2. Related Work

Prior work studies shortcut learning through challenging distribution-shift benchmarks and explanation-based regularization. These approaches commonly guide model reasoning toward predefined foreground regions using attribution methods or annotated masks.

Robustness and Shortcut Learning: Shortcut learning arises when models use training-data correlations such as background textures that do not generalize to out-of-distribution data.Benchmarks include ImageNet-A, ObjectNet, ImageNet-R, ImageNet-Sketch, and SI-Score.
Robustness and Shortcut Learning: Distribution-shift benchmarks measure vulnerabilities across natural adversarial examples, novel viewpoints, artistic renditions, sketches, and synthetic transformations.Performance is typically compared with in-distribution accuracy on standard ImageNet benchmarks.
Saliency-Guided Model Regularization: Saliency-guided regularization constrains model explanations to predefined foreground regions to combat shortcut learning.Right for the Right Reasons is an example that matches explanations to annotated foreground regions.
Saliency-Guided Model Regularization: Vision attribution methods include gradients, permutation techniques based on Shapley values, and theory-driven Layer-wise Relevance Propagation.These methods provide different ways to identify regions influencing classifier decisions.

3. Method

CFT fine-tunes ViTs to align internal relevance with automatically generated, spatially grounded concept regions while suppressing attribution to non-concept areas. Its objective combines relevance alignment with classification consistency, using AttnLRP-derived relevance maps and validated visual concepts.

CFT fine-tunes ViTs on a small example set by aligning internal relevance with semantically meaningful concept regions.The framework is presented as data-efficient and primarily targets ViTs, with an alternative CNN implementation.
AttnLRP produces patch-level relevance maps by backpropagating the class output score through the ViT.The maps cover the ViT patch grid, and LRP conservation makes relevance a redistribution of the prediction signal.
Class-discriminative textual attributes are automatically proposed and visually grounded into segmentation masks using GroundedSAM.GroundedSAM combines GroundingDINO with SAM, providing zero-shot concept masks without describing manual annotation.
Concept validation retains concepts that occur frequently enough and spatially cover their target class regions.Validation uses occurrence rate and spatial coverage based on mean IoU with class-level segmentation masks.
The alignment loss increases relevance inside concept masks and penalizes residual relevance in non-concept regions.The concept term optimizes attribution over concept pixels, while the non-concept term suppresses background attribution.
A classification-consistency loss preserves the original model’s predicted output distribution during relevance-guided fine-tuning.The final objective combines alignment and classification losses to avoid accuracy collapse while changing explanations.

4. Experiments

Experiments evaluate CFT across OOD benchmarks, implementation settings, qualitative failures, relevance alignment, and ablations. Results indicate improved robustness, generalization beyond fine-tuned classes, and stronger supervision from concept-based than object-based masks.

Experimental setup: CFT is evaluated against GradMask, RRR, and RRDA on five OOD benchmarks, including adversarial, viewpoint, artistic, sketch, and synthetic transformation settings.The evaluation also includes in-distribution accuracy and relevance-map alignment.
Experimental setup: The fine-tuning protocol samples three images per class from half of ImageNet-1K classes, producing a 1,500-image dataset.Classes are selected randomly to provide diverse semantic coverage while testing data efficiency.
Robustness results: CFT corrects qualitative OOD prediction failures by shifting ViT-B relevance from misleading textured backgrounds toward the object’s body and semantic concepts.One example changes a “common newt” prediction from “scorpion” to “common newt.”
Robustness results: CFT achieves substantial gains on IN-A and ObjectNet, while improvements are less pronounced on IN-R and IN-Sketch, where backgrounds are less varied.The reported pattern is consistent with stronger benefits when background cues are misleading.
Robustness results: CFT can slightly reduce IN-V and IN-V2 accuracy, but it produces more pronounced gains on SI-Score’s geometric transformations.The paper relates the SI-Score gains to reliance on object structure and relevant features rather than absolute position or orientation.
Generalization and ablations: Robustness gains extend to non-training classes, while concept-guided masks outperform object-segmentation guidance across in-distribution and OOD datasets.Ablations also identify the non-concept and classification-consistency loss terms as important for OOD robustness.

5. Conclusion

CFT steers model reasoning toward semantic concepts and improves OOD robustness across architectures, including classes excluded from fine-tuning. Ablations support concept-level supervision over conventional foreground–background masks.

CFT is a fully automated framework that redirects classification away from spurious correlations toward semantically meaningful concepts.
CFT consistently improves robustness across five OOD benchmarks and multiple vision architectures.
Robustness gains generalize to classes not observed during fine-tuning, supporting a broader robust reasoning process.
Concept-level masks provide stronger robustness supervision than conventional foreground–background segmentation masks.

6. Implementation Details

CFT uses fixed loss weights across models and datasets, while concept validation filters automatically generated concepts using occurrence and spatial-coverage criteria. The implementation uses 30 images per class and covers half of ImageNet-1K classes.

CFT uses fixed loss weights λ_non-concept = 1.2, λ_concept = 0.5, λ_align = 0.8, and λ_cls = 0.2 across models and datasets.The non-concept loss receives the largest weight because spurious cues are identified as the primary issue to correct.
Concept validation requires occurrence rate ≥15% and spatial coverage ≥20%, producing 1,852 validated concepts on ImageNet.Concepts appeared in 29% of images on average.
The filtering procedure first applies the occurrence-rate threshold and then evaluates the spatial-coverage criterion.In the reported experiments, every concept passing the occurrence-rate filter also satisfied the coverage threshold.
The concept set uses P = 30 samples per class to measure occurrence rate and spatial coverage before filtering candidates.

7. Concept Validation Effect

Concept validation consistently improves CFT performance, while looser or stricter thresholds reduce performance by introducing noise or discarding informative concepts.

With validation, CFT reaches 27.92 on IN-A and 48.51 on IN-R, compared with 26.01 and 47.19 without validation.The validated variant achieves state-of-the-art performance and outperforms other robustness-oriented approaches.
Concept validation is optional, but the validated configuration provides a consistent performance boost over the non-validated variant.
Strict thresholding yields 24.59 on IN-A and 44.23 on IN-R, presumably because many informative concepts are discarded.
Relaxing thresholds to 5%/10% increases the concept count to 2,435 but introduces substantial noise, yielding 25.13 on IN-A and 44.92 on IN-R.

8. Main evaluation - full results

The reported results are averaged over five random seeds while varying the ImageNet classes used for fine-tuning and holding other parameters fixed.

Table 1 results are averaged over five random seeds with the fine-tuning class subset varied and all other parameters fixed.Corresponding standard deviations are reported in Table 9.

9. Limitations and Future Work

CFT has failure modes involving concepts that are not visually grounded, very small object parts, and domain mismatch between the LLM and target imagery. Future work proposes adaptive and compositional concepts, plus applications beyond image classification.

Failure Cases: Abstract or non-visual concepts can produce empty high-confidence masks because GroundedSAM cannot localize them.
Failure Cases: For object parts occupying < 2% of image area, Grounded-SAM’s hit rate decreases, potentially impairing fine-grained part-level reasoning.The reported impact on final accuracy is limited because remaining concepts can provide sufficient coverage.
Failure Cases: In specialized domains such as medical imaging and satellite imagery, GPT-4o-mini’s concept vocabularies may be imprecise or incomplete.
Limitations: CFT depends on GroundedSAM for concept localization, so inaccurate segmentation can degrade guidance masks and limit effectiveness.
Limitations: Concept creation processes 30 samples per class through GroundedSAM, creating a potential preprocessing bottleneck for datasets with thousands of classes.CFT uses 1,500 images, but AttnLRP relevance computation also adds training overhead compared with standard gradient-based methods.
Limitations: Extending CFT to architectures beyond the demonstrated ViTs and CNNs may require additional architectural considerations.
Future Work: Future work could dynamically weight concepts according to their discriminative power or relevance to particular OOD scenarios.Suggested approaches include masking-response methods and concept activation vectors for concept-class importance weights.
Future Work: Future work could model hierarchical and compositional relationships instead of treating concepts as independent entities.The proposed direction builds complex concepts from simpler primitives to reflect real-world object structure.