Source-linked AI summary
CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
Mintu Dutta, Ritesh Vyas, Mohendra Roy *
TL;DR
Self-supervised I-JEPA predicts masked-region representations but does not explicitly model responses to specified visual interventions. CI-JEPA predicts intervention-induced latent changes and evaluates semantic-versus-nuisance sensitivity; its Flowers102 results show a relative ordering for the tested changes, not complete nuisance invariance.
Problem
I-JEPA predicts latent representations of masked image regions, but its objective does not explicitly model how representations respond to specified visual interventions.
Source: §1 · p. 1, Motivation for the Counterfactual Block · p. 2Method
CI-JEPA adds a counterfactual branch that predicts the latent representation change between original and modified images and assesses selective sensitivity to semantic versus nuisance interventions.
Source: Motivation for the Counterfactual Block · p. 3Results
78.14% best validation accuracy versus 77.55% for both pretrained ViT-B/16 and original I-JEPA; mean L2 changes were 4.42 for center occlusion, 3.48 for background tint, and 2.83 for background blur.
Source: §3.3 · p. 7, Robustness Assessment and Limitations · p. 8Takeaways & Limitations
The response ordering is consistent with relative selectivity for the evaluated interventions, not complete nuisance invariance or proof of causal feature discovery.
Source: Robustness Assessment and Limitations · p. 8, §4 · p. 9Baseline encoders lack reported intervention effects, so these results do not establish improved selective sensitivity over the baselines.
Source: Robustness Assessment and Limitations · p. 8
Abstract
from arXiv · showhide
Self-supervised visual representation learning learns useful features without manual annotations during representation training. The image-based joint-embedding predictive architecture (I-JEPA) predicts latent representations of masked image regions, but its objective does not explicitly model responses to specified visual interventions. We introduce CI-JEPA, a counterfactual intervention-aware extension that learns to predict the representation change $\Delta Z = Z_{\mathrm{CF}} - Z$ between an original image and a modified counterpart. We assess representation robustness through selective sensitivity: stronger responses to task-relevant semantic changes than to nuisance changes. Experiments on Flowers102 use flower-center occlusion as a candidate semantic intervention and background blur and tint as candidate nuisance interventions. With frozen-encoder linear probing, CI-JEPA achieves a best validation accuracy of 78.14\%, compared with 77.55\% for both the pretrained ViT-B/16 and the I-JEPA baseline, a gain of 0.59 percentage points. The reported mean $L_2$ representation changes are 4.42 for center occlusion, 3.48 for background tint, and 2.83 for background blur. This ordering is consistent with relative semantic selectivity for the evaluated interventions, rather than complete nuisance invariance. The accuracy comparison is complementary and does not establish improved robustness over the baselines. These controlled image modifications provide a framework for studying intervention-induced changes in JEPA representations; they do not establish causal feature discovery or robustness to all visual changes.
1 Introduction
The introduction asks whether representations respond more to potentially discriminative flower information than to nuisance backgrounds. CI-JEPA makes intervention-induced representation changes part of learning and evaluates robustness as selective sensitivity.
1 Introduction
- I-JEPA predicts latent representations of masked image regions from visible context rather than reconstructing pixel values.The target encoder is updated by an exponential moving average of the context encoder, providing a slowly changing target representation.Source: §1 · p. 2
- The central question is whether an encoder responds differently to potentially discriminative object changes and nuisance background changes.The motivation is that recognition accuracy alone cannot show whether a model relies on the object or incidental information.Source: Motivation for the Counterfactual Block · p. 2
- Flower-center occlusion is treated as a candidate semantic intervention, while background blur and tint are candidate nuisance interventions.The expected ordering is an experimental assumption: class variation, intervention magnitude, and editing artifacts may affect measured responses.Source: Motivation for the Counterfactual Block · p. 3
- The counterfactual branch trains a predictor to estimate the latent representation change between paired original and modified images.Unlike adding modified images as samples, the objective explicitly relates an intervention to its effect in latent space.Source: Motivation for the Counterfactual Block · p. 3
- Robustness is operationalized as greater sensitivity to semantic counterfactuals than to nuisance counterfactuals, with relative stability under nuisance changes.The criterion does not require every modification to leave the representation unchanged, since removing relevant object information may legitimately change it.Source: Motivation for the Counterfactual Block · p. 3
- The work adds an intervention-aware branch, generates controlled flower-image counterfactuals, measures representation changes, and compares semantic with nuisance interventions.Its stated robustness assessment is based on relative sensitivity to those intervention types.Source: Motivation for the Counterfactual Block · p. 3
2 Background
Prior work uses counterfactuals for explanations and robustness tests, while this work focuses on intervention-induced changes in I-JEPA representations.
2 Background
- Counterfactual learning compares model outputs for an original image and a modified version in which part of the input changes while other information remains fixed.The background places this work alongside research on explainability and causal and interventional representation learning [9].Source: §2 · p. 3
- Counterfactual visual explanations have replaced selected image regions to change predicted classes, while other work generated natural-language explanations or examined causal factors in learned features.The cited examples include spatial replacement, class-discriminative language explanations, and interventions on underlying factors.Source: §2 · p. 3
- Unlike approaches focused on final class labels, this work applies controlled visual interventions to I-JEPA and measures changes in internal representations.It compares nuisance background modifications with semantic flower-center occlusion.Source: §2 · p. 3
3 Proposed Methodology
CI-JEPA adds a counterfactual branch to I-JEPA that predicts intervention-induced representation changes alongside masked-region representations. On Flowers102, its frozen-encoder linear-probing accuracy is highest among the evaluated models, while intervention responses show relative selectivity rather than complete nuisance invariance.
3.1 Overview
- The counterfactual branch uses paired original and modified images to predict ΔZ = ZCF − Z, while retaining masked-region prediction as a complementary objective.An intervention embedding and binary mask condition the predictor; the counterfactual loss measures how accurately it models intervention-induced change.Source: §3.1 · p. 4
3.2 Experimental Setup
- Experiments used Flowers102 and compared frozen-encoder linear probes for pretrained ViT, original I-JEPA, and counterfactual I-JEPA; interventions included center removal, background tint, and blur.Representation robustness was assessed by comparing sensitivity to semantic changes with sensitivity to nuisance changes.Source: Dataset · p. 5, Counterfactual Interventions · p. 6, Evaluation Methods · p. 6
3.3 Results
- 78.14% best validation accuracy for the counterfactual I-JEPA model exceeds the 77.55% achieved by both pretrained ViT-B/16 and original I-JEPA.These are linear-probing results; the reported 0.59-percentage-point gain is not established as statistically significant.Source: Table 1 · p. 7, §3.3 · p. 7, Robustness Assessment and Limitations · p. 8
- Mean L2 changes were 4.42 for center occlusion, 3.48 for background tint, and 2.83 for background blur, consistent with relative selectivity rather than complete nuisance invariance.The ordering does not establish improved selective sensitivity over baseline encoders; intervention effects for those baselines are not reported.Source: Robustness Assessment and Limitations · p. 8, Table 2 · p. 8
4 Conclusion
The paper introduces a counterfactual intervention-aware I-JEPA extension and evaluates robustness by whether representations respond more to semantic than nuisance changes. The reported ordering supports relative selectivity for the tested interventions, not universal robustness or causal feature discovery.
4 Conclusion
- The counterfactual block predicts intervention-induced changes, while representation distances and the Selective Intervention Ratio assess the desirability of the response pattern.The evaluation treats stronger responses to semantic changes alongside relative stability under nuisance changes as the desired pattern.Source: §4 · p. 8
- Flower-center occlusion produced larger representation changes than background blur or tint, consistent with relative semantic selectivity for the evaluated interventions.This does not guarantee robustness to all visual changes or establish causal feature discovery.Source: §4 · p. 9
- Future work will compare intervention sensitivity across baselines, control intervention strength and edited area, and extend evaluation to concept-level interventions and additional datasets.Source: §4 · p. 9