PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Risk in Pretrained Image Encoders

Ardyn Nordstrom, Morgan Nordstrom, Vamuyan Sesay, Matthew D. Webb

arXiv:2609.35470v1econ.GNcs.CVq-fin.EC

TL;DR

Pretrained image encoders are often treated as implementation details, yet plausible choices can yield different downstream conclusions. The paper compares ten frozen encoders across six applied tasks, then evaluates validation-based selection and combination. Encoder rankings vary by task, and combining representations adds less consistent gains than choosing a strong representation.

  • Problem

    Applied image prediction lacks direct comparisons of plausible pretrained encoders, although the choice can change downstream performance and empirical conclusions.

    Source: §1 · p. 1

  • Method

    The paper compares ten frozen encoders across six applied tasks under common downstream protocols and uses separate validation data to select or combine candidates.

    Source: §1 · p. 2

  • Results

    For houses, validation-selected SigLIP 2 raises test R2 from 0.396 for ResNet50 to 0.629, while no encoder wins universally and combination gains are less consistent.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Benchmark plausible representations, select on locked validation data, combine only when validation evidence justifies the inference cost, and report uncertainty and cost.

    Source: §6 · p. 8, §7 · p. 9

    The dataset count is too small to establish a general law linking residual diversity to ensemble gains.

    Source: §6 · p. 9

Abstract

from arXiv · show

Applied researchers increasingly convert images into features with pretrained encoders, then use those features in a downstream prediction model. The encoder is often treated as an implementation detail. We show that it can instead be a consequential source of model uncertainty. We call this uncertainty representation risk: plausible pretrained encoders map the same images into different feature spaces and can yield sharply different out-of-sample conclusions from predictive performance. We compare ten modern and legacy frozen encoders across applications involving house prices, racehorse performance, breast-cancer histology, chest radiographs, continuous facial age, and rice disease. With common dimension control, heads, and group-safe splits, validation selects SigLIP 2 for houses, raising test $R^2$ from 0.396 for ResNet50 to 0.629, and DINOv2 for horses, raising $R^2$ from 0.029 to 0.105. No encoder is best in every task. Candidate procedures are constructed using training data and compared on a separate validation partition. The selected procedure reaches 0.658 for houses and 0.979 accuracy for pneumonia. Fixed-split gains are small for horses and rice, while repeated partitions reveal instability in horse feature union. Continuous age selects SigLIP 2 at 4.786 years MAE. The principal representation gaps persist with neural heads, similarly sized DINOv2 and ViT models, and limited adaptation. These results support a simple workflow: benchmark plausible representations, select on locked validation data, combine only when separate validation evidence justifies the additional cost, and report paired and split-level uncertainty. We implement this workflow in LOOKAGAIN-ML, the software package used to conduct the analyses in this paper.

1 Introduction

Pretrained encoder choice can materially change out-of-sample prediction, making representation selection a substantive source of uncertainty. Across applications, validation selects different encoders, while combining strong representations provides smaller, less consistent gains.

1 Introduction

  • For houses, validation-selected SigLIP 2 raises test R2 from 0.396 for ResNet50 to 0.629, and a linear stack reaches 0.658.The representation change accounts for most of the improvement; stacking adds a further gain after strong representations are identified.Source: §1 · p. 1
  • Representation risk is the uncertainty that plausible pretrained encoders of the same images yield different downstream performance and empirical conclusions.The paper treats this as an applied model-selection problem, not a new learning paradigm.Source: §1 · p. 1
  • For horses, DINOv2 raises workout-speed R2 from 0.029 to 0.105, while feature union reaches 0.111 with a small, uncertain gain over DINOv2.Across twelve group-safe splits, DINOv2 remains selected, but feature union has one extreme test-performance failure.Source: §1 · p. 2
  • Across six applied image tasks, encoder performance varies materially and the validation-selected encoder differs by task; combination helps in some tasks but is no reliable default.The paper introduces LOOKAGAIN-ML, the software package used for its analyses and common workflow.Source: §1 · p. 2

2 Related Work

The paper situates its work alongside transferability, ensemble, and applied-image research, while distinguishing its empirical focus on representation choice from established methods for combining models.

Pretrained representations and model selection

  • Prior transferability work estimates or ranks how pretrained representations may transfer, while this study compares locked-split outcomes of plausible encoders on applied tasks.The related work distinguishes these applied regression and classification settings with group-level leakage control from standard classification-transfer benchmarks.Source: §2 · p. 3

Ensembles and feature fusion

  • Averaging, stacking, and feature fusion are established methods; the paper instead separates representation-choice consequences from conditional combination value using validation-locked decisions.It relates potential gains to representation strength, error complementarity, and inference cost.Source: §2 · p. 3

Images in applied prediction

  • The house application predicts property-level sale prices while holding sample, split, head family, preprocessing, and tuning rule common across encoders.The paper compares encoder choice across economic, medical, biological, and demographic applications rather than designing a domain-specific multimodal system.Source: §2 · p. 3

Transfer variation, underspecification, and analytic flexibility

  • Representation risk concerns variation in held-out performance across defensible representations, unlike underspecification accounts centered on near-equivalent performance or conflicting predictions.The paper also connects its framing to multiverse and specification-curve analysis, which jointly report defensible analytic choices.Source: §2 · p. 3

3 Framework

The framework compares frozen encoders under common downstream conditions, defines representation risk through performance sensitivity, and separates validation-based selection from combination. It also diagnoses complementarity and reports uncertainty conditional on fitted models and splits.

3 Framework

  • Within each dataset, the framework holds observations, split, preprocessing discipline, and head family fixed across frozen encoders to compare held-out loss.The loss difference captures the practical consequence of representation choice, including how readily a common head extracts task-relevant information.Source: §3 · p. 3
  • Representation risk is the sensitivity of downstream performance to the choice among defensible pretrained encoders, measured through performance spread, paired differences, and task-varying selections.The definition includes linear accessibility, with nonlinear heads used to test whether conclusions persist.Source: §3 · p. 4
  • Combination candidates include linear stacking, feature union, and sparse averages; group-safe cross-fitting and training-only fitting precede comparison on outer validation.After selection, candidates are refitted using training plus validation, while test outcomes remain excluded from fitting and model selection.Source: §3 · p. 4
  • Standalone performance and pairwise prediction and error correlations distinguish encoder strength from complementarity; these diagnostics are not a universal selection formula.For classification, the primary error-diversity measure is correlation of per-observation log loss.Source: §3 · p. 4
  • Paired 95% intervals resample locked test groups 2,000 times from fixed fitted predictions, while twelve group-safe splits provide a separate stability summary.Models are not retrained within bootstrap draws, so intervals condition on fitted models and the locked split.Source: §3 · p. 4

4 Experimental Design

The experimental design compares frozen encoders across six tasks using group-aware partitions and common dimension control and prediction heads. Fixed and repeated splits, automated checks, and a locked-test comparison support the evaluation.

4 Experimental Design

  • The six-task designs use saved leakage groups for houses, reconstructed identity groups for horses, patient-aware splits for histology and chest X-ray, and duplicate-image groups for age and rice.Where no defensible pre-existing split exists, the study uses one locked 70/15/15 split and twelve alternative group-safe splits fixed in advance.Source: §4 · p. 4, Table 1 · p. 5
  • Ten frozen encoders are compared after train-only PCA with a 128-dimensional cap, using standardized ridge or regularized logistic heads.Regression and classification tasks are evaluated with task-appropriate metrics, including R2, MAE, accuracy, and log loss.Source: §4 · p. 4
  • Figure 2 uses the same observations, dimension control, and ridge-head protocol within each motivating application, and its final row shows the procedure selected on outer-validation loss.The selected procedure is a linear stack for houses and feature union for horses.Source: Figure 2 · p. 5
  • Automated checks verify data and cache alignment, group/hash separation, train-only scaling and PCA, group-safe meta-model fitting, and untouched test outcomes.Caches without recoverable row identifiers are excluded.Source: §4 · p. 5

5 Results

Across six tasks, validation-selected encoder performance varies by task, and combinations improve on selected single representations only in some settings. Repeated splits, robustness checks, and inference costs qualify the size and reliability of those gains.

5.1 Representation choice changes prediction

  • SigLIP 2 raises house test R2 from 0.396 to 0.629 and DINOv2 raises horse R2 from 0.029 to 0.105; validation winners vary by task.Validation selects SigLIP 2 for houses and continuous age, DINOv2 for horses and rice, VGG16 for BreaKHis, and ConvNeXt for pneumonia; rice remains at a test-accuracy ceiling.Source: §5.1 · p. 5, Table 2 · p. 6, Figure 3 · p. 7

5.3 Split stability separates large gaps from close calls

  • Across twelve splits, houses gain about 0.030 R2 on average, age selects SigLIP 2 throughout, and pneumonia gains about 0.018 accuracy on average.Mean age MAE is 4.762 for the selected single and 4.638 for the preferred procedure, with preferred-procedure split SD 0.094.Source: Table 3 · p. 6, §5.3 · p. 6

5.2 Combination helps conditionally

  • For houses, a training-OOF linear stack raises R2 from 0.629 for SigLIP 2 to 0.658; BreaKHis averaging raises accuracy from 0.878 to 0.931.The house gain is 0.029 (95% CI [0.017, 0.042]); pneumonia feature union raises accuracy from 0.965 to 0.979, while other combinations are conditional.Source: §5.2 · p. 6

5.3 Split stability separates large gaps from close calls

  • Horse feature union has a test R2 failure of −49.112 and a preferred-procedure mean of −4.009, although DINOv2 is selected in all twelve splits.The numerical safeguard does not constrain test predictions, and the precise cause of the failure was not verified.Source: Table 3 · p. 6, §5.3 · p. 7

5.4 Dimension, capacity, transferability, and data geometry

  • Dimension-controlled and similar-scale comparisons retain representation gaps; limited adaptation leaves a 0.216 house R2 gap, while LogME agrees with validation winners in 4/6 tasks.For age, excluding test observations above the fixed cosine-similarity threshold changes DINOv2 MAE from 5.908 to 5.915; rice remains a ceiling case even after excluding highly similar images.Source: §5.4 · p. 7

5.5 Representation risk survives nonlinear heads

  • With neural heads, the DINOv2–ResNet50 R2 gap remains 0.225 for houses and 0.071 for horses; median within-task encoder-ranking correlation is 0.962.Nonlinearity helps selectively, supporting linear heads as the primary comparison and neural heads as a robustness check.Source: Figure 4 · p. 8, §5.5 · p. 8

5.6 Cost and deployment

  • Combinations require every constituent encoder at deployment, and latency was not measured; image-only encoder rankings are distinct from incremental gains over structured covariates.Extraction measurements vary with hardware, batch size, sample, and I/O, so they do not establish that extraction dominates fitting costs.Source: §5.6 · p. 8

6 Discussion and Practical Recommendation

The recommended workflow benchmarks plausible encoders, selects procedures on separate validation data, and combines representations only when validation gains warrant their cost. LOOKAGAIN-ML implements this process, while the paper notes limits in the available evidence and encoder coverage.

6 Discussion and Practical Recommendation

  • Benchmark plausible representations, select on separate validation data, combine only when validation gains justify inference cost, and report uncertainty and cost.Equal averaging can add disagreement without useful signal when weak encoders are included.Source: §6 · p. 8
  • LOOKAGAIN-ML implements the workflow with cached frozen embeddings, group-safe splits, training-only out-of-fold meta-models, paired intervals, and extraction-cost measurements.The package makes encoder choice explicit and verifiable, but does not eliminate model uncertainty or promise a universal winner.Source: §6 · p. 9
  • The dataset count is too small to establish a general law linking residual diversity to ensemble gains, and the encoder zoo is necessarily incomplete.Bootstrap intervals condition on fixed fitted predictions, while twelve splits address only one additional uncertainty source.Source: §6 · p. 9

7 Conclusion

Across six applied tasks, encoder choice can materially change out-of-sample performance, and no encoder wins universally. Validation-based combination helps in some settings but may add little or fail under feature shift.

7 Conclusion

  • Across six applied tasks, changing the encoder can alter out-of-sample performance enough to change predictive-performance conclusions, and no encoder wins universally.The principal representation gaps persist with flexible neural heads.Source: §7 · p. 9
  • Validation-based combination adds gains in some tasks, but adds little in highly correlated or ceiling-effect settings and can fail under feature shift.The conclusion recommends comparing plausible encoders, selecting without reusing assessment outcomes, and combining selectively with uncertainty and cost reported.Source: §7 · p. 9

Reproducibility Statement

The paper reports that its results were generated with accompanying code and checked against saved analysis outputs. Replication materials include package code, configurations, aggregate outputs, and public or synthetic examples.

Reproducibility Statement

  • Primary analyses used LOOKAGAIN-ML commit 877908c916573109b2106cd5fec82ec83a6153de, while neural-head analyses used commit d3d96e09bb49f33dcf3e5cb0a875ac5d661ae0ad.The reported results were checked against saved analysis outputs.Source: Reproducibility Statement · p. 9
  • Replication materials include package code, configurations, aggregate outputs, and public or synthetic examples.The repository's TensorFlow Flowers demonstration illustrates the software workflow and is not one of the six analyzed applications.Source: Reproducibility Statement · p. 9
  • Restricted datasets and row-level derivatives are not distributed and must be obtained independently under applicable access terms.Code and replication materials are available at the stated project repository.Source: Reproducibility Statement · p. 9

Ethics Statement

The study uses existing datasets and treats predictions as methodological examples rather than decision tools. The authors caution that facial and medical applications can encode biases and require domain-specific review before deployment.

Ethics Statement

  • Predictions are methodological examples and are not intended for diagnosis, valuation, lending, or personnel decisions.Source: Ethics Statement · p. 9
  • Facial and medical applications can encode demographic or institutional biases.The authors report aggregate predictive performance and avoid causal claims.Source: Ethics Statement · p. 9
  • The authors do not recommend deployment without domain-specific fairness, privacy, and external-validity review.The study uses existing medical, demographic, property, agricultural, and auction-image data.Source: Ethics Statement · p. 9

AI Use Statement

The analyses define representation risk as downstream performance sensitivity to encoder choice and compare frozen representations with group-safe data splits and validation-based selection. Encoder rankings vary across tasks, while combinations and robustness diagnostics reveal conditional gains and instability.

AI Use Statement

  • Generative AI tools assisted with code execution, result cross-checking, literature discovery, and drafting, while the authors retain responsibility for the analyses and claims.The authors also remain responsible for interpretation against saved machine-readable outputs.Source: AI Use Statement · p. 9

A Dataset Construction, Splits, and Quality Control

  • The horse sample links identifying information conservatively into 266 multi-row groups, with maximum group size three and zero crossing identities in fixed and repeated assignments.Fuzzy names and test-informed links are excluded.Source: A Dataset Construction, Splits, and… · p. 13

A.1 Outcome and task notes

  • The tasks cover house prices, within-sale horse workout speed, benign-versus-malignant histology, pneumonia, continuous facial age, and four-class rice disease.Age uses chronological-age labels and MAE in years as its primary metric.Source: §A.1 · p. 13

B Nested Validation, Selection, and Refit

  • Every outer split uses group-disjoint training, validation, and untouched test partitions, with training-only fitting and validation-only candidate comparison before final evaluation.Base-head penalties and combination procedures are constructed within outer training; test outcomes are reserved for final evaluation.Source: B Nested Validation, Selection, and… · pp. 13–14

C Complete Fixed-Split Results

  • Across twelve splits, SigLIP 2 is selected for continuous age in every split, with mean single MAE 4.762 and mean preferred-procedure MAE 4.638 (split SD 0.094).The primary age-selection metric is MAE in years.Source: §A.1 · p. 13, C Complete Fixed-Split Results · p. 15

D Combination, Diversity, and Sequential Selection

  • House residuals retain more complementarity among strong encoders, while horse residuals are highly correlated for many pairs; disagreement alone does not establish useful signal.The correlation matrices are descriptive diagnostics, not a universal selection formula.Source: D Combination, Diversity, and… · p. 15, Figure 5 · p. 15

D.1 Horse feature-union instability

  • Horse feature union reaches test R2 −49.112 in a severe split; a possible explanation is an ADE20K feature with little training variation but extreme test values.The numerical safeguard does not constrain such test values, and test-informed clipping is not applied.Source: §D.1 · p. 16

E Dimension Control, Capacity, and Transferability

  • With dimension-controlled features, matched DINOv2-B/14 exceeds supervised ViT-B/16 by mean R2 0.178 for houses and 0.078 for horses across twelve splits, but this does not isolate a causal training-objective effect.LogME winner agreement is 4/6, and its rice disagreement and low BreaKHis association rule it out as a universal replacement for validation.Source: E Dimension Control, Capacity, and… · p. 16, Table 8 · p. 16

F Limited Adaptation and Data-Geometry Diagnostics

Limited adaptation and data-geometry checks qualify the encoder comparisons: the house adaptation experiment is narrowly scoped, while age and rice diagnostics examine close neighbors and sample-size sensitivity.

Limited adaptation

  • In a separate fixed-split house experiment, limited adaptation yields test R2 0.426 for ResNet50 and 0.642 for DINOv2, with paired difference 0.216 (95% CI [0.142, 0.274]).Earlier encoder parameters are frozen; the experiment uses validation MSE for early stopping and no further validation comparison or train-plus-validation refit.Source: F Limited Adaptation and Data-Geometry… · p. 16

Data-geometry diagnostics

  • For continuous age, removing eight DINOv2 test rows with cosine similarity at least 0.99 changes MAE from 5.908 to 5.915; exact hashes remain the only identity safeguard.The authors characterize this as a sensitivity analysis, not proof of person-level separation.Source: F Limited Adaptation and Data-Geometry… · p. 16
  • For rice, 281/759 test observations (37.0%) have a nearest training image at cosine similarity at least 0.99, and all share its class.A 1-NN classifier obtains 1.000 accuracy, and DINOv2 remains perfect after close-neighbor rows are excluded.Source: F Limited Adaptation and Data-Geometry… · p. 16
  • Across ten training-subsample seeds, rice mean test accuracy rises from about 0.936 with 50 images per class to approximately 0.999 with the full training sample.The authors suggest similar images throughout the dataset make the rice task relatively easy.Source: F Limited Adaptation and Data-Geometry… · p. 17

G Neural-Network Head Ablation

The public TensorFlow Flowers demonstration applies the workflow to compare frozen representations and combinations under a locked, group-safe design. SigLIP 2 is selected as the single encoder, while stack gains are small and split-dependent.

G Neural-Network Head Ablation

  • Across tasks, the median within-task Spearman correlation between linear and neural encoder rankings is 0.962, so neural heads do not systematically eliminate ranking differences.Neural-head experiments reuse cached representations, apply the same 128-component cap, and average predictions across five fixed seeds.Source: G Neural-Network Head Ablation · p. 17, Table 9 · p. 17, Figure 6 · p. 18

I Bootstrap and Selection Details

  • Paired intervals use 2,000 group-aware test resamples of fixed predictions, recomputing metrics without retraining and conditioning on fitting, selection, and the locked split.Classification selection uses validation log loss even when accuracy is reported; test-best columns remain descriptive diagnostics.Source: I Bootstrap and Selection Details · p. 18

J Reproducible Public Demonstration with TensorFlow Flowers

  • The Flowers analysis demonstrates LOOKAGAIN-ML’s public reproducible workflow rather than adding a seventh headline application or replacing restricted-data analyses.The example holds observations, labels, head family, and split fixed to examine representation and combination differences.Source: J Reproducible Public Demonstration… · p. 18, §J.5 · p. 22

J.1 Data preparation and locked design

J.2 Representation comparison

  • SigLIP 2-B/16 has the lowest outer-validation log loss, 0.0397, and locked-test accuracy 0.9836; DINOv2 is close, while semantic class-share representations are weaker.The ten frozen representations share a multinomial logistic head; selection uses validation log loss.Source: §J.2 · p. 19, Table 13 · p. 20, Figure 7 · p. 20

J.3 Combination and paired uncertainty

  • The stack’s outer-validation log loss is 0.0448 versus 0.0397 for SigLIP 2; stack-minus-single test differences are 0.0000 for accuracy and 0.0180 for log loss.The selected primary procedure remains the single encoder.Source: Table 14 · p. 21, Figure 8 · p. 21, §J.3 · p. 21

J.4 Supplementary split stability

  • Across twelve group-safe splits, SigLIP 2-B/16 is selected in 12 of 12 and averages test accuracy 0.9852 (SD 0.0042); the stack is preferred in five splits.The stack averages test accuracy 0.9861 (SD 0.0037); empirical ranges measure partition sensitivity, not confidence intervals.Source: Table 15 · p. 21, §J.4 · p. 21, Figure 9 · p. 22

H Computational Cost

  • Representation-creation times ranged from 20.5 to 84.6 seconds for 3,668 images on the replication machine’s RTX 3060, excluding model-download time.These device- and I/O-specific measurements are not deployment latency or a standardized encoder comparison.Source: §J.1 · p. 19, Table 11 · p. 19, §J.5 · p. 22
Loading 2609.35470v1…