PaperlayerView exact revision on arXiv

Source-linked AI summary

When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

Ahmad AlMughrabi, Albert Clop, Benjamin Busam, Ricardo Marques, Petia Radeva

arXiv:2608.21098v1cs.CV

TL;DR

It is unclear when combining hand-crafted knowledge with learned representations helps, substitutes, or harms. Across a cost-normalized benchmark of 13 datasets and nine backbones, the paper finds that different-currency sources can stack, same-currency sources substitute, and full-strength fusion can interfere.

  • Problem

    It remains unclear when fusing hand-crafted and learned information helps, substitutes, or harms under controlled comparisons.

  • Method

    The paper benchmarks a fixed hand-crafted prior against data-driven alternatives under one frozen, cost-normalized recipe and uses frozen-feature diagnostics to assess combinations.

  • Results

    Across measured combinations, different-currency sources stacked, same-currency sources substituted, full-strength fusion interfered, and frozen-feature gain predicted end-to-end gain to within 0.17 points.

  • Takeaways & Limitations

    A source’s scarce currency can be measured from its frozen features before combination, helping distinguish complementary, redundant, and interfering fusions within the measured regime.

  • Takeaways & Limitations

    The proposed law is an empirical regularity demonstrated for supervised training from scratch, while self-supervised and ImageNet-initialized cells lie outside its stated scope.

Abstract

from arXiv · show

Fusing prior knowledge with data-driven learning is attractive where data is scarce, yet no controlled account says when it helps, is redundant, or harms. We benchmark one fixed hand-crafted knowledge source, a pinned bank of Gabor targets injected only during training at $\sim$2\% overhead, against data-driven alternatives (SimCLR, SimSiam, DINO, ImageNet transfer, augmentation, learned teachers) under one frozen recipe with fixed subsets: 13 datasets, 9 backbones, 150 to 1.28M images, 32--224\,px, 2.5M--86M parameters ($\computeCells$ classification configurations over $\computeRuns$ runs, plus segmentation and detection transplants). Across the training-time combinations we measure, three outcomes recur (decision-level fusion differs). Different-\emph{currency} sources can stack: the prior composes with DeiT augmentation on attention backbones and is worth $+26$ points to ViT-B/16 at $224$\,px, $+6.7$ at twice that budget. Same-currency sources substitute: against effective self-supervised pretraining, the combination never usefully exceeds the better single source. Fusing at full strength into an already-informed initialization interferes in proportion to what it carries: ImageNet transfer, $-15$ to $-17$ points, removed by a weaker auxiliary weight. Frozen-feature diagnostics measured on each source alone separate these outcomes retrospectively but do not predict them: a rule built on them calls one of nine unseen pairs. At a practitioner's own label budget, the frozen-feature gain predicts the end-to-end gain to within $0.17$ points across 30 cells and seven datasets; the underlying decomposition, $Δ= G + \readout(\mathrm{base})$, holds in sign on $\auditRate\%$ of testable cells and is called an unseen backbone family's feature gain in advance. The project page is https://amughrabi.github.io/MomentAux.

1. Introduction

The paper frames knowledge fusion as a controlled measurement problem: sources can stack, substitute, or interfere depending on their information currency, architecture, and application strength. A frozen-feature gain explains end-to-end effects at a practitioner’s label budget, while a pre-registered taxonomy correctly calls only one of nine unseen pairs.

  • Motivation: +26 accuracy points at 224 px and +6.7 when its budget doubles show that the same Gabor prior can help, disappear, or cost 15 to 17 points.It adds +26 to ViT-B/16 at 224 px, is worth nothing after contrastive pretraining, and costs 15 to 17 points after ImageNet initialization.
  • Core claim: Two sources add only when their currencies differ, while realized gains still depend on architecture and the strength of the second source.The paper names the outcomes stack, substitute, and interfere, and treats differing currencies as necessary but not sufficient for stacking.
  • Measurement instrument: ∼2% extra training compute and nothing at inference define MomentAux’s austere cost profile against SimCLR, SimSiam, DINO, transfer, augmentation, and learned teachers.MomentAux uses fixed oriented-energy targets through a decaying auxiliary regression loss, measured under a frozen recipe and declared compute normalization.
  • Predictive diagnostics: 0.17 points across 30 cells and seven datasets is the label-budget prediction error for frozen-feature gain, whereas the pre-registered taxonomy calls one of nine unseen pairs correctly.The decomposition Δ = G + readout(base) holds in sign on the broader fixed protocol, but the taxonomy’s limited unseen-pair result leads the paper to withdraw its predictive claim.
  • Headline findings: +13.0 points for ViT-S/16 and +26.0 for ViT-B/16 on ImageNet-100 at 224 px illustrate the prior filling a feature deficit under the standard DeiT recipe.After 1.28 million images, the gain persists at +3.2 while the ResNet baseline is exactly neutral (+0.04); doubling the budget changes the gains to +4.52 ± 0.1… in the supplied passage.

2. Related work

Prior work classifies source interactions as complementary, redundant, or cooperative, while fusion research has paid less attention to when combining sources helps. This paper positions its contribution as a controlled, cost-normalized paired comparison with a predictive decomposition rather than a new interaction taxonomy.

  • Fixed filters that never enter the forward path distinguish this work from architectural proposals such as learned Gabor filters and scattering-based token mixers.
  • Fusion literature distinguishes complementary, redundant, and cooperative sources according to whether they describe different aspects, the same aspect, or jointly yield otherwise unavailable information.
  • The when of fusion has received the least attention because architecture alone cannot answer it; this study examines a hand-crafted source combined with one learned from data.
  • In representation learning, redundancy is treated as waste rather than a reliability benefit because repeating shared information does not improve performance.
  • The literature lacks paired comparisons that measure the same intervention end-to-end and on frozen features, while combinations of priors with augmentation, self-supervised weights, or pretrained initialization remain particularly understudied.
  • The registered decomposition Δ = G + readout(base) is presented as a predictive account for unmeasured quantities, rather than merely a descriptive benchmark.

3. The instrument

The instrument makes comparisons controlled through a frozen training recipe, fixed data subsets, a pinned Gabor-target prior, and cost-normalized interventions. It also predeclares measurement rules and documents limitations from compute coupling, shared cells, and seed counts.

  • Experimental controls: A frozen stochastic-gradient recipe and fixed, byte-identical stratified subsets make paired cells comparable and attribute accuracy differences to interventions rather than sampling.Architectural or scale-specific deviations are diagnostic and excluded from headline tables; paired differences remain valid within those cells.
  • Experimental controls: The fixed epoch schedule jointly varies data and compute, so data-axis results can reflect whether baselines have finished training.The study reports reversals and budget-sensitive effects linked to this coupling, including the envelope’s right flank changing at matched steps.
  • Knowledge source: +2.71 at the same cell comes from phase-invariant Gabor energy, whereas oriented-edge targets reach −0.09.The target uses the magnitude of complex Gabor responses, stacked over eight filters and pooled to the tapped feature stage.
  • Fusion mechanism: λ(T) = 0 exactly removes prior gradients during final epochs, and the auxiliary weights are discarded after training.The passage identifies both properties as load-bearing for the paper’s claims and notes that a matched optimizer budget does not produce the envelope’s right flank.
  • Measurement and limitations: The audit uses a predeclared two-SEM threshold, cluster-level reporting, and ten-seed headline envelopes because three-seed deltas are liberal, cells are dependent, and deltas shrink with more seeds.Deepening three CIFAR-100 cells to ten seeds shrank every Δ by 0.07, 0.15, and 0.39 points without changing envelope shape.

4. The grid

The benchmark defines results as seed-aggregated cells spanning 13 datasets, six visual domains, multiple backbones and interventions under a frozen training recipe. Across this grid, the prior’s gains are data-dependent, comparator-dependent, and largest on attention backbones, especially with DeiT augmentation.

  • Grid definition: Each cell combines a dataset, data fraction, backbone, and fixed-hyperparameter intervention, with quoted results reported as seed means and standard errors.Configurations are grouped by configuration name rather than parsed fields.
  • Datasets: The grid spans 13 datasets across six visual domains chosen to vary image statistics, structure, texture, staining, and fine-grained part discrimination.The domains include satellite imagery, texture, food, histopathology, and fine-grained birds alongside photographic data.
  • Training recipe: The frozen recipe uses SGD with momentum 0.9, learning rate 0.1, weight decay 5×10−4, cosine scheduling, 200 epochs, batch size 128, and random-crop plus horizontal-flip augmentation.Training is pinned to specified PyTorch, timm, CUDA, mixed-precision, and metric-checking implementations.
  • Prior envelope: The prior’s gain is unimodal across data: it rises from extreme scarcity, peaks in a dataset-specific band, and decays toward zero as data becomes sufficient.Easy ten-way CIFAR-10 plateaus across 1–2%, with the two cells differing by 0.29, and is already negative at 15%.
  • Comparator envelope: +0.85 to +5.0 points is SimCLR’s advantage over the 1.02× prior on CIFAR-100 from 1% to 10% at 2× compute, converging to parity by 50%.SimSiam gains ≤+ 0.9 anywhere on this envelope at that budget, indicating a budget-dependent limitation.
  • Attention backbones: +25 points is the prior’s gain for ViT-tiny under the standard DeiT recipe, while the best full-CIFAR-100 result reaches 75.3% versus 61.4% for the recipe alone.From scratch, ViT-tiny gains +9.9 points from the prior even at full CIFAR-100.

5. The organizing law

At matched label budgets, frozen-feature gain closely tracks end-to-end gain, supporting a decomposition into representation gain and a baseline-dependent readout term. The empirical law predicts signs broadly but is strongest below the crossing, with limited coverage and weaker discrimination above it.

  • Budget-matched organizing law: 0.17 points average error separates frozen-feature and end-to-end gains across 30 cells and seven datasets at matched label budgets.The median error is 0.11 and the worst case is 0.77, versus 0.89 under the full-label protocol.
  • Budget-matched organizing law: The end-to-end gain Δ decomposes into a feature term G and a readout term governed by baseline accuracy.Linear evaluation is treated as a representation-side diagnostic rather than a replacement for evaluating an already-trained classifier.
  • Empirical scope: R2 = 0.27 for the readout term when binned on baseline accuracy, while dataset identity explains 0.13 of residual variation and backbone explains 0.003.The residual standard deviation at fixed baseline is 2.0 points, making baseline accuracy the only consistently signed determinant across datasets and backbones.
  • Empirical scope: 86.4% of 455 resolvable cells fall on the predicted side, at 47.5% coverage, with Wilson 95% CI [82.9, 89.2].The audit spans seven backbones and nineteen of the twenty dataset identities in scope.
  • Limitations and disaggregation: 94.9% of below-crossing cells have the predicted sign, compared with 67.4% above the crossing, where the predicted readout is small.PathMNIST reaches 52.6% out of sample because its linear evaluation is treated as an unreliable compressed measuring stick.
  • What G is a function of: G is a property of the checkpoint and evaluation label space, falling from 4.2 to 2.9 when supervision shifts from 200-way fine to 20-way coarse labels on identical pixels.Self-supervised cells carry G = +7.68 against +4.23 for the prior, while oriented-energy alignment is −0.16 relative to their own baselines.

6. Fusion outcomes: stack, substitute, interfere

Fusion outcomes follow the information currency already supplied by each source: different currencies can stack, overlapping sources substitute, and full-strength shaping can interfere with mature representations. These patterns depend on partner strength and architecture, while frozen-feature overlap explains outcomes retrospectively but does not reliably predict unseen combinations.

  • Taxonomy: Feature-level currency measurements separate stack, substitute, and interfere retrospectively but did not predict unseen combinations.The prior and SimCLR show higher representation agreement than the prior and augmentation, with centered kernel alignment 0.85 versus 0.53, yet the comparative measurements failed as an unseen-pair predictor.
  • Stack: Different-currency sources can stack when architecture and intervention strength cooperate.DeiT-strength augmentation supplies nuisance-transform invariance while the prior supplies oriented-energy structure; the prior’s gain grows 1.5–2.4× on CIFAR-100 and 1.4–2.4× on Tiny-ImageNet.
  • Substitute: Overlapping sources substitute rather than stack when the partner already learned the same features.Prior plus SimCLR falls below the better single source in all six cells, with −0.14 to −2.41, whereas ineffective SimSiam leaves the prior’s gain recoverable.
  • Substitute: Partial or mismatched sources can still exceed the better single arm modestly, so substitution is not universal.The prior with DINO exceeds the better arm by +0.92 and +1.02 on ViT-tiny, while three prospective pairs exceed it by +1.04 to +1.43 points.
  • Interfere: Full-strength prior fusion interferes with mature ImageNet representations, imposing a −15.2 to −16.8 interference tax.Across 92 qualifying transfer cells, every Δ was negative, with median −7.0 and worst −26.3; the tax essentially vanishes at λ0 ≤0.3 in tested cells.

7. Validation at scale

At ImageNet scale, convolutional redundancy remains neutral while attention gains persist and grow with model size. Across data fractions, envelope shape depends on the backbone’s position on the readout curve, and one preregistered fraction-space prediction fails.

  • Scale validation: +0.04 ± 0.08 end-to-end and +0.05 ± 0.10 feature-side: ResNet-18 at 1.28M images remains neutral, establishing scale-stable convolutional redundancy.ResNet-50 at native 224 px likewise measures −0.02±0.26 across three matched seeds.
  • Scale validation: +26.0 points: ViT-B/16 gains twice ViT-S/16’s +13.0 at native 224 px and 100 epochs, while ViT-tiny still gains +3.2 at 1.28M images.The attention deficit therefore persists under full-data scaling and grows with model size.
  • Data-fraction envelope: +1.90 at 1% to zero by 10–15%: ResNet-18 has no predicted interior peak, so the preregistered fraction-space falsifier fired.One percent of ImageNet64 already contains 12,820 images, placing the peak below the smallest measurable fraction.
  • Data-fraction envelope: +2.14 to +7.75 at 25%: ViT-tiny rises monotonically, while ResNet-18 falls to zero and MobileNetV3 also climbs across the measured range.With dataset, recipe, subsets, and label space fixed, the differing envelope shapes are attributable to the backbone.
  • Data-fraction envelope: +0.5 to +2.3 from 1 to 25%: ResNet-50 at native 224 px remains positive in the low-data band before returning to neutrality at full data.The registered magnitude pattern missed, but the direction held: convolutional gains survive native resolution before sufficiency.

8. Buying the comparator more compute

Buying comparators more compute changes the conclusions: the prior’s convolutional advantage becomes a cost claim, while on attention its outcome depends on comparator budget and data fraction. Frozen-feature gains retrospectively track end-to-end ordering across the tested self-supervised comparisons.

  • Budget dependence: 5× SimSiam gains up to +4.33, reversing its published-budget range of −0.11 to +0.87; its budget response is data-dependent, with +3.7 at 10% and nothing at 1%.The prior wins from 1 to 7% by up to 3.25 but loses from 10 to 25%, including −1.10 at 15%.
  • Convolutional backbones: The prior’s convolutional case is a cost case, not an accuracy case, after 5× SimCLR beats it by 3.12 to 9.23 points from 1–15% data.At 25%, SimCLR’s advantage narrows to 2.61 points.
  • Attention backbones: At 5× comparator compute, the prior leads ViT-tiny only in the higher-data band: it loses by 1.57 at 5%, is level at 10%, and wins by 1.47 at 25%.At 2×, the prior leads at every CIFAR-100 fraction.
  • Limitations: At Tiny-ImageNet 25%, increasing pre-training from 2× to 5× worsens comparator accuracy from 38.48 to 36.53, a drop of 1.95 ± 0.42.This is the only cell in the study where extra pre-training budget costs accuracy.
  • Frozen-feature mechanism: Across both datasets and six measured fractions, the intervention with the larger frozen-feature gain also wins end-to-end, matching the sign of G_SSL − G_prior six times out of six.For SimSiam, frozen-feature ordering matches end-to-end ordering at seven of eight cells; the mismatch is the near-zero crossing cell at −0.08.

9. Fusing genuinely different sources

Fusing genuinely complementary bands stacks, whereas combining cross-modal satellite inputs does not reliably improve over the stronger source. Decision-level fusion can still help despite training-time substitution, while feature diagnostics explain gains retrospectively and a prospective rule fails on unseen pairs.

  • Sensor fusion: +0.45 to +2.96 fusion gains on EuroSAT-MS contrast with −0.73 to ±0.00 on So2Sat, where radar adds nothing to optical despite 35–53% standalone accuracy.EuroSAT’s band sets are complementary within one instrument; So2Sat’s radar and optical sources differ in modality and physics.
  • Limits of explanation: 0.11–6.68 points of source asymmetry on EuroSAT-MS versus 25.98–33.77 on So2Sat cannot distinguish unequal quality from cross-modality.The experiment supports a contrast, not a predictor, because the two populations are confounded and no third population was run.
  • Prior transplant: +1.13, +1.91, +1.19, +0.68 and −0.56 are the prior’s optical gains at 1, 2, 5, 10 and 25%, while SAR gains are +1.24, −3.09, +2.59, −0.46 and −0.24.SAR interpretation is ambiguous because the channel reduction averages backscatter without photometric meaning, and the 2% cell has unstable seeds.
  • Feature and readout effects: +1.31, +1.54 and +0.88 feature gains on So2Sat’s radar, optical and fused arms coexist with end-to-end gains of +2.59, +1.19 and +1.35.The corresponding EuroSAT feature gains are +0.98, +1.29 and +1.16 against end-to-end gains of +1.63, +2.60 and +3.30, indicating readout contributions.
  • Decision-level fusion: 28% disagreement between same-source models yields no ensemble benefit, whereas 47% disagreement with a self-supervised model yields ∼1.2 points; prior–SimCLR decision fusion gives +1.20 despite training-time substitution.Training-time redundancy and decision-level error correlation are distinct claims.
  • Prospective prediction: 1.7 points below the better arm at 7.4 standard errors is the rule’s miss on CIFAR-100 at 10%, where it incorrectly predicts STACK.The rule calls one of nine unseen pairs, and four of five resolved misses have this same-currency pattern.

10. Does any of this hold off classification?

Off classification, the prior’s gains remain modest and regime-dependent: semantic segmentation confirms its positive branch but reaches neutrality or worse with sufficient data, while detection finds no practically useful feature gain. The evidence supports task- and regime-dependent transfer rather than a universal dense-prediction advantage.

  • Semantic segmentation: +0.39 mIoU (6.2%) at VOC 1% accompanies G = +0.31, showing the dense-task result is not a readout artifact.VOC at 1% uses 107 images over 20 classes, about five per class.
  • Semantic segmentation: 1–22% relative gains across the 1–25% band match classification’s range, while three populations become neutral or negative at full data.The relative framing is required because segmentation baselines differ eightfold.
  • Semantic segmentation: 9 of 9 testable segmentation cells fall on the law’s predicted side, confirming its positive branch on a new task and metric.All nine cells are above the crossing, so the negative branch remains untested.
  • Semantic segmentation: +0.27 feature gain for Swin at 25% shows the attention advantage can collapse in dense prediction and is task- and regime-dependent.Swin leads ResNet-18 at 2% and 5% but trails at 1%, 10%, and 25%.
  • Object detection: +0.0073 ± 0.0024 G(fg_iou) at 10% is the only resolvable detection-side movement, and it is too small to support a substantive localization result.At full data, the detector reaches 48.9 AP50 from scratch, so the null is not attributed to a broken head.

11. Ablations and controls

The ablations identify moment structure, training-only injection, and stabilization of unstable backbones as distinct sources of benefit, while forward-path fusion can impose lasting accuracy costs. Results also delimit the tested bank, schedule, pooling, head, and luminance design space.

  • Target and mechanism controls: Random fixed targets move CIFAR-100 accuracy by at most 0.12 points, while a same-data FitNets teacher costs twice as much and changes accuracy −0.49 to +0.73.Phase-invariant magnitude wins among hand-crafted targets by 1.6 to 2× over the next-best target, while cosine loss forfeits most of the gain.
  • Out-of-selection controls: +0.67 ± 0.13 at Tiny-ImageNet 5% and +0.37 at 10% are achieved by the random control, whereas CIFAR-10 gives −0.70 and −1.46.All Tiny-ImageNet targets help (+0.23 to +1.78), and magnitude remains the best measured single choice, but target rankings are less discriminating off the selection set.
  • Design knobs: Tap depth is flat through the first three stages and cliffs only at the final stage, while schedule starts work best at 2.0 for 1–2%, 1.0 for 3–10%, and 0.3 for 15–25%.Three retuning attempts failed to push Δ past measured feature gain, indicating that the schedule realizes rather than manufactures G.
  • Backbone and schedule transfer: +4.3, +4.0, and +3.9 are obtained by ResNet-18, -34, and -50 at CIFAR-100 with 10% data, using one reference setting across the backbone sweep.Across six further domains, the depth sweep gives +0.4 to +9.0 at 5–10%, with CUB and DTD at 5% as named exceptions.
  • Forward-path versus auxiliary fusion: −7.4 to −11.9 at 25% data is reached by nonlinear energy forward stems, which remain −1.0 to −2.1 even at 100%, unlike training-only auxiliary injection.The mild k5 Gabor stem reaches −0.19 at 10%, while energy-magnitude reaches +2.5 and +3.5 at CIFAR-100 1–2%.
  • Stabilization and scope: Baseline seeds are bistable across several backbones, while the prior arm trains tightly every time; Swin feature variance falls from σ up to 4.3 to 0.2.The study treats stabilization as separate from routine variance reduction and does not vary several recipe, pooling, head, bank-envelope, or luminance constants.

12. A decision guide

The decision guide recommends the prior as a low-overhead complement for modern attention backbones, while indicating that SSL can lead under extreme scarcity and full-strength fusion can interfere with ImageNet-transferred features. For convolutional, photo-like settings, the prior is competitive at modest cost but does not consistently beat SSL at larger budgets.

  • Convolutional backbones: At 5× compute, SimCLR leads by +3.1 in extreme scarcity, whereas the prior costs 1.02× and is near SSL parity at 2×.For convolutional, photo-like data, the prior is recommended when a 2× budget is unavailable; extreme scarcity is defined as ≤1–2%.
  • Modern attention backbones: The prior costs 1.02× and gains +13 to +26 at 224 px under 100 epochs, or +4.5 to +6.7 at 200 epochs, while stabilizing Swin.This is the guide’s most effective modern-ViT recommendation: integrate the prior with the standard recipe rather than treating it as a competing source.
  • Modern attention backbones: At 1.28M images, the prior yields +3.2 at 1.02× compute and remains competitive from scratch.The recommendation applies to the from-scratch attention backbones evaluated in this study.
  • Transferred initialization: −15 to −17 points of feature-side interference occurs when fusing into ImageNet transfer at λ0=1.0, becoming neutral at λ0≤0.3.The guide therefore cautions against full-strength fusion for domain-shifted data such as stains.

13. Discussion

The discussion presents fusion as an operational phenomenon governed by non-overlapping feature contributions, architecture, and intervention strength. The proposed decomposition organizes trends and signs but does not predict individual cell values or fusion outcomes from source-alone measurements.

  • What the decomposition buys, and what it does not: The decomposition accounts for trends and signs rather than predicting a cell’s value.It offers one low-dimensional structure for organizing much of the benchmark grid.
  • What the decomposition buys, and what it does not: Five labels per class can prevent improved features from producing gains because the available data cannot express a better representation.This explains why gains collapse at extreme scarcity even when features improve.
  • What the decomposition buys, and what it does not: Fusion adds value only when sources contribute non-overlapping information in feature space, moderated by architecture and intervention strength.Each source’s contribution can be measured through an inexpensive linear evaluation of that source alone.
  • What the decomposition buys, and what it does not: Comparing source-alone linear evaluations does not yet predict which fusion outcome will follow.The measurements distinguish each source’s contribution, but not the resulting outcome.

14. Limitations and future directions

The proposed law is an empirical, scarce-data account with a bounded domain and substantial unresolved regions. Its main patterned failure occurs at sufficiency, where features improve but accuracy falls on fine-grained or texture-dominated data, motivating late-training analysis.

  • Domain and validity: The law is empirical and demonstrated across 13 datasets, six visual domains, 2.5M–86M-parameter backbones, 32–224 px images, and training sets from 150 onward.It is informative on the scarce-data flank, while semantic segmentation confirms only the positive branch on a different task and metric.
  • Domain and validity: 95% below the crossing versus 67% above it: the predicted sign is right more often below the crossing, while 409 of 958 in-scope cells are unresolvable.Another 94 cells fall inside the bracket where no sign is predicted.
  • Patterned failure: Seven resolvable cells violate the decomposition above the crossing: frozen features improve measurably while end-to-end accuracy falls.Five are Food-101 and PathMNIST cells at ≥20% of their data with the largest magnitudes; two smaller cells are near the threshold.
  • Future directions: At sufficiency, fine-grained and texture-dominated data expose a missing conversion term: the prior improves representation quality, but the network fails to convert it into accuracy.The overshoot account predicts the sign of Δ but not the simultaneous rise in G; late-training dynamics are the clearest next target for instrumentation.

15. Conclusion

The conclusion argues that fusion outcomes depend on the scarce currency represented by each source rather than its modernity. The attention-backbone prior remains the strongest survivor, delivering substantial gains at 224 px that persist, though shrink, with more budget and data.

  • Conclusion: Sources trade in currencies: same-currency pairs substitute, while each source’s value is measurable from its own frozen features before combination training.The conclusion frames the observed outcomes as depending on which currency is scarce, not on how modern a source is.
  • Attention result: +4.5 and +6.7 are the prior’s gains for ViT-S/16 and ViT-B/16 when the 224 px budget is doubled.Doubling the budget closes part of the gap, while costing more than the prior does.
  • Attention result: +3.2 points remains after 1.28 million images, while the convolutional baseline is neutral to within 0.04.The prior benefits the larger attention model more at both budgets.

CRediT authorship contribution statement · A. The self-supervised margin

The authorship statement assigns broad methodological and project responsibilities across the team. The self-supervised margin over the prior is positive at low-to-moderate label fractions, peaks around 15%, and diminishes at higher fractions.

  • CRediT authorship contribution statement: Ahmad AlMughrabi contributed conceptualization, methodology, software, validation, formal analysis, investigation, data curation, writing, visualization, and project administration.
  • CRediT authorship contribution statement: Albert Clop contributed methodology and writing, review & editing.
  • CRediT authorship contribution statement: Benjamin Busam contributed writing, review & editing, while Ricardo Marques contributed supervision and writing, review & editing.
  • CRediT authorship contribution statement: Petia Radeva contributed supervision, conceptualization, resources, project administration, funding acquisition, and writing, review & editing.
  • A. The self-supervised margin: +0.85 to +5.01 points separate a 2×-compute SimCLR initialization from the 1.02× prior across 1–15% of CIFAR-100 training data.The comparison uses ResNet-18 with plain augmentation and three seeds per cell; additional margins are +0.73 at 50% and +0.22 at 100%.
  • A. The self-supervised margin: The SimCLR-over-prior margin is unimodal, is weak below 1–2% because too few images are available to contrast, and is gone by 50%.

B. The law’s components

The law’s 𝐺 component is non-monotone and dataset-specific, with distinct data-scaling curves across CIFAR-10, CIFAR-100, Tiny-ImageNet, and STL-10. Comparisons must fix evaluation label space because label-space crossings can produce negligible or opposing effects on 𝐺.

  • Component curves: 𝐺 is non-monotone in data and dataset-specific: CIFAR-10 peaks at 5.52, CIFAR-100 at 6.26, Tiny-ImageNet declines monotonically, and STL-10 peaks at 4.97.The reported sequences are CIFAR-10: 4.81, 5.52, 3.97, 0.65; CIFAR-100: 4.16, 5.14, 6.26, 3.55; Tiny-ImageNet: 4.19, 2.66, 1.70; STL-10: 4.70, 4.97, 3.20.
  • Label-space controls: On byte-identical CIFAR-100 pixels, crossing training and evaluation label spaces changes 𝐺 by at most 1σ.This result comes from a full 2×2 comparison of training against evaluation label space.
  • Label-space controls: On Tiny-ImageNet, the same label-space crossing produces two opposing ∼5σ effects that cancel on the diagonal.Therefore, 𝐺 comparisons fix the evaluation space.

C. Per-domain and dataset notes · D. Selected controls · E. Operational reproducibility

Dataset notes show domain-specific reversals, contamination- and granularity-robust gains, and an operational pipeline designed to regenerate, audit, and verify results. These findings span EuroSAT, DTD, Food-101, PathMNIST, CIFAR-100 controls, and reproducibility procedures.

  • C. Per-domain and dataset notes: At 7 of 11 fractions, the prior beats SimCLR on EuroSAT by +0.6–+1.9 in the 1–10% band.The passage attributes this to SimCLR’s photographic view-invariances transferring poorly to satellite statistics, while the spectral target is domain-agnostic.
  • C. Per-domain and dataset notes: From a few hundred images, SimCLR dominates on DTD, whereas Food-101 favors SSL on convolutional backbones.
  • C. Per-domain and dataset notes: Above ∼10–15% added data, every method declines on PathMNIST, from 92.0 to 86.7, consistent with its center-shifted test split.The passage therefore limits interpretation of PathMNIST to its low-data regime.
  • D. Selected controls: On ciFAIR-100, replacing 927 near-duplicates changes the prior’s gain by at most 0.20 points at any fraction.Both arms experience identical contamination, so it cancels in Δ.
  • D. Selected controls: Relabeling CIFAR-100 with 20 coarse labels reproduces the full gain at matched data: +5.84 versus +5.30 at 5%.Pixels and training steps remain byte-identical, supporting gain dependence on data and task performance rather than label granularity.
  • E. Operational reproducibility: One entry point trains any cell from its configuration name and seed, while one exporter per task family regenerates result tables from run records.
  • E. Operational reproducibility: A released script reruns the sign-law audit of Table 8, and runs ship with subset indices, filter-bank fingerprints, and per-run environment records.
  • E. Operational reproducibility: Training refuses to overwrite completed cells, locks each run, writes checkpoints atomically, and verifies mirrors by evaluating checkpoints again.
Loading 2608.21098v1…