PaperlayerView exact revision on arXiv

Source-linked AI summary

Samples, Sources, Space: Decomposing Data Scale in Spatially Structured Representation Learning of Human Brain Microarchitecture

Christian Schiffer, Mathis Bode, Thomas Lippert, Katrin Amunts, Timo Dickscheid

arXiv:2609.31201v1cs.AI

TL;DR

Sample count alone does not describe data scale when observations are organized across sources and space. This study varies sample count, source diversity, and spatial coverage in whole-brain histology across controlled pretraining runs. Representations generalize better to subjects seen during pretraining, but distributing a fixed sample budget across more subjects yields no detectable improvement.

  • Problem

    Scaling studies typically represent data by sample count, although the same count can differ in source distribution and spatial coverage.

    Source: §1 · p. 1

  • Method

    The study varies unique sample count, source count, spatial coverage, compute, and model capacity across controlled SpatialNCE pretraining runs on whole-brain histology patches.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Representations generalize substantially better to pretraining subjects, while redistributing a fixed sample count across more subjects produces no detectable improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Inter-source variation and the marginal value of acquiring additional sources are distinct; allocation depends on the value of samples from new versus already represented sources.

    Source: §5 · p. 9

    The conclusion is limited to B = 1–18, approximately 450,000 unique locations, SpatialNCE pretraining, and cytoarchitectonic targets.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Scaling studies typically represent training data by a single count of samples. For hierarchically and spatially structured data, however, the same number of samples can be drawn from few or many sources and distributed differently across the underlying domain. We therefore study data scaling as an allocation problem, separating unique sample count, source diversity, and spatial coverage. We study this decomposition in microscopic whole-brain histology, where a source is an individual brain, and a sample is an image patch at a specific spatial location. Across 93 controlled pretraining runs of a contrastive model that uses spatial proximity for supervision, we vary data allocation, compute, and model capacity over 11.6 million spatially anchored image patches from 21 human brains. Performance improves with more unique samples, broader spatial coverage, additional compute, and larger model capacity. At fixed sample count, distributing samples across one to 18 subjects produces no detectable improvement, even though representations generalize substantially better to subjects encountered during pretraining. Inter-subject variation therefore strongly affects generalization, but additional subjects provide no benefit when a fixed sample budget is distributed across more sources. These results establish sample count, source diversity, and spatial coverage as distinct axes of data scaling in spatially structured representation learning.

1 INTRODUCTION

The introduction frames data scaling as an allocation problem: sample count alone does not capture source diversity or spatial coverage. In whole-brain histology, the authors vary these dimensions under controlled conditions and find that pretraining exposure improves generalization, while redistributing a fixed sample budget across more subjects does not.

1 INTRODUCTION

  • Hierarchical and spatial structure means the same sample count can represent different source distributions and different coverage of the underlying domain.Source diversity, sampling density, and domain coverage are distinct dataset properties.Source: §1 · p. 1
  • The study treats data composition as an allocation problem across distinct samples, contributing sources, and domain coverage.A unique sample is one distinct sampled position, counted once regardless of how often it is presented to the optimizer.Source: §1 · p. 1
  • Across 93 controlled pretraining runs, the authors study 11.6 million locations from 21 human brains using spatially anchored patches and linear probing for cytoarchitectonic classification.The study varies unique sample count, source count, spatial coverage, compute, and model capacity.Source: §1 · p. 2
  • Representations generalize substantially better to pretraining subjects, but redistributing a fixed sample count across more subjects yields no detectable improvement.The value of additional sources depends on their trade-off with within-source sampling density.Source: §1 · p. 2
  • The experimental decomposition separately measures contributions from samples, sources, and coverage under matched compute and model capacity.The authors frame this as a way to identify which form of additional data is most valuable.Source: §1 · p. 2

2 RELATED WORK

Prior work studies scaling by example or token count and recognizes that data composition and sampling structure matter. This paper distinguishes its focus as allocating a fixed budget of unique samples across sources and spatial coverage, including in spatially supervised representation learning.

2 RELATED WORK

  • Scaling laws typically use example or token count, while data-curation work studies composition within a fixed count without decomposing source diversity, within-source density, and spatial coverage.Muennighoff et al. (2023) study uniqueness versus repetition; this work asks how to allocate unique samples across sources and space.Source: §2 · p. 2
  • Hierarchical datasets create a trade-off between denser sampling within sources and broader sampling across sources, but cross-model pathology comparisons cannot isolate these factors from architecture, objectives, datasets, and compute.Related ecology and geospatial work also addresses allocation across locations or environmental gradients.Source: §2 · p. 2
  • The experiments vary sample allocation across sources and coverage, then add controls for compute, model capacity, cross-subject supervision, and held-out subjects.In histology, sources are subjects and samples are spatially anchored image patches.Source: Figure 1 · p. 3
  • Prior work on sampling structure does not establish how to allocate a fixed pretraining budget across samples, sources, and spatial coverage.This is the allocation question the study sets out to address.Source: §2 · p. 3
  • SpatialNCE weights pairwise relationships continuously by spatial distance, making training-set spatial composition relevant to the supervisory relationships available to the objective.Related spatial-proximity approaches include Tile2Vec, SGCL, and histological methods.Source: §2 · p. 3

3 STUDY DESIGN

The study uses whole-brain histology to test how data composition, model capacity, and compute relate to representation quality. Its design defines the data and evaluation units, specifies spatially weighted contrastive pretraining, varies experimental factors, and uses mixed-effects analysis with prespecified detection thresholds.

3.1 DATA, TASK, AND GENERALIZATION LEVELS

  • The dataset contains 11,644,201 spatial locations from 21 postmortem brains, and frozen representations are evaluated by linear probing on 191 cytoarchitectonic areas.Three subjects are excluded from pretraining and probe training; evaluation distinguishes unseen, transfer, and held-out-section subjects.Source: §3.1 · pp. 3–4

3.2 OBJECTIVE AND MODELS

  • SpatialNCE uses patch content and reference coordinates to encourage nearby anatomical patches to have more similar representations, while coordinates are withheld from the encoder.The objective weights pairwise relationships continuously by spatial distance, and data composition changes the positive-weight balance.Source: §3.2 · p. 4

3.3 EXPERIMENTS

  • Experiments vary unique samples, subject count, and spatial coverage, with complementary controls for compute, capacity, subject gating, and held-out-subject assignment.At fixed sample count, increasing source count reduces the mean number of locations per source; the coverage contrast matches sample count and positive weight.Source: §3.3 · pp. 4–5, Table 1 · p. 5

3.4 STATISTICAL ANALYSIS

  • A linear mixed-effects model analyzes logit-transformed macro-F1, with logarithmic terms for unique samples, sources, compute, and positive weight, plus categorical capacity and coverage effects.The analysis reports 95% confidence intervals and defines minimum detectable effects from fitted variance components at 80% power.Source: §3.4 · pp. 5–6

4 RESULTS

Seen-subject exposure was associated with better probe performance, but distributing a fixed sample budget across more subjects produced no detectable gain. More unique samples, compute, and capacity improved representation quality, while narrower coverage reduced it in some tested conditions.

4.1 SOURCE INCLUSION IN PRETRAINING DOMINATES PERFORMANCE

  • Seen-subject exposure raised macro-F1 by 0.124 on average across 14 held-out-section subjects and 0.148 across three transfer subjects; the familiarity advantage was associated more strongly with Nunique/B than with B.All 17 subject-level contrasts were positive and exceeded the applicable MDE.Source: §4.1 · pp. 6–7

4.2 MORE SOURCES DO NOT HELP AT A FIXED SAMPLE BUDGET

  • At fixed Nunique ≈450,000, distributing samples across B=1–18 produced no detectable gain across capacities; observed differences stayed below the applicable MDE.The B=1 versus B=18 macro-F1 differences were 0.015 for ViT-B, −0.002 for ViT-L, and 0.017 for ViT-H.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 MORE SAMPLES, COMPUTE, AND CAPACITY IMPROVE REPRESENTATIONS

  • Doubling unique locations raised logit macro-F1 by βN = 0.0916 [0.0695, 0.1135], and doubling steps by βC = 0.1067 [0.0847, 0.1286]; pooled capacity terms favored ViT-L and ViT-H.The capacity coefficients relative to ViT-B were γP = 0.081 [0.031, 0.130] for ViT-L and 0.127 [0.094, 0.159] for ViT-H.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 BROAD SPATIAL COVERAGE IMPROVES REPRESENTATIONS

  • At B=13, four-block sampling lowered macro-F1 by 0.038 versus every-10th-section sampling, while the 16-block contrast was unresolved; no coverage condition differed detectably at B=1 or 4.The four-block condition spanned 78% of the full section range yet showed the largest deficit, indicating that the arrangement of covered and uncovered regions also matters.Source: §4.4 · pp. 8–9
  • At B=18, gating cross-subject pairs reduced macro-F1 from 0.300 to 0.243 as those pairs supplied 94.2% of positive weight; most coverage conditions also changed positive weight M.The gating contrast was a decrease of 0.057, exceeding the within-subset MDE.Source: §4.4 · p. 9, Figure 5 · p. 9

5 DISCUSSION

The discussion separates source-level variation from the value of acquiring additional sources, and argues that data composition should be treated as a distinct dimension of scaling. It also reports the study’s ethics and reproducibility procedures.

5 DISCUSSION

  • Inter-source variation affects generalization, but broader source sampling does not improve representations when it redistributes a fixed sample budget.The relevant allocation question is the marginal value of samples from new versus already represented sources.Source: §5 · p. 9
  • A sample count alone does not specify whether dataset growth adds observations, sources, or coverage, so data composition is an explicit dimension of empirical scaling.The authors establish this distinction in spatially supervised whole-brain histology and motivate its use for spatially structured datasets.Source: §5 · pp. 9–10

ETHICS STATEMENT

  • The study’s use is covered by the Heinrich Heine University Medical Faculty ethics committee vote #4863, and all body donors signed an agreement.Postmortem brains came through donor programs at the universities of Düsseldorf, Rostock, and Aachen, under applicable regulations and guidelines.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The analysis plan was prespecified before the first pretraining run, and code, configurations, and derived data were version-controlled with DataLad.Contrasts outside the plan are identified as exploratory; the plan fixed replication, statistical specification, interval procedure, and minimum detectable effect.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The authors describe their use and review of generative AI, the human-brain tissue dataset, and the generation and mapping of sampling locations. The encoder dimensions vary across standard ViT variants while token grid and register count remain fixed.

AI USE STATEMENT

  • AI tools assisted with research design, analysis discussion, framing, manuscript work, and analysis and figure code; authors reviewed outputs and verified reported numerical results.The authors state that pretraining, evaluation, and statistical fits were produced by their own computational pipelines.Source: AI USE STATEMENT · p. 10

A.1 SUBJECTS, TISSUE PROCESSING, AND CONSENT

  • The dataset contains 21 postmortem human brains and 11,644,201 sampling locations, from tissue sectioned at 20 µm and digitized at 1 µm per pixel.Sixteen brains were cut coronally and five horizontally; E4 uses the 13 coronal subjects in the pretraining pool.Source: §A.1 · p. 14

A.2 SAMPLING-LOCATION GENERATION

  • Sampling locations were selected independently within each section using distance transforms, then mapped to Colin27 reference space to define cross-subject Euclidean distances.The procedure yielded 11,644,201 locations, approximately 554,000 per subject, without requiring surface reconstruction or section-to-section alignment.Source: §A.2 · p. 14
  • Across encoder capacities, depth, width, and head count follow standard ViT variants, while token grid and register count remain identical.Source: Table 3 · p. 15

A.3 DOWNSTREAM TASK AND EVALUATION SEMANTICS

Evaluation uses a frozen encoder and a linear probe to classify cytoarchitectonic areas, with distinct subject sets for primary unseen evaluation and transfer. Macro-F1 is calculated per subject and averaged across evaluation subjects.

A.3 DOWNSTREAM TASK AND EVALUATION SEMANTICS

  • The downstream probe predicts one of 191 cytoarchitectonic areas using a single affine layer trained on frozen encoder representations.The class vocabulary is fixed before evaluation and derived from the Julich Brain atlas.Source: §A.3 · p. 15
  • Three subjects are excluded from all controlled pretraining and probe training for primary unseen evaluation, while three additional probe-excluded subjects define transfer evaluation.The remaining annotated subjects contribute to probe training, with separate sections held out for evaluation.Source: §A.3 · p. 15
  • Macro-F1 is computed for each evaluation subject over the classes present there, then averaged across subjects.The three primary unseen subjects contain 137, 171, and 172 of the 191 classes, respectively.Source: §A.3 · p. 15

B.1 ARCHITECTURE AND OPTIMIZATION

The methods specify a ViT encoder and SpatialNCE training setup, with common batch and optimization controls across capacities. They also report augmentation, total compute, and the shared software and execution environment.

B.1 ARCHITECTURE AND OPTIMIZATION

  • Each 2048×2048 input is tokenized into 4096 patch tokens, and the class token supplies the encoder representation.The sequence also includes a class token and four register tokens; the architecture uses modernized ViT-5 blocks.Source: §B.1 · p. 15
  • Pretraining uses AdamW at a constant 10−4 learning rate, fixed global batch size 2048, and no warmup or decay; representations are gathered across accelerators for SpatialNCE.A two-layer projection head maps class-token representations to normalized 256-dimensional vectors and is discarded before downstream evaluation.Source: §B.1 · p. 15
  • Patches are rotated, mirrored, and translated, while intensity transformations and probabilistic blur or sharpening provide additional augmentation.Rotation is sampled across −π to +π; mirroring and each blur or sharpening operation have probability 0.5 and 0.25, respectively.Source: §B.1 · pp. 15–16

B.2 COMPUTE EFFORT

  • Pretraining consumed 315,973 accelerator-hours over 7,320,000 optimization steps and 94,200 exaFLOPs.The reported 348,724 accelerator-hours total includes an additional 32,751 hours for 124 linear probes.Source: §B.2 · p. 16

B.3 SOFTWARE AND EXECUTION ENVIRONMENT

  • All jobs ran on the JUPITER supercomputer in a common Apptainer image, using PyTorch and distributed data parallel training.Pretraining used 64, 128, or 256 accelerators for ViT-B, ViT-L, or ViT-H, respectively; the global batch size remained 2048.Source: §B.3 · p. 16

C.1 DERIVATION OF THE TWO DETECTION THRESHOLDS

This appendix defines contrast-specific detection thresholds, checks collinearity in the full-grid model, and assesses the sensitivity of E1 to a capacity-by-subject-count interaction.

C.1 DERIVATION OF THE TWO DETECTION THRESHOLDS

  • Paired evaluation-subject contrasts cancel the evaluation-subject effect, while the pretraining-subset effect also cancels for within-subset contrasts.Across-subset contrasts treat the subset effects as independent; residual contributions decrease when averaged across evaluation subjects.Source: §C.1 · p. 16
  • 0.226 logit units across pretraining-subject subsets and 0.166 within subsets are the respective MDEs for pairwise configuration contrasts.At a baseline macro-F1 of 0.21, these correspond to approximately 0.038 and 0.028 macro-F1.Source: §C.1 · p. 17

C.2 COLLINEARITY AMONG THE FIXED EFFECTS

  • 1.92 is the largest VIF, for log2 Nunique, indicating limited linear dependence among the encoded fixed effects.VIFs quantify design-matrix collinearity but do not test whether the common additive model is adequate across experiments.Source: §C.2 · p. 17, Table 4 · p. 17

C.3 SENSITIVITY OF THE CAPACITY INTERACTION

  • 0.05 logit units per subject-count doubling is the approximate E1 interaction size detectable with 80% power at the fitted residual variance.Smaller interactions are not reliably detectable under E1.Source: §C.3 · p. 18, Figure 6 · p. 18

C.4 PRE-SPECIFICATION AND MULTIPLICITY

The appendix records which analyses were prespecified and details how coverage, SpatialNCE pair composition, spatial familiarity, and training behavior vary across configurations.

C.4 PRE-SPECIFICATION AND MULTIPLICITY

  • The 16-block versus every-10th-section E4 contrast was primary; localized-coverage and subject-gating contrasts were exploratory and reported without multiplicity correction.No primary conclusion depends solely on whether an exploratory confidence interval excludes zero.Source: §C.4 · p. 18

D.1 COVERAGE GEOMETRY AND SUBJECT GATING

  • 31 central sections provide approximately as many sampling locations as 97 posterior or 88 anterior sections, while localized-center coverage has the smallest anterior–posterior extent.Its positive weight M is not the largest because M depends on the full three-dimensional distribution of pairwise distances.Source: §D.1 · p. 18

D.2 COMPOSITION OF THE SPATIALNCE OBJECTIVE

  • 94.2% of E1 positive weight came from cross-subject pairs at B = 18, while total positive weight M remained approximately constant as B increased from 1 to 18.At fixed Nunique, subject count changes the composition of contributing pairs without changing overall positive weight.Source: Figure 7 · p. 19
  • E7 gating reduced M to 8.34 at B = 2 and 0.92 at B = 18, while spatially concentrated E4 conditions reached M values of 53–115.Across ungated configurations outside E4, M ranged only from 14.5 to 17.6 despite a 224-fold range in Nunique.Source: §D.2 · p. 19

D.3 SPATIAL LOCALIZATION OF THE FAMILIARITY BENEFIT

  • Macro-F1 decreased with distance to the nearest pretraining section in five of six E4 coverage conditions, though spatial difficulty and class composition confound this pattern.In the anterior-localized condition, performance increased with distance, potentially reflecting higher performance in more posterior areas.Source: §D.3 · pp. 19–20, Figure 8 · p. 19

D.4 PRETRAINING BEHAVIOR

  • 146 400 optimization steps did not produce a SpatialNCE loss plateau, and higher-capacity models reached lower loss over the same budget.The trajectories include standard and doubled-compute ViT-B, capacity ViT-L/H, and subject-gated ViT-B configurations.Source: §D.4 · pp. 20–21, Figure 9 · p. 21

D.5 ALTERNATIVE HELD-OUT-SUBJECT ASSIGNMENTS

Alternative unseen-subject assignments leave the subject-count effect unresolved, and reference encoders provide an absolute-performance comparison under the study's frozen-feature protocol.

D.5 ALTERNATIVE HELD-OUT-SUBJECT ASSIGNMENTS

  • βB = 0.0131 [−0.0164, 0.0429] logit macro-F1 per doubling of B left the subject-count effect unresolved on an alternative unseen-subject assignment.The figure reports that absolute performance varied across assignments while the direction and approximate magnitude of the E1–E2 difference were preserved.Source: Figure 10 · p. 21, §D.5 · p. 22

D.6 REFERENCE ENCODERS

  • Virchow2 achieved the highest macro-F1 among reference encoders at 0.251, compared with 0.300, 0.311, and 0.316 for ViT-B/32, ViT-L/32, and ViT-H/32.Architecture, pretraining, physical resolution, and staining differ, so this is an absolute-performance reference only.Source: §D.6 · p. 22, Table 7 · p. 22

E REMAINING LIMITATIONS

The study's conclusions are bounded by its finite subject pool, evaluation and training design, and frozen linear-probe setting; the appendix inventories the pretraining runs and retained configurations.

E REMAINING LIMITATIONS

  • E1 tests fixed-sample subject scaling only at approximately 450 000 unique locations and B = 1–18, so effects at smaller budgets or sparser sampling remain open.The primary evaluation uses three unseen subjects, and each configuration has a single optimization seed, leaving optimization-seed variability unquantified.Source: E REMAINING LIMITATIONS · p. 22

F COMPLETE RUN INVENTORY

  • The study conducted 93 pretraining runs; the full-grid fit used 96 configurations and 288 subject-level scores after excluding four alternative-assignment runs and adding seven retained half-budget checkpoints.The fixed-sample E1 fit contained 18 configurations and 54 subject-level scores.Source: F COMPLETE RUN INVENTORY · p. 23, Table 8 · p. 23

Related papers

Loading 2609.31201v1…