PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Aware Data Selection from In-and-Out Information Interplay

Yifan Wang, Xiaomin Li, Yuexing Hao, Dongwon Jung, Hemanth Neelgund Ramesh, Ananth Grama, Varun Chandrasekaran, Yu Hu, Andrzej Banburski-Fahey, Jaron Lanier

arXiv:2609.33010v1cs.AI

TL;DR

Post-training data selection is difficult because an example’s value depends on what the model has already absorbed, while common signals may not distinguish useful unassimilated knowledge from difficulty or junk. CAP compares model-generated and reference responses across layers to target that gap. Across math, code, and science, it reports larger average improvement than the strongest baseline across selection budgets.

  • Problem

    Data selection signals often ignore what the current model has absorbed or reduce its response to a scalar that cannot distinguish unassimilated knowledge from intrinsic difficulty or junk.

    Source: §1 · p. 2

  • Method

    CAP compares layer-wise divergences for a reference response and the model’s own generation, using shallow divergence to residualize deep divergence and isolate unassimilated information.

    Source: §1 · p. 2

  • Results

    Across math, code, and science and all four selection budgets, CAP’s improvement over the base model is 35.4% larger than LESS’s, and CAP wins or ties in 8 of 12 settings.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The results suggest that effective post-training data should target what the model has not yet assimilated rather than what is merely difficult.

    Source: §6 · p. 10

    The theory depends on addressing and bandwidth assumptions, and does not determine the empirical depth split, response horizon, or downstream effect size.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

LLMs are effective representations that assimilate vast amounts of knowledge during pretraining, but post-training is necessary for models to reliably access this knowledge and "know what they know." We observe an interesting rank equilibrium between knowledge stored in the weights and the data stream passing through the model. Across all model layers, we find that the hidden states (data stream) follow a U-shaped pattern, showing substantial compression in early layers and a steep rise during the late-layer decoding phase. In contrast, the weight rank follows an inverted U-shaped pattern, with very low rank in the early and late layers and high rank in the middle. We interpret this as an in-and-out information interplay: intermediate activations do not need to carry content that the weights can supply later, so they primarily preserve what the weights cannot provide. Motivated by this observation, we propose a model-aware data selection method, CAP (Counterfactual Assimilation Profile), which can determine whether a data candidate contains information accessible to the current model by utilizing the divergence gap in early- and late-layer representations between model-generated and reference responses. Across math, code, and science domains, CAP delivers 35.4% greater average improvement over the base model than the strongest baseline under different selection budgets. With only 10% of the data pool, CAP surpasses or matches full-pool training on math and science. We further show that CAP transfers to multimodal data selection and is robust to response horizon and noise.

1 Introduction

Post-training data is valuable when it makes knowledge already stored in a model more accessible, but identifying examples that add useful, not merely difficult, information is challenging. CAP uses layer-wise counterfactual divergence to target unassimilated content, with reported gains across domains and budgets.

1 Introduction

  • SFT data selection is a practical lever for improving accessibility, especially when compute is limited and candidate examples vary in utility.A small, well-chosen subset can outperform a much larger indiscriminate one.Source: §1 · p. 2
  • Existing selection signals either ignore the model or reduce its response to a scalar that cannot distinguish unassimilated knowledge from intrinsic difficulty or junk.The cited approaches include text-only quality raters, embedding-space diversity methods, and loss-based scores.Source: §1 · p. 2
  • The authors interpret complementary weight-rank and hidden-state patterns as an in-and-out interplay, with deep layers where incoming data meets knowledge stored in the weights.The patterns emerge over pretraining and recur across model families, scales, computation types, and modalities.Source: §1 · p. 2
  • CAP compares layer-wise divergences between a reference response and the model’s own generation, residualizing deep divergence on shallow divergence to isolate unassimilated, learnable information.Structureless noise falls to the bottom of its ranking rather than the top, where loss-based scores may place it.Source: §1 · p. 2
  • A memory-augmented computation model derives the observed rank profiles as optimal capacity profiles and treats the activation trough as a measure of assimilated knowledge.The contribution describes a coupled equilibrium between weights and hidden states across architectures, scales, and modalities.Source: §1 · pp. 2–3
  • Across math, code, and science and selection budgets, CAP delivers 35.4% greater average improvement over the base model than the strongest baseline.It also provides nearly 1.9× the gain of random selection.Source: §1 · p. 3
  • CAP transfers to vision-language, finance, and medicine, and remains effective across nearly an order of magnitude of response horizons while rejecting injected noise.The passage contrasts its noise rejection with perplexity ranking injected noise above the entire clean distribution.Source: §1 · p. 3

2 Related Work

Related work spans external scoring, distributional coverage, and model-internal data selection, alongside research on low-dimensional computation and depth-wise structure. CAP adds a depth-localized comparison of model and reference responses, while the paper’s stated novelty is the coupling between weight and activation rank profiles.

2 Related Work

  • Post-training data selection includes external quality scoring, geometric or distributional coverage, and model-internal signals such as perplexity, instruction-following difficulty, and gradient influence.Examples include QuRating (Wettig et al., 2024), k-center selection (Sener and Savarese, 2018), and LESS (Xia et al., 2024).Source: §2 · p. 3
  • CAP differs from other model-internal methods by contrasting reference and model-generated responses across depth, rather than using scalar likelihood or gradient influence.The passage notes that gradient methods require a target evaluation set and backward passes.Source: §2 · p. 3
  • Prior work studies low-rank computation, depth-varying compression, activation structure, and middle-layer factual recall; the stated novelty is coupling weight and hidden-state ranks into a data-selection signal.The passage places this finding alongside low-rank adaptation (Hu et al., 2022) and factual-recall localization (Meng et al., 2022; Dai et al., 2022).Source: §2 · p. 3

3 In-and-Out Information Interplay: Rank Equilibrium Between Weights and Data

The section reports complementary rank profiles across model depth and develops a capacity-based account of their shape and relevance to data selection. It then defines an assimilation gap and derives a residualized score intended to distinguish unassimilated learnable content from surface difficulty and noise.

3.1 The Observation

  • Across examined models, middle layers have high-rank weights and low-rank activations, while late layers reverse the pattern; the complementary profiles emerge over pretraining.Activation compression and late re-expansion are the most consistent features, while weight profiles are noisier but generally concentrate rank in the interior.Source: §3.1 · pp. 3–4
  • The authors interpret middle-layer activation compression as relying on information stored in weights, followed by late-layer re-expansion as that information is retrieved for decoding.This motivates comparing real and model-generated responses, splitting divergence by depth, and residualizing deep divergence on shallow divergence.Source: §3.1 · p. 4

3.2 Theoretical Analysis

  • The capacity model predicts deferred retrieval and a U-shaped activation profile that descends to a knowledge-dependent trough before rising toward the output.Retrieval is delayed within bandwidth constraints, and the activation trough is r⋆ = g(K).Source: §3.2 · p. 4
  • The matching component of weight capacity vanishes at both boundaries and peaks under the activation trough; the full weight profile peaks in the interior in the matching-dominated regime.This interior peak requires the stated matching-cost assumptions.Source: §3.2 · p. 4
  • At each depth, activation information plus information still available from memory must meet task demand, and the optimal activation trough deepens monotonically as retrievable knowledge grows.The theory predicts that domain-specific continued pretraining should deepen the trough on in-domain data.Source: §3.2 · p. 4
  • The assimilation gap is zero when the model already retrieves the response’s relevant content or when the response has no storable structure, isolating unassimilated but learnable content.The section contrasts this gap with scalar loss, which cannot isolate that content.Source: §3.2 · p. 5
  • Under the stated factor model, CAP ranks unassimilated content after removing the component predictable from surface difficulty, with leakage from surface difficulty limited to an attenuation term.The result identifies a positive covariance with the surface-orthogonal gap; when σεe = 0, CAP equals bG⊥ + εl.Source: §3.2 · p. 5
  • Under the stated structural assumptions, residualization makes high-surface-difficulty noise receive a low CAP score, and top-m selection retains sufficiently high-gap examples with failure probability exponentially small in the gap margin.The section presents noise rejection as a consequence of residualization rather than a particular distance metric.Source: §3.2 · p. 5

4 CAP: Counterfactual Assimilation Profile for Model-Aware Data Selection

CAP selects examples by comparing model-generated and reference responses, separating deep divergence from shallow surface mismatch. Across the reported domains and budgets, it outperforms the strongest average baseline and shows transfer to multimodal and specialized-domain selection.

4.1 Method

  • CAP compares real prompt-reference-response runs with counterfactual runs using the model’s greedy generation, then selects examples by deep divergence residualized against shallow divergence.Responses are truncated to R tokens, 48 by default; CAP is z-scored within each domain and the highest-scoring examples are selected.Source: §4.1 · p. 6

4.2 Experimental Setup

  • The experiments use matched full-parameter SFT settings across methods and deduplicated 80k-example pools spanning math, code, and science.The evaluated tasks are GSM8K strict-match accuracy, HumanEval pass@1, and MMLU-Pro STEM with 5-shot CoT.Source: §4.2 · p. 6

4.3 Main Results

  • CAP is strongest overall across math, code, and science at four budgets, delivering 35.4% greater average improvement over the base model than LESS and winning or tying in 8 of 12 settings.At 8k examples, CAP gains an average of +8.77 points over the base model, compared with +6.81 for LESS and +4.63 for random selection.Source: Table 2 · p. 7, §4.3 · p. 7
  • With 8k examples, or 10% of the pool, CAP exceeds full-pool training on GSM8K and comes within one point on science; code is an exception where full-pool training performs better.CAP reaches 76.04 on GSM8K versus 72.48 for full-pool training, and 39.05 on science versus 39.82.Source: §4.3 · p. 8

4.4 Generalizability: Multimodal and Further Domains

  • On Molmo2-O-7B document understanding, unchanged CAP leads from 2k onward and reaches 46.15 at 8k, beating the runner-up by 4.39 points.The candidate pool contains 20k deduplicated examples; CAP trails random at 1k in a high-variance regime.Source: §4.4 · p. 8, Table 3 · p. 8
  • CAP transfers unchanged to finance and medicine, leading on every reported metric with relative improvements over PPL of 13.8% on finance and 8.2% on medicine.The reported gains span domain-knowledge and reasoning-oriented evaluations.Source: §4.4 · p. 8

5 Analysis and Ablation

The analysis weighs CAP’s scoring cost against data efficiency, tests response-horizon sensitivity, and examines noise rejection and optimization trajectories. The reported results show a cost trade-off alongside robustness and different training dynamics.

5.1 Selection Cost versus Performance

  • CAP uses 2.814 utilization-weighted GPU-hours versus 3.175 for LESS and 2.259 for IFD, but reaches random’s 69.37 GSM8K score with roughly 2.5k rather than 8k examples.The latter corresponds to a 3.2× reduction in selected data; CAP is 11.4% cheaper than LESS but 24.6% more expensive than IFD.Source: §5.1 · p. 8, Figure 4 · p. 9

5.2 Ablation: The Response-Length Parameter R

  • Across three response horizons, CAP beats random in all nine domain-horizon settings and exceeds the strongest baseline in six, matching it in one more.R = 48 performs best on math and code, while science improves with a longer horizon and reaches 39.72 at R = 144.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 Noise Rejection

  • With 5% random junk added, CAP ranks the noise below 99.7% of clean examples, while perplexity ranks it above the entire clean distribution.The authors attribute the distinction to residualization: structureless noise has no assimilation gap despite high surface difficulty.Source: §5.3 · p. 9, Figure 5 · p. 10

5.4 The Optimization Path

  • In the 8k math trajectories, CAP reaches a lower final held-out loss than random selection, 0.572 versus 0.584, while the two runs converge to distinct regions in the projection.The authors caution that a two-dimensional projection cannot establish distinct basins in the full parameter space.Source: Figure 6 · p. 10, §5.4 · p. 10

6 Conclusion

The conclusion presents CAP as a model-aware way to select data that targets information not yet accessible to the model. The appendix grounds its interpretation in a capacity model and states important limits on the theory-to-measurement bridge.

6 Conclusion

  • 35.4% greater average improvement than the strongest baseline; with 10% of the data, CAP surpasses or matches full-pool training on math.The conclusion also reports transfer to multimodal and specialized domains, and robustness across response horizons and against unstructured noise.Source: §6 · p. 10

A.1 The Memory-Augmented Computation Model

  • The capacity model requires activations to carry content not supplied later plus an address for content that will be retrieved, while minimizing activation capacity.Addressing stored content is assumed cheaper than carrying it directly; total weight capacity is a tiebreaker.Source: §A.1 · p. 14

A.3 Mapping the Model to Transformers

  • The transformer mapping uses activation and weight stable rank as capacity proxies, with activation rank less reliable for strongly anisotropic spectra and weight rank a coarser measure.The depth split 0.33L/0.66L is a conservative estimate of the model's phase boundaries.Source: §A.3 · p. 15

A.10 Scope of the Theory

  • The theory predicts profile shapes rather than values; the weight inverted-U additionally depends on matching-cost assumptions and the matching-dominated regime.Stable rank is a proxy, and predicted-versus-measured curve comparisons are qualitative.Source: §A.10 · p. 18
  • The bridge from the theory to CAP depends on a factor model, whose identification result is conditional; the theory predicts signs and mechanisms, not effect sizes.The empirical case for CAP rests on Section 4.3.Source: §A.10 · p. 18

A.4 Proof of Theorem 1

Theorem 1 reduces the optimization to activation capacities and retrieval timing, yielding a maximally deferred retrieval schedule and a U-shaped activation profile. The accompanying results connect the profiles to knowledge retrieval and describe their limits and scope.

A.4 Proof of Theorem 1

  • The optimal retrieval schedule is uniquely maximally deferred, producing a descending activation profile, a flat trough, and a retrieval-phase ascent.The profile is quasi-convex; its weight profile has an interior maximum under Assumption 4.Source: §A.4 · pp. 16–17

A.5 Proof of Theorem 2

  • The weight-side inequality is loose by transport and matching costs; exact conservation links activation capacity to retrievable knowledge, not hidden-state rank one-for-one to weight rank.Source: §A.5 · p. 17

A.6 Proof of Corollary 1

  • At zero retrievable knowledge, activations do not re-expand: their capacity is non-increasing, with no interior minimum below the output value.The mapping ψ(K) = cK − φ(K) is strictly increasing, and is affine exactly when φ is affine.Source: §A.6 · p. 17

A.7 Proof of Theorem 3

  • Theorem 3’s residual expression separates a surface-aligned term, a G⊥ component, and noise.Source: §A.7 · p. 17

A.8 Proof of Corollary 2

  • For contaminants, the expected CAP expression has negative slope when δ < bγ/(a′ + bγ); only the sign and affine shape transfer to measurable quantities without further assumptions.Source: §A.8 · p. 18

A.9 Proof of Proposition 1

  • Pairwise CAP differences contain a residual-difference term and sub-Gaussian noise; the top-m claim uses a union bound over comparisons.Source: §A.9 · p. 18

B Full Cross-Model Rank Survey

  • Figure 7 surveys weight and data stable rank across normalized decoder depth for 12 models, while Table 5 describes the surveyed checkpoints.The checkpoints span architectures and attention mechanisms, including dense, MoE, and vision-language models.Source: B Full Cross-Model Rank Survey · p. 19

C Full Main-Domain Results

This appendix section points to the complete main-domain numerical results and explains how selection compute is measured.

C Full Main-Domain Results

D Selection Cost Details

  • Utilization-weighted GPU-hours are the primary compute measure because they account for observed utilization rather than charging equally for active and idle GPU time.Source: D Selection Cost Details · p. 19

E Further Text Domains

The further-text-domains section applies CAP unchanged to finance and medicine, with supplementary tables identifying each evaluation setting.

E Further Text Domains

  • CAP is applied unchanged to finance and medicine using the same protocol as the main experiments and a 2k selection budget.Source: E Further Text Domains · p. 19

F Scaling the Budget on Code

The code budget sweep extends selection to 60k examples, or 75% of the pool, and CAP maintains a margin over random and PPL at every tested budget.

F Scaling the Budget on Code

  • At 60k selected examples, CAP reaches +15.24 over the base model and stays ahead of random and PPL at every budget.The results indicate a useful total ordering over the pool, rather than only a reordering of its head.Source: F Scaling the Budget on Code · p. 21

G The Optimization Path

CAP’s trajectory reaches a lower held-out GSM8K loss than random selection, and the appendix describes the shared training protocol, data pools, and scoring configurations used for the experiments.

G The Optimization Path

  • The runs start from a shared base model at loss 0.666 and are compared on one held-out 5-shot GSM8K loss surface.Their parameter paths are projected onto the top two joint-trajectory principal components, which capture 89% of variance.Source: G The Optimization Path · p. 22
  • After 25 steps CAP reaches 0.577, below random’s best 0.582 at step 75, and settles at 0.572 versus random’s 0.584.Random selection rises from 0.582 at step 75 to 0.584 by step 125, while CAP is monotone to within noise.Source: Figure 9 · p. 23, G The Optimization Path · p. 23

H.1 Supervised Fine-Tuning Protocol

  • All text experiments use full-parameter SFT with the same pool, budget, shuffling, seed, optimizer state, schedule, and epochs across selection methods.Only the selected example indices vary between methods.Source: §H.1 · p. 23, Table 13 · p. 24
  • Single-epoch training makes budget curves conflate more data with more optimizer steps; fixed-budget method comparisons use the same steps and schedule.The 2048-token maximum truncates a small tail of long math chain-of-thought responses, identically across methods.Source: §H.1 · p. 23

H.2 Pool Construction

  • Pool construction normalizes and deduplicates prompts, then samples 80,000 examples for each main math, code, and science pool.The multimodal pool deduplicates on normalized questions alone, so identical questions paired with different documents can be merged.Source: §H.2 · p. 24
  • Finance, MedCaseReasoning, and MedMCQA pools are filtered and deduplicated before applying the selection budget.Their valid or unique prompt counts differ, and MedMCQA removes 34 duplicate rows.Source: §H.2 · p. 24

H.3 Scoring Configuration

  • CAP scores candidates using up to 48 response tokens, a greedy counterfactual generation, and hidden states from the first and final thirds of decoder blocks.Its residualizing regression is fitted separately on each domain pool.Source: §H.3 · p. 25
  • Baseline scoring uses perplexity, QuRating’s educational-value logit, IFD ratios, or normalized prompt-response embeddings for k-center.IFD excludes ratios of one or greater; k-center embeds both prompt and response.Source: §H.3 · p. 25

H.4 Evaluation

The evaluation section specifies shared inference settings and benchmark-specific prompts, decoding, and scoring for math, code, and science.

H.4 Evaluation

  • All three text benchmarks use lm-evaluation-harness with bfloat16 inference, a 4,096-token context, and batch size 16.Source: §H.4 · p. 25
  • GSM8K evaluation uses all 1,319 test examples with five demonstrations and reports strict exact match after extracting the answer following ####.Decoding is greedy with up to 1,024 generated tokens.Source: §H.4 · p. 26
  • HumanEval evaluation uses all 164 test problems without demonstrations and reports pass@1 using the official execution-based tests.Decoding is greedy with up to 1,024 generated tokens.Source: §H.4 · p. 26
  • MMLU-Pro STEM reports macro-average exact match over five STEM subjects using their complete test subsets and five chain-of-thought demonstrations per subject.Decoding is greedy with up to 512 generated tokens.Source: §H.4 · p. 26

Related papers

Loading 2609.33010v1…