PaperlayerView exact revision on arXiv

Source-linked AI summary

Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

Zijun Wang, Haoqin Tu, Weidong Zhou, Yiyang Zhou, Xiaohuan Zhou, Bingni Zhang, Weiguo Feng, Taifeng Wang, Cihang Xie, Fengze Liu

arXiv:2604.15706v1cs.CL

TL;DR

Target-oriented pretraining needs data selection aligned with desired capabilities, while existing approaches often use opaque distilled signals. The paper introduces training-free NAG-based Ranking, which selects data through sparse neuron-level similarity, and reports consistent gains across benchmarks, settings, and model backbones. Its analyses identify a sparse, layer-distributed functional backbone underlying the method.

  • Problem

    Existing target-oriented data-selection methods distill internal signals into auxiliary classifiers, making the resulting signals opaque and difficult to diagnose or refine.

  • Method

    NAG-based Ranking quantifies neuron impact in an off-the-shelf LLM, builds compact layer-wise Neuron-Activated Graphs, and ranks candidate data by similarity to target examples without additional training.

  • Results

    NAG-based Ranking consistently improves target-oriented pretraining across benchmarks, target settings, and backbone models, including a 4.9% average gain over random sampling.

  • Takeaways & Limitations

    NAG isolates a sparse functional backbone of high-impact neurons and captures task-discriminative signals distributed across layers.

  • Takeaways & Limitations

    Main experiments use a 1.2B model trained on 30B RefinedWeb tokens; larger models and more diverse corpora remain future work.

Abstract

from arXiv · show

Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented language model (LM) pretraining by introducing Neuron-Activated Graph Ranking (NAG-based Ranking), a training-free and interpretable framework for target pretraining data selection. Rather than using black-box representations, our approach directly characterizes each target input by a sparse set of high-impact neurons in any off-the-shelf LLMs. Concretely, we quantify neuron impact and select the most influential neurons across layers into a compact Neuron-Activated Graph (NAG), and rank candidate data by NAG similarity to target examples. We conduct experiments across six benchmarks, where our NAG-based Ranking improves target-oriented pretraining by 4.9% on average over random sampling, and also outperforms state-of-the-art baselines by 5.3% accuracy on HellaSwag. It also remains effective under a more applicable multi-target setting, where our best setup surpasses two baselines by 1.1% and 4.1%, respectively. Furthermore, we provide a comprehensive analysis on why and how our NAG works, e.g., deactivating NAG-selected neurons (only 0.12% of all) causes a 23.5% performance collapse, and restricting NAG to the final layer incurs a 4.1% average drop, indicating that NAG captures a sparse "functional backbone" for learning target features. We release the code at https://github.com/asillycat/NAG.

1. Introduction

Target-oriented pretraining requires selecting data aligned with specific capabilities, but existing quality heuristics and black-box target signals provide limited interpretability. NAG-based Ranking addresses this by ranking data through interpretable neuron-level similarity and shows consistent performance and mechanistic gains.

  • Motivation: Specific capabilities, rather than generic quality, should determine which pretraining data are selected for target-oriented language-model training.The paper frames education, medicine, and research domains as examples of targeted scenarios.
  • Motivation: Existing selection pipelines rely on heuristics or implicit quality assumptions, leaving data choice misaligned with the capabilities models need to develop.
  • Motivation: Black-box distillation of embeddings or performance-correlated losses produces opaque signals that are difficult to diagnose or refine.
  • Method: NAG-based Ranking quantifies neuron impact, organizes influential neurons across layers into compact graphs, and ranks candidates by similarity to target examples without additional training.The method uses interpretable signals from any off-the-shelf LLM.
  • Results: 4.9% average improvement over random sampling demonstrates consistent gains across target-oriented pretraining benchmarks and settings.The method also outperforms strong general-quality classifiers and target-oriented baselines, while multi-target performance exceeds two baselines by 1.1% and 4.1%, respectively.
  • Analysis: Deactivating 0.12% of model neurons causes a 23.5% performance collapse, while restricting NAG to the final layer causes a 4.1% average drop.These analyses identify sparse high-impact neurons and distributed layer-wise signals as central to NAG's behavior.

2. Method

NAG characterizes inputs through influential neurons and ranks candidate pretraining data by structural similarity to target examples. It provides a compact, layer-wise, neuron-level representation for targeted data selection.

  • Neuron impact and NAG construction: NAG characterizes each input through neurons that strongly influence the model’s computation, rather than black-box representations.The framework quantifies neuron impact, organizes influential neurons across layers, and compares these structures across inputs.
  • Neuron impact and NAG construction: A neuron is defined as a column of a projection matrix in Attention or FFN modules, and its impact is estimated by the output change from deactivation.Deactivation zeros the neuron’s corresponding weight-matrix column; the resulting impact reduces to the magnitude of that column’s contribution to the layer output.
  • Neuron impact and NAG construction: NAG selects a fixed number K of highest-impact neurons from each layer and represents the input as layer–neuron index pairs.The selected neurons are treated as activated neurons for the input, forming a structured, layer-wise graph.
  • NAG-based data selection: NAG similarity measures structural overlap between inputs or between an input and a group profile formed from neuron-activation frequencies.When every sample selects K neurons per layer, the group similarity can be computed as an efficient equivalent of average pairwise similarity.
  • NAG-based data selection: Candidates are scored by similarity to the aggregated target NAG profile, ranked in descending order, and selected according to a predefined fraction.This prioritizes samples whose neuron-level processing structures align most closely with the target group.

3. Experiments

Experiments evaluate NAG-based Ranking across six benchmarks, backbone models, and single- and multi-target settings. NAG consistently improves target-oriented pretraining and complements general-quality selection.

  • Experimental Setup: 20% of the 150B-token source pool is selected as 30B pretraining tokens, with evaluation spanning six reasoning, factual, narrative, and commonsense benchmarks.Experiments use document-level ranking and fixed training settings across methods.
  • Single-Target Results: 4.9% average improvement over random selection is achieved across target benchmarks.NAG also outperforms FineWeb-Edu by 2.4% on average.
  • Single-Target Results: 1% average improvement over BETR is observed, with larger gains on ARC-C (+2.4%) and HellaSwag (+2.2%).NAG aggregates neuron-level signals across all layers, unlike BETR’s last-layer embedding similarity.
  • Backbone Dependence: 4.7%-5.0% target-specific enhancements remain consistent across backbone models.This result indicates effectiveness across the evaluated architectures.
  • Multi-Target Results: 3.1% improvement over random selection and 0.6% over FineWeb-Edu are obtained under mixed targets.Using Qwen3-1.7B-Base, the best setup gains 3.6% over random selection.
  • Combined Ranking: 1.8% average improvement over FineWeb-Edu is achieved by integrating NAG with its quality signals.On ARC-C, the combined approach reaches 35.4%, versus 34.7% for FineWeb-Edu and 34.3% for NAG alone.

4. Analysis

The analysis explains NAG through sparse critical neurons, task-discriminative representations, and rankings aligned with downstream utility. Multi-layer, up-projection, and highly sparse neuron signals are especially effective.

  • Why NAG Works: 0.12% of model neurons selected by NAG cause a 23.5% performance drop when deactivated.The measured performance falls from 60.6% to 37.1%, supporting NAG’s identification of critical neurons.
  • Why NAG Works: NAG representations form clusters aligned with task identities, with mathematically related MathQA and GSM8K clusters closer together than XNLI.The clustering analysis uses pairwise NAG distances and t-SNE visualization.
  • Why NAG Works: NAG ranking improves performance as lower-ranked samples are removed, unlike random selection and BETR.Random selection drops from 51.6% to 50.9%, while BETR degrades by 0.5% as filtering increases.
  • How NAG Works: 60.6% performance is reached with up-projection neurons, compared with 58.0% for down-projection and 56.7% for k-projection neurons.The paper hypothesizes that up-projection layers better isolate task-specific signals in a higher-dimensional latent space.
  • How NAG Works: 4.1% average performance is lost when NAG is restricted to the final layer.The comparison supports distributing task-relevant signals across multiple layers.
  • How NAG Works: 0.3% layerwise neuron sparsity yields peak performance across Qwen3-Base model scales.Increasing the ratio beyond this level provides little or reverse gain, while larger models perform better at fixed sparsity.

5. Related Works

Prior data-selection methods use general-quality proxies or compressed representations to align pretraining data with targets. NAG instead derives task alignment directly from neuron-level computation.

  • General-Quality Selection: General-quality methods use curated classifiers or coarse heuristics that do not explicitly consider downstream capabilities.Examples include educational-value classifiers, perplexity filtering, language identification, and deduplication.
  • Target-Oriented Selection: Target-oriented methods estimate relevance through proxy signals such as learned embedding similarity.These approaches infer task alignment indirectly through compressed black-box representations.
  • NAG’s Distinction: NAG derives task alignment directly from neuron-level computation in an off-the-shelf LLM.This produces an interpretable signal for target-oriented data selection.

6. Conclusion

NAG-based Ranking consistently improves target-oriented pretraining while providing an interpretable neuron-level account of its effectiveness. Its main experimental scope remains bounded by model, corpus, and multi-target mixture choices.

  • NAG-based Ranking represents inputs with Neuron-Activated Graphs and ranks data by neuron-level similarity to target examples without additional training.It relies on interpretable signals from off-the-shelf LLMs.
  • NAG-based Ranking shows consistent gains over random sampling and strong baselines across benchmarks, target settings, and backbone models.
  • Deactivating NAG-selected neurons reveals a sparse functional backbone, while task-discriminative signals are distributed across layers.These analyses connect NAG effectiveness to high-impact neurons and multilayer representations.
  • The main experiments train a 1.2B model on 30B RefinedWeb tokens, leaving larger models and more diverse corpora for future work.Preliminary 7B HellaSwag results are described as encouraging.
  • The multi-target setting uses a simple equal-budget mixture as a lower-bound scenario, while more advanced mixture strategies remain future work.

A.1. Training

The training and evaluation setup uses a 1.2B-parameter model, RefinedWeb data, six reasoning and commonsense benchmarks, and target sets drawn from train or validation splits. Neuron deactivation experiments quantify the functional importance of NAG-selected neurons under controlled settings.

  • Training: The model has 1.2B parameters and is trained with a maximum sequence length of 2048 using cosine decay, AdamW, and specified learning-rate and warm-up settings.The initial and final learning rates are 5×10^-4 and 5×10^-5, respectively.
  • Evaluation: The evaluation covers six benchmarks spanning multiple-choice reasoning, factual question answering, narrative understanding, and coreference-based commonsense reasoning.The benchmarks include ARC-Challenge, HellaSwag, MMLU, TriviaQA, XStoryCloze, and XWinograd.
  • Target construction: Target sets for NAG extraction are sampled from corresponding training or validation splits and checked for 13-gram overlap with benchmark test instances.
  • NAG configuration: NAG width uses K ≈ rk × d_internal with rk = 0.3%, while the reported effective widths vary across backbone models.
  • Neuron deactivation: Deactivating NAG-selected neurons causes a 23.5% average performance drop across tasks, whereas equal-sized random deactivation has negligible effects.The experiment deactivates approximately 0.12% of all neurons.

B.3. Fine-Grained Ablation within NAG

Fine-grained ablations compare neuron-selection criteria and validate the impact score against loss change, while the similarity analysis establishes an equivalence between group and average pairwise similarity.

  • Fine-grained neuron ablation: Deactivating 28 High-∆impact neurons causes a 17.8% performance drop, while High-Mean and random selections yield negligible degradation.High-∆impact neurons are selected by differences between target and random-input mean impact scores.
  • Fine-grained neuron ablation: High-∆impact neurons are more discriminative and task-sensitive because they respond selectively to different samples rather than being uniformly activated.
  • Impact-score validation: The impact score correlates with behavioral loss change at +0.71 ± 0.02, supporting it as a local proxy for expensive end-to-end output changes.The validation groups neurons by rank across layers to stabilize loss-change measurements.
  • Impact-score validation: Deactivating the top 0.8% neurons causes 159× more loss change than deactivating mid-rank neurons.
  • Similarity equivalence: When each sample selects exactly K neurons per layer, group similarity is mathematically equivalent to average pairwise similarity.The frequency-weighted form avoids enumerating all pairs and is therefore more efficient to compute.

E. Clustering datasets

The efficiency analysis measures both downstream compute savings and the cost of running NAG selection across six benchmarks. NAG improves compute efficiency over baselines while using a one-time extraction stage and linear-time CPU ranking.

  • Compute efficiency: 1.27–2.42× average compute-multiplier improvement is achieved by NAG over baselines across benchmarks.On HellaSwag, gains range from 1.54–2.65×; on XStoryCloze, the maximum improvement relative to Random is 3.7×.
  • Selection cost: NAG extraction requires one forward pass per candidate document, is embarrassingly parallel, and costs 192 GPU-hours for a 150B-token pool on H100-SXM-80GB.Extracted features are target-independent and reusable across target tasks.
  • Compute efficiency: Compute multipliers summarize the relative compute required by NAG and baselines to reach the same accuracy across six benchmarks.A higher multiplier indicates that NAG requires less compute for equivalent accuracy.
  • Selection cost: NAG features from a smaller Qwen3-0.6B-Base extraction model still outperform all baselines, reducing the required extraction-model scale.
  • Selection cost: Ranking is CPU-only with overall complexity O(N), using a threshold estimated from a small random subset instead of globally sorting all candidates.
  • Future work: Further cost reductions through throughput optimization and coarse-to-fine selection are left for future work.

G. Preliminary Analysis on the Relationship Between Task-Level Discriminability of NAG Signals and Downstream Utility

The analysis tests whether task-level discriminability in NAG representations predicts the usefulness of selected pretraining data. Across NAG widths, stronger clustering alignment is associated with higher downstream utility, while small target sets produce stable rankings.

  • Task-level discriminability: NAG widths K = 5, 20, 40 produce task representations whose separability varies, with K = 5 visibly reducing separation between tasks.MathQA and GSM8K remain relatively close because both require mathematical reasoning, while linguistic tasks are more separated.
  • Clustering evaluation: K-Means clustering based on NAG similarity evaluates task alignment using Purity, NMI, and ARI.These metrics assess cluster homogeneity, global agreement with dataset labels, and pairwise consistency corrected for chance.
  • Task-level discriminability: Task-level cluster positioning reflects task relevance: MathQA and GSM8K form closer clusters while remaining separated from XNLI.This pattern supports NAG’s ability to capture task-relevant features for target-oriented data selection.
  • Discriminability and utility: Higher Purity, NMI, and ARI consistently correlate with higher HellaSwag accuracy when the corresponding NAG configuration selects data.The analysis compares clustering quality with downstream performance under the same NAG settings.
  • Target-set sensitivity: NAG-based ranking remains highly robust to target set size and choice, with Spearman ρ ≥0.999 across sizes and 94% top-20% overlap using 200 targets.The rankings are nearly identical across repeated samples and relative to the full 10k-target baseline.

I. Scaling Experiments

Scaling experiments examine whether NAG remains effective when either the trained model or extraction model changes scale. The reported results show comparable gains at larger training scale and strong performance with a smaller extraction model.

  • Larger trained model: +8.4% improvement over random is achieved at 7B scale, compared with +9.0% at 1.2B scale on HellaSwag.The 7B experiment trains on 100B selected tokens from a 500B RefinedWeb pool, while NAG is extracted from Qwen3-1.7B-Base.
  • Smaller extraction model: NAG extracted from Qwen3-0.6B-Base outperforms all baselines on HellaSwag when selecting 30B tokens for training a 1.2B model.The extraction model is smaller than the trained model in this setting.
  • Scaling conclusion: NAG transfers across extraction-model and trained-model scale gaps, supporting its practical scalability.This conclusion combines the larger-trained-model and smaller-extraction-model experiments.

J. Statistical Reliability of the Main Results

The paper evaluates whether its main improvements are reliable across repeated runs and test-set sampling uncertainty. Both analyses report variation smaller than NAG’s observed gains.

  • Run-to-run variance: 0.18%–0.55% standard deviations across five random-baseline runs are an order of magnitude smaller than NAG’s gains across all benchmarks.The repeated-run analysis measures variance in the training and evaluation pipeline.
  • Overall reliability: The reliability checks support the statistical consistency of the reported NAG improvements.The evidence combines repeated-run variance with evaluation standard errors.
  • Evaluation standard errors: NAG’s gains consistently exceed the binomial standard-error range across all benchmarks.The standard error is computed as p(1 −p)/n, where n is the test-set size.
Loading 2604.15706v1…