What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions

1 Introduction

• Data valuation is presented as a potential response to the lack of agreed-upon mechanisms for crediting or compensating data providers, while existing methods face scalability challenges on LLMs and large datasets.Sampling-based approaches such as Shapley value and Datamodels require repeated retraining; gradient-based methods avoid that but face high compute and memory costs from high-dimensional gradients.Source: §1 · pp. 1–2
• LOGRA uses backpropagation’s gradient structure to improve gradient-projection efficiency, compute projected gradients without materializing full gradients, and support implementation with small add-on layers.The introduction describes space and time complexity as improving from O(nk) to O(k), where n and k are model and projection dimensions.Source: §1 · p. 2
• LOGRA achieves competitive accuracy against more costly baselines, with up to 6,500× higher throughput and 5× lower GPU memory on Llama3-8B-Instruct and the 1B-token dataset.The comparison is against EKFAC influence, described as the state-of-the-art and only runnable baseline at this scale.Source: §1 · p. 2
• The paper interprets influence-function damping as spectral gradient sparsification and derives a theoretical motivation for gradient projection plus a specialized PCA initialization scheme for LOGRA.The same contribution overview introduces LOGIX as software for converting training code into data-valuation code that is compatible with LLM ecosystem tools and extensible to other algorithms.Source: §1 · p. 2

2 Scalability Bottlenecks in Influence Functions

• Data valuation assigns an example value from the change in utility when that example is included or excluded from a training subset.The definition weights subset-level utility differences by an algorithm-specific term.Source: §2 · pp. 2–3
• Leave-one-out valuation considers subsets missing one example, but calculating it directly requires retraining for each example.Influence functions instead use a Newton-step approximation and a first-order Taylor estimate of the effect on test utility.Source: §2 · p. 3
• Influence functions approximate removing a training example using gradient information, but their iHVP and all-training-data gradient requirements make LLM-scale use costly.Naive iHVP has O(n^2) space and O(n^3) time complexity; computing gradients for all training data can cost more than $1M in LLM pretraining contexts.Source: §2 · p. 3
• Low-rank gradient projection replaces n with k ≪ n in iHVP complexity and lets stored training projections support later test-data queries without recomputing gradients.This turns influence scoring into a vector-similarity-search problem.Source: §2 · p. 3
• Naive batched projection costs O(bkn), while full per-sample gradients and the projection matrix require O(bn) and O(kn) memory, respectively.The passage notes that for an 8B model, each memory cost amounts to 32GB×b or 32GB×k; prior approaches address these costs separately and use small k and b.Source: §2 · p. 3

3 Scaling Data Valuation & Influence Functions

• The section combines an efficiency-focused algorithm, a theoretical rationale for gradient projection in influence functions, and software intended to facilitate data valuation research.The stated motivation is to address compute and memory costs as well as implementation challenges.Source: §3 · p. 4

3.1 Algorithm: Memory and Compute Efficient Gradient Projection

• LOGRA projects forward and backward activations into low-dimensional spaces, then reconstructs the projected gradient without first materializing the full gradient.This uses the Kronecker-product structure of backpropagated gradients; at an 8B model/4k projection size, its projection matrix is about 1GB versus 128TB for naive projection.Source: §3.1 · p. 4

3.2 Theory: Why Gradient Projection Works in Influence Functions

• Damping in influence functions softly limits small gradient components, motivating projection as a hard limit and a KFAC-based PCA initialization for LOGRA.The initialization uses top eigenvectors of forward and backward activation covariances to approximately discard smaller Hessian components.Source: §3.2 · p. 5

3.3 Software: Compatibility, Extensibility, and Usability

• LOGIX uses PyTorch hooks to intercept training logs and compute statistics, with a context manager that adds the needed hooks and primitives with minimal code changes.The hook-based design supports integration with tools such as FSDP, autocast, and compile, and allows users to add custom primitives.Source: §3.3 · p. 5

4.1 Quantitative Accuracy with Counterfactual Evaluation

• Evaluations use brittleness and LDS on MLP/FashionMNIST, ResNet-9/CIFAR-10, and GPT2/WikiText, with TRAK omitted for WikiText.The setup allows up to 1,800 retraining rounds; top-k data identification itself is not allowed to use repeated retraining.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Scaling to Billion-Scale Models & Datasets

• The billion-scale study values a random 1B-token OpenWebText subset for GPT2-XL, Pythia-1.4B, and Llama3-8B-Instruct.The efficiency comparison uses one A100 GPU; Table 1 reports logging throughput in tokens/s and influence throughput in train-test pairs/s.Source: §4.2 · p. 7, Table 1 · p. 7
• LOGRA slightly underperforms EKFAC influence but noticeably outperforms other baselines, while achieving almost 6,500× EKFAC’s throughput.Table 1 compares memory and compute efficiency; the throughput comparison is for influence computations.Source: Table 1 · p. 7, §4.1 · p. 7, §4.2 · p. 8
• LOGRA avoids recomputing training gradients by storing projected gradients, trading compute for disk space and data-I/O latency.For the stated setup, it overlaps loading with influence computation and reaches almost 6,500× EKFAC’s throughput.Source: §4.2 · pp. 7–8
• Most valuable examples for Llama3-8B-Instruct and GPT2-XL share qualitative similarities with queried outputs, including semantics, style, or token overlap.Examples include material on lucid dreaming, coal-industry emissions, and weightlifting.Source: §4.2 · pp. 8–9, Figure 5 · p. 8
• Some retrieved data lack apparent qualitative similarity to outputs, especially for Pythia-1.4B; proposed explanations include incoherent outputs and missing similar data in the valuation subset.The authors also suggest gradients may encode features that are difficult for people to perceive.Source: §4.2 · p. 9

Data Valuation

• Game-theoretic valuation methods such as Data Shapley [15] often require repeated model retraining, which is costly even for small models.Other approaches use reinforcement learning [61], meta-learning [9], or training-free methods, with some requiring other model training or high computation.Source: §5 · p. 9

Influence Functions

• Influence functions estimate the effect of adding or removing training data without retraining, but iHVP computation and per-example gradients challenge scaling.Prior work explores gradient projection, last-layer approximations, and filtering; subset-layer influence may miss overall influence [12].Source: §5 · p. 9
• DataInf [34] and LESS [59] use LoRA for influence functions in finetuning, whereas LOGRA also supports pretraining analyses.Representation-space and TF-IDF filtering are alternatives, but can bias sequence selection; TF-IDF may miss semantically related examples with few shared tokens.Source: §5 · p. 9

6 Conclusion

• The work develops a gradient-projection algorithm to improve influence-function scalability and a simple, interoperable software package.The system is explored as a potential technical route to credit or compensate LLM training-data providers.Source: §6 · p. 9
• Experiments show LOGRA achieves competitive counterfactual accuracy against more expensive baselines while scaling to billion-scale models and datasets.The authors describe this as demonstrating the system’s initial practical potential.Source: §6 · p. 9
• The conclusion points readers to Appendix F for discussion of broader impacts and limitations.Source: §6 · p. 9

A Qualitative Analysis

• The appendix shows top-2–4 valuable data for each query and includes cases where selected data do not qualitatively resemble the queried output.Source: A Qualitative Analysis · p. 14

A.1.5 Example 5

• LOGRA identified novel literature as the most valuable data in one Llama3-8B-Instruct example.Source: Figure 10 · p. 18

A.3 Pythia-1.4B (with many failure cases)

• Pythia-1.4B generally produced poorer most-valuable-data examples than Llama3-8B-Instruct and GPT2-XL.The authors hypothesize that training-data quality and training steps may affect which features influence functions identify.Source: §A.3 · p. 25
• For a soccer query, LOGRA captured the broad topic but generally lacked specificity in its Pythia-1.4B example.The passage notes one exception: the third-ranked data point mentions Cristiano Ronaldo and the Ballon d’Or.Source: Figure 17 · p. 26
• The authors suspect a random URL in one Pythia-1.4B model output dominates the query gradient and affects the valuation result.Source: Figure 18 · p. 27

B Code Examples

• The LogIX code examples register a model, add LoGra, and log gradients during the existing training loop before loading logs to compute influence scores.Source: B Code Examples · p. 30, §B.1 · p. 30, §B.2 · p. 30

• For EKFAC influence and LOGRA, damping is set to 0.1× the mean eigenvalue for all layers, following Grosse et al..Source: C Experiment Details · p. 31

C.1 Quantitative Counterfactual Experiments

• LOGRA projects gradients with k_i = k_o = 128; the counterfactual experiments use one A100 GPU with 80GB VRAM and the cited baseline configurations.Source: §C.1 · p. 31
• Brittleness is measured by the fraction of 100 initially correct classification examples misclassified after removing up to k influential points; language modeling reports mean test perplexity after removal and retraining.Classification averages across three random seeds; the selected language-modeling set contains 50 test sequences.Source: §C.1 · p. 31
• LDS uses 100 half-sized data subsets, with 10, 20, and 5 retrains per subset for FashionMNIST, CIFAR-10, and WikiText, respectively.Reported Figure 4 results give mean and standard deviation across five distinctly trained models.Source: §C.1 · p. 31

C.2 Scaling to Billion-Scale Models and Datasets

• The LLM experiments use k_i = k_o = 64, up to four A100 GPUs with 80GB VRAM, and tf32 precision to prevent gradient-quality degradation.Source: §C.2 · p. 31
• Unlike the counterfactual evaluations, the LLM experiments require no retraining and have no other noticeable hyperparameters to report.Source: §C.2 · p. 31

D Derivation of Lemma 1

• The derivation assumes an empirical Fisher Information Matrix in place of the Hessian and approximately matching train and test gradient distributions.Lemma 1 is introduced under these assumptions, but the supplied passage does not include its conclusion.Source: D Derivation of Lemma 1 · p. 32

E LOGIX Details

• LOGIX is designed to convert existing training code into data-valuation code by reusing training logs and statistics rather than rebuilding gradient collection.The design aims for compatibility with the LLM development ecosystem and reuses gradients as a by-product of training.Source: §E.1 · p. 33

E.1 Differences with Other Tools

• Unlike mechanistic interpretability tools, LOGIX supports dataset-level statistics and prioritizes efficient data I/O for projected-gradient similarity search.The authors identify TransformerLens and pyvene as examples of tools with different priorities.Source: §E.1 · p. 33

E.2 Optimizations

• LOGIX uses memory-mapped files for sequential gradient access and overlaps data transfers with computation; CPU offloading may slow logging depending on communication bandwidth.It also delays distributed-statistics synchronization until the end of an epoch.Source: §E.2 · pp. 33–34

F.1 Broader Impacts

• The authors leave equitable data-value determination open because no agreed social norm exists, calling for further accuracy and efficiency research alongside social discussion.The work frames itself as an initial technical effort rather than a proposal for setting exact values.Source: §F.1 · p. 35

F.2 Limitations & Future Work

• Influence-function methods are susceptible to large-gradient outliers, especially when a few tokens dominate a sequence’s summed gradient and hurt valuation accuracy.The work tried self-influence normalization; other filtering heuristics, including an L2/L1 norm ratio, remain a possible research direction.Source: §F.2 · p. 35
• LOGIX lacks extensive system support such as a high-performance vector database, while further system optimizations and alternatives to low-rank spectral compression remain future directions.The authors specifically mention top-k and low-bit compression as strategies worth exploring or combining.Source: §F.2 · p. 35