PaperlayerView exact revision on arXiv

Source-linked AI summary

Still funded, no longer counted: how NIH's 2025 award reviews changed what the government counts as minority health research

Fangfang Xie, Jingwen Zhang, Haining Wang

arXiv:2610.03443v1cs.DLcs.CY

TL;DR

Funders use award text to classify research, but little evidence addresses how counts behave when the funder itself governs the text being classified. This paper traces population names through continuing NIH awards and finds that category assignments followed edited summaries even as projects kept funding, while the records cannot establish whether the underlying research changed.

  • Problem

    Prior work largely treats classified award text as applicant-written, leaving open how a funder’s edits affect the validity of counts used to describe its portfolio.

    Source: §1 · p. 2

  • Method

    The study follows population names across continuing awards, compares summary versions with independent practice records, and tracks changes in NIH’s category assignments.

    Source: §4 · p. 7, What this adds to research evaluation · p. 19

  • Results

    In most reviewed losses, edits recast aims concerning the population, and FY2025 summaries read alone no longer met the category definition even though projects kept funding.

    Source: §1 · p. 3, §7 · p. 22

  • Takeaways & Limitations

    The category followed edited text, so funded research that still involved a population could disappear from the government’s count.

    Source: §1 · p. 3, §7 · p. 22

    Practice records capture involvement rather than REMHR’s purpose, and the short observation window cannot show whether reviews changed research practice.

    Source: Limitations and future research · p. 21

Abstract

from arXiv · show

Funders know their portfolios through software that classifies award text. The US National Institutes of Health (NIH) reports its spending in more than 300 categories mined this way, and work on classification and indicators treats the text as the applicant's to write. In 2025 NIH required "DEI language" removed from awards not supporting DEI activities, and so policed the words it also counts. We followed population names through 37,790 continuing awards, checking them against practice records. Names were informative: in new awards with trial baselines, a title naming Black populations predicted a 60-percentage-point higher enrolled share. Text features predicted which names survived, and practice records added little. NIH's minority health category followed the names: of continuing projects carrying it, 99.4% with unchanged summaries kept it, against 15.7% of those whose summaries no longer named a racial or ethnic population, while the projects kept their funding. Blinded reviewers judged 42 of a random 50 such losses to have met its definition in FY2024. Comparing both summaries of 70 name losses, they found aims concerning the population recast in 49. Read alone, 63 FY2025 summaries no longer met the definition. Awards naming sexual and gender minorities were recorded as terminated 37.8 points more often after adjustment for listed terms, institute and activity. We call the mechanism word targeting, its boundary population targeting, and its product uncounted science: funded research the count no longer records. The count followed the edited text, and the edited record cannot say whether the research changed with it.

1. Introduction

NIH and other funders use software to classify award text and report portfolio totals, while prior research largely treats that text as written by applicants. This study traces how NIH’s 2025 text reviews affected category counts and finds that the count followed edited summaries even when projects retained funding.

1. Introduction

  • NIH’s 2025 guidance required removal of “DEI language” from awards not supporting DEI activities, directly governing text that its classification system also counts.A federal court vacated the directives in June 2025, and NIH later described renegotiated titles and abstracts as terms of award.Source: §1 · p. 2
  • Prior work leaves open how edits to award text affect NIH’s count, while external validation of its Research, Condition, and Disease Categorization system remains sparse and category-specific.Earlier work documented term declines and unequal terminations, but not what edits did to the funder’s count.Source: §1 · p. 2
  • 99.4% of continuing projects with unchanged summaries kept NIH’s minority-health category, versus 15.7% when summaries no longer named a racial or ethnic population; the projects remained funded.Reviewers judged 42 of 50 randomly sampled losses to have met the category definition in FY2024, and found aims concerning the population recast in 49 of 70 name losses.Source: §1 · pp. 2–3
  • The study traces population names through 37,790 continuing awards, checks them against practice records, and finds that text features predict which names survive while practice records add little predictive information.Practice records include linked trial enrollment, article indexing, and trial eligibility.Source: §1 · p. 3
  • When the population appeared to be the target, awards naming it were terminated far more often; the authors call this boundary population targeting and the resulting funded-but-unrecorded work uncounted science.The edited record cannot establish whether the research itself changed.Source: §1 · p. 3
  • The paper argues that changing an indicator’s text input can make the indicator record the edit as a change in science, while text-only checks cannot detect the resulting divergence from independent records.Following the same projects across text versions and comparing categories with records made for other purposes can show when and how divergence arose.Source: §1 · p. 3

2. When the funder edits the text it counts from

The section frames NIH’s count as a record built from text that both describes and can be changed independently of research practice. It develops word targeting as a mechanism, states its boundary, and sets out the 2025 review, vocabulary, data, and term definitions used to examine it.

2.1 A record edited while the funding continued

  • NIH’s category is mechanically assigned from text, so editing that text can change the count even when awards retain their funding.Practice records are also documentary accounts, but are produced under other rules and only exist for projects with trials or indexed articles.Source: §2.1 · pp. 4–5

2.2 Word targeting

  • Word targeting is editing that selects what to remove by the text; it predicts removal will follow sentence, name-form, and population features more than whether a project studies the population.When era vocabulary and informative population names share sentences, edits aimed at the vocabulary can remove names too; the count then inherits those edits.Source: §2.2 · p. 5

3.1 The 2025 reviews

  • The 2025 NIH guidance sorted awards by DEI’s role in a project and required removal of “DEI language” from awards not supporting DEI activities; subsequent legal and administrative actions shaped the review’s timeline.The later 235-term screen postdated the edits studied and, according to a complaint, was built from words in grants terminated in 2025.Source: §3.1 · pp. 5–6

3.2 The vocabulary the review would act on

  • Screen-term prevalence in new awards rose from 6.4% in FY2015 to 15.8% in FY2024, and by 2024 population names used in study design shared sentences with era vocabulary.The rise of screen-specific stems within investigators and across agencies suggests they marked a period of writing more than a kind of research.Source: §3.2 · p. 6

3.3 Data

  • The analysis links NIH award files to articles and trials, using curated MeSH only for articles published in 2021 or earlier because indexing became automated in 2022.Trial snapshots supply status, eligibility, registrations, and posted baseline counts; award text and spending categories come from NIH ExPORTER.Source: §3.3 · p. 6

3.4 Terms

  • The study uses REMHR as its outcome, defines practice records as enrollment, curated MeSH, and trial eligibility, and distinguishes six signaled from eight comparison populations.Its Senate list contains 699 terms; the 235-term screen list is used for describing the event, and revisions compare FY2024 with FY2025 summaries.Source: §3.3 · pp. 6–7, §3.4 · p. 7

4. Methods

The methods link continuing NIH awards across FY2024 and FY2025, track population mentions in summaries, and compare those changes with practice records and category retention. Separate definitions specify the study standard, reviewer process, and termination analysis.

4. Methods

  • 37,790 continuing award pairs with text in both years form the mention analysis, where a population name survives if it appears anywhere in the FY2025 summary.A mention is collateral to a listed term when both occur in one sentence; deleting that sentence counts as collateral deletion.Source: §4 · p. 7
  • Strict practice evidence marks a project as studying a population when qualifying curated articles, trial eligibility, or specified trial enrollment records show its involvement.This standard captures involvement rather than REMHR’s purpose-based definition, and projects below the threshold may still study the population.Source: §4 · p. 7
  • Three reviewers assessed projects, with two taking no part in data analysis; review packets omitted project identifiers and category status.Reviewer 1 designed the study, and all reviewers knew its aims.Source: §4 · p. 8
  • Retention measures the share of FY2024 REMHR projects with strict evidence that kept the category in FY2025, grouped by summary changes.Termination analyses cover FY2024 awards active after January 20, 2025, and adjusted differences hold listed-term count, institute, and activity fixed.Source: §4 · p. 8
  • The study uses Wilson intervals for proportions and a Newcombe interval for retention differences; other intervals resample projects or institutions unless specified otherwise.Definitions, models, and sensitivity analyses appear in Supplementary Methods 1–9.Source: §4 · p. 8

5. Results

The results assess how population names and summary wording relate to NIH’s minority-health classification and research records in still-funded projects and trials. Category retention tracked name retention, while projects that dropped a population name registered more new trials overall, new registrations naming the population barely differed, and estimates for trial endings were uncertain.

5.1 Four in ten flagged summaries lost every screen term in one year

  • 2,169 of 5,461 flagged FY2024 awards with FY2025 continuation abstracts (39.7%) contained no screen term, versus 4 of 5,242 (0.08%) a year earlier.Loss rates rose from 27.8% for continuations issued on time to 54.6% for those issued more than 90 days after budget start, and were higher when terms were used in the listed sense than incidentally (40.6% versus 2.4%).Source: §5.1 · p. 8

5.2 Population names recorded whom a study enrolled

  • 60 percentage points more Black participants were enrolled in trials linked to new awards whose titles named Black populations than in awards not naming them.The enrollment analysis covered awards with linked trials posting baselines, 2% of new awards; naming Black populations outside titles was associated with a 30-point lift.Source: §5.2 · pp. 8–9, Figure 2 · p. 10

5.3 Text features predicted which names survived

  • Among revised continuations, signaled-population names survived less often when every mention shared a sentence with a listed term (22.8% Hispanic, 24.6% Black, 10.4% SGM) than when one stood apart (43.9%, 43.9%, 34.7%).Source: §5.3 · p. 9

5.4 Funded projects lost the category when their summaries lost the name

  • 42 of 50 sampled additional category losses had an FY2024 purpose meeting REMHR’s definition, while 9 of 50 showed 2025–2026 research meeting it.The reviewers found 8 of those 9 cases involved work done before 2025, and the ninth gave no dates; the authors noted that the low share of current evidence also reflects how few records projects leave.Source: §5.4 · p. 12
  • 15.7% of continuing projects whose summaries no longer named a racial or ethnic population kept REMHR, versus 99.4% with unchanged summaries, although the projects remained funded.Among projects that lost REMHR after the name was removed, 127 retained another equity category and 110 retained none; those 110 had $59.5 million in FY2025 funding. Reviewers found an FY2024 purpose meeting the definition in 42 of 50 sampled losses, and judged aims recast in 49 of 70 name-loss summaries.Source: §5.4 · pp. 12–13, Figure 4 · p. 15

5.5 Awards naming sexual and gender minorities were terminated more often, after adjustment for listed terms

  • 37.8 percentage points more SGM-naming awards were recorded as terminated after adjustment for listed terms, institute and activity.The excess persisted within HIV/AIDS-category awards (+31.8 points) and when excluding institutions subject to institution-wide actions; the authors interpreted it as population targeting.Source: §5.5 · p. 14, Figure 5 · p. 16

5.6 Summaries changed more than the practice records we can follow

  • 46.7% of 150 pairs that lost every screen term were judged to have research content stated in the earlier summary that was lacking or redefined in the later one.The coding could not distinguish removing a population name from changes to aims, outcomes or interventions. In linked trials, 15.6% whose summaries dropped the name no longer named the population in eligibility criteria, versus 4.8% when summaries were unchanged (difference 10.8 points; 95% CI −0.5 to 24.7).Source: §5.6 · pp. 16–17
  • 0.51 versus 0.37 new trials per project were registered for projects that dropped a name versus unchanged projects, a difference of 0.14.The 95% interval for the difference was 0.01–0.27.Source: §5.6 · p. 17
  • New registrations naming the population barely differed; trial endings differed by 2.0 points, and same-population eligibility changed more often on small numbers.The trial-ending interval (−1.4 to 5.8) does not exclude more than doubling the 2.5% rate for unchanged summaries.Source: §5.6 · p. 17

6. Discussion

The discussion argues that review-driven edits changed NIH’s minority-health count even as projects stayed funded. It examines how those edits shaped the record, what later reviewers and practice records can show, and what remains uncertain.

An edited record

  • 49 of 70 reviewed name losses recast population-related aims, but paired summaries cannot establish whether the research itself changed.The authors call research that remains funded and evident in other records but is no longer counted “uncounted science.”Source: An edited record · p. 17

What this adds to research evaluation

From one review to the next

  • By September 30, 2026, 3 of 200 projects whose summaries lost the name named the population again, versus 444 of 496 projects with unchanged summaries.The article notes a possible feedback route: words from one review may inform a later screen, but the FY2026 categories were not yet published.Source: From one review to the next · p. 20

Population targeting in 2003 and 2025

  • Awards naming sexual and gender minorities were recorded as terminated 37.8 points more often after adjustment, with listed and unlisted forms lost at similar rates.In 2003, investigators also reported removing “red flag” words from later titles and abstracts, though NIH rescinded none of the grants it reviewed.Source: Population targeting in 2003 and 2025 · pp. 20–21

What the record can no longer show

  • Projects that lost the category after a name was removed carried $125.0 of the $553.7 million net fall, while the public record retains neither prior text nor edit dates.The authors say a reader of the record alone cannot distinguish a research change from a change in its description.Source: What the record can no longer show · p. 21

Limitations and future research

  • The test is imprecise: the text model’s AUC was 0.69, the strict cell rested on 20 projects, and the short observation window cannot show whether reviews changed research.Enrollment evidence came from the 2% of awards with trials posting baselines.Source: Limitations and future research · p. 21

7. Conclusion

NIH’s 2025 reviews changed award-summary wording, and the annual list dropped categories from some projects that kept their funding. The paper argues that when a government polices text it counts, its record can report edits as changes in science.

7. Conclusion

  • The annual list dropped a population’s category from some continuing projects even though they kept funding.Source: §7 · p. 22
  • In most reviewed losses, edits recast stated aims as well as population names, and the new summaries alone no longer met the category definition.Source: §7 · p. 22
  • For a few projects, active trial listings were evidence that work on the population continued, but the edited record cannot say whether research changed.Source: §7 · p. 22

Data and code availability

The authors list public data inputs and reproducibility materials provided to editors and reviewers, but award-level files and individual-award lists are not publicly released.

Data and code availability

Funding

The supplementary information maps methods, analyses, figures, and tables, and defines key terms used in the article.

Funding

  • Supplementary Methods 1–9 provide full methods and supporting results, followed by Supplementary Figures 1–4 and Tables 1–26.Source: Supplementary Information · p. 28
  • The supplement defines the summary, practice records, population-study standard, and a project’s count as its yearly REMHR assignment.Source: Supplementary Information · p. 28
  • The supplement’s analysis index includes text-predicted name survival, category changes after edits, and terminations by population named.Source: Supplementary Information · p. 28

Supplementary Methods 1.

The methods link NIH award records across fiscal years to publications and trials, define population and term lists, and compare population names with practice-record measures.

Data, samples and time ordering

  • Continuing-award pairs link each project’s latest eligible FY2024 entry to its FY2025 non-competing entry, with mention analyses requiring text in both years.FY2025 project files were regenerated in 2026, so edit dates are unknown.Source: Data, samples and time ordering · p. 28

Word lists, populations and forms

  • The analysis uses Senate-listed terms, six signaled populations, eight comparison populations, and distinct forms of Hispanic and SGM names.SGM descriptors include earlier indexing; Hispanic and SGM forms on and off the list are compared within awards using both.Source: Word lists, populations and forms · p. 29

Population names and practice records

  • For new awards with linked trials and posted baselines, the enrolled-share lift compares awards naming a population with awards not naming it, by name placement.For Black participants, enrollment lifts excluding trials whose eligibility named the population were 47 points and 23 points; the cohort test of declining informativeness was null.Source: Population names and practice records · p. 29

Supplementary Methods 4.

These methods compare how population mentions survive edits and assess practice evidence, NIH category retention, and reviewers’ judgments of changed summaries.

What the edits removed

  • Sentence matching treats a FY2024 sentence as gone if no FY2025 sentence reaches token Jaccard similarity 0.5; otherwise it tracks whether the name remains.Human reviewers checked a stratified sample of 100 sentences, agreeing on fate for 94 (κ = 0.89).Source: What the edits removed · p. 30
  • A sentence-level model estimates listed-term association with name loss while holding award × population fixed and accounting for aims or methods and sentence length.The analysis also examines mention counts and name survival; the comparison of Hispanic and SGM forms on and off the list is restricted to awards using both forms.Source: What the edits removed · p. 30
  • Practice evidence marks a population as studied through curated articles, trial eligibility, or specified Black and Hispanic trial enrollment, while capturing involvement rather than REMHR’s purpose criterion.The measured subset requires at least three curated-era articles or a linked trial.Source: What the edits removed · p. 30
  • The text-adjusted survival gap is an association with text features held fixed; weighting and misclassification sensitivity analyses address incomplete and imperfectly measured practice records.Full rewrites were too few to hold text features fixed; surgical edits had an adjusted gap of +7.2 points.Source: What the edits removed · pp. 30–31

The count

  • Blinded reviewers separately judged whether FY2024 projects met the REMHR definition and whether 2025–2026 evidence showed work on the population.For the random sample, reviewers agreed on 58 of 60 purpose labels and 52 of 60 current-purpose labels before arbitration.Source: The count · pp. 32–33
  • After arbitration, 42 of 50 sampled losses met the definition in FY2024 (84%; 95% CI 71–92), while the remaining 8 involved the group only.Among 17 projects that lost REMHR, both reviewers verified 2025–2026 work for 10; verification did not establish unchanged scope.Source: The count · pp. 32, 34

Terminations

The methods define termination and practice-record comparisons, then test whether the reported contrasts persist under alternate term lists and analysis choices.

Terminations

  • Termination is defined from Grant Witness status and date records, with adjusted differences estimated using institute and activity effects and listed-term counts.The analysis also considers HHS-reported terminations and excludes institution-wide actions and non-training awards in specified checks.Source: Terminations · p. 34

Sensitivity analyses

  • The SGM termination excess remained between +31.8 and +41.2 points across several category, term-count, and HIV/AIDS-category checks.Replacing the Senate list with screen terms left the SGM excess at +32.3 points and removed the small Hispanic excess (−0.4 points).Source: Terminations · p. 34, Sensitivity analyses · p. 35

Practice records and stated research

  • For award–population pairs, linked trials are checked for endings or eligibility changes, and reviewers assess whether paired texts lack or redefine stated research content.Reviewers both judged 70 of 150 pairs to lack or redefine content stated in the earlier text; they also judged 66 of those 70 to state content the other lacked.Source: Practice records and stated research · pp. 34–35

ity

  • The analysis reports each comparison’s population, exposure, outcome, and contrast, with Wilson or bootstrap intervals as specified.Bootstrap draws resample projects or, for termination analyses, institutions.Source: ity · p. 35

Supplementary Figures

The supplementary figures display term-associated mention loss, name survival by mention count, text-adjusted survival gaps, and prediction-model AUCs.

Supplementary Figures

  • Supplementary Figure 1 compares term-associated mention loss within award × population for workforce-adjacent and other comparison populations.Bootstrap intervals resample awards; none are shown for signaled populations.Source: Figure 1 · p. 36
  • Supplementary Figure 2 plots signaled and comparison name survival by FY2024 mention count, contrasting observed survival with independent-deletion predictions.Observed values are solid and predicted values dashed.Source: Figure 2 · p. 36
  • Supplementary Figure 3 compares studied-minus-not-studied survival gaps, showing raw and text-held-fixed estimates for surgical edits and raw estimates for full rewrites.Intervals resample projects.Source: Figure 3 · p. 37
  • Supplementary Figure 4 displays out-of-fold AUC for name survival by model, with 95% intervals resampling projects.Source: Figure 4 · p. 37

Supplementary Tables

The supplementary material reports name-retention and trial-change results, reviewer agreement, and checks against the study’s practice-record standard. It also documents that this standard is not RCDC classification accuracy.

  • Reviewers agreed on 58 of 60 purpose labels (κ = 0.84) and 52 of 60 current-purpose labels (κ = 0.77) before arbitration.Source: Supplementary Tables · p. 40
  • 66 of 67 unchanged projects, 35 of 38 revised projects still naming a population, and 3 of 20 revised projects no longer naming one remained on NIH’s REMHR list.Source: Supplementary Tables · p. 41
  • The cohort trend in name informativeness was null for signaled and comparison populations alike.Source: Supplementary Tables · p. 45
  • Practice-record comparisons describe agreement with the study’s standard, not RCDC classification accuracy; curated MeSH and trial records only approximate practice.Source: Supplementary Tables · pp. 46, 48
Loading 2610.03443v1…