PaperlayerView exact revision on arXiv

Source-linked AI summary

The Diversity-Innovation Paradox in Science

Bas Hofstra, Vivek V. Kulkarni, Sebastian Munoz-Najar Galvez, Bryan He, Dan Jurafsky, Daniel A. McFarland

arXiv:1909.02063v2cs.SIcs.CLstat.APstat.ML

TL;DR

Whether the diversity-innovation paradox also holds among scientists is the question this paper examines. Analyzing doctoral dissertations and comparing scholarly novelty and career outcomes, the authors find higher novelty among underrepresented groups, but lower uptake and career returns for their novel contributions.

  • Problem

    Whether the diversity-innovation paradox also holds among scientists is a central question the paper sets out to examine.

    Source: Introduction · p. 2

  • Method

    Using 1,208,246 dissertations, the study compares minority scholars’ scientific novelty across institutions.

    Source: Introduction · p. 3, Data · p. 12

  • Results

    Underrepresented groups show higher novelty rates, but their novel contributions receive less uptake and yield lower career returns for gender and racial minorities.

    Source: Discussion · p. 11

  • Takeaways & Limitations

    The findings suggest academic careers may reproduce stratification by discounting diversity’s role in scientific innovation.

    Source: Discussion · p. 11

    The analysis assumes dissertation abstracts approximate full texts, while the authors note theoretical difficulties with using full texts to study concept co-occurrence.

    Source: On Analyzing Abstracts Versus Full Texts · pp. 37–38

Abstract

from arXiv · show

Prior work finds a diversity paradox: diversity breeds innovation, and yet, underrepresented groups that diversify organizations have less successful careers within them. Does the diversity paradox hold for scientists as well? We study this by utilizing a near-population of ~1.2 million US doctoral recipients from 1977-2015 and following their careers into publishing and faculty positions. We use text analysis and machine learning to answer a series of questions: How do we detect scientific innovations? Are underrepresented groups more likely to generate scientific innovations? And are the innovations of underrepresented groups adopted and rewarded? Our analyses show that underrepresented groups produce higher rates of scientific novelty. However, their novel contributions are devalued and discounted: e.g., novel contributions by gender and racial minorities are taken up by other scholars at lower rates than novel contributions by gender and racial majorities, and equally impactful contributions of gender and racial minorities are less likely to result in successful scientific careers than for majority groups. These results suggest there may be unwarranted reproduction of stratification in academic careers that discounts diversity's role in innovation and partly explains the underrepresentation of some groups in academia.

Introduction · Innovation as Novelty and Impactful Novelty in Text

The diversity–innovation paradox is that underrepresented scholars are expected to generate innovation and achieve career success, yet persistent inequalities may reflect the discounting of their innovations. Using a near-complete population of US doctorate recipients, the study measures conceptual novelty, its uptake, and career returns.

Introduction

  • Underrepresented scholars are expected to innovate and gain career success, yet persistent group inequalities may reflect the discounting of their innovations.The paper frames this tension as the diversity–innovation paradox.Source: Introduction · p. 2
  • The study examines novelty, uptake by other scholars, and career returns across a near-complete population of about 1.2 million US doctorate recipients.The analysis spans three decades, scientific disciplines, and US doctorate-awarding institutions.Source: Introduction · pp. 2–3

Innovation as Novelty and Impactful Novelty in Text

  • ProQuest dissertation records from 1977–2015 provide thesis text and metadata for measuring innovation at the start of scholarly careers and linking it to later trajectories.The records include titles, abstracts, disciplines, student names, advisors, and institutions.Source: Innovation as Novelty and Impactful… · p. 3
  • The study uses phrase extraction and structural topic modeling to identify thesis concepts, then counts concept pairs first related in a thesis as novel links.The number of new links measures the conceptual novelty of a thesis and its author.Source: Innovation as Novelty and Impactful… · p. 4
  • Impactful novelty is measured by the average future adoption of each new conceptual link, distinguishing the creation of novelty from its subsequent uptake.The text cautions that novelty need not be adopted to count as innovation, and adoption may itself reflect structural processes.Source: Innovation as Novelty and Impactful… · p. 4
  • Conceptual recombination offers an innovation measure that is insensitive to differences in journal indexing across disciplines and the many reasons scholars cite work.Source: Innovation as Novelty and Impactful… · pp. 4–5

Results

Underrepresented groups introduce more scientific novelty, but their contributions receive less uptake and yield lower career returns. More semantically distant novelty is also taken up less, partly explaining this discounting.

Results

  • Women and non-white scholars introduce more novelty but have less impactful novelty than men and white scholars; intersectional minority groups likewise have higher novelty and white men higher impactful novelty.For women and non-white scholars, both novelty differences have p < .001 and both impactful-novelty differences have p < .05; intersectional comparisons have p < .001 for novelty and p < .01 for impactful novelty.Source: Results · p. 6, Figure 2 · p. 7
  • Greater underrepresentation by gender or race in a discipline is associated with more novel linkages, while having more same-gender peers is associated with greater uptake of those linkages.The reported associations are significant for gender underrepresentation (p < .001), racial underrepresentation (p < .05), and same-gender peer representation (p < .01).Source: Results · p. 6, Figure 2 · p. 7
  • Underrepresented-gender students and women introduce more semantically distant novelty, and more distant new links receive less uptake, potentially explaining part of the discounting.The reported associations have p < .001.Source: Figure 3 · p. 8, Results · p. 8
  • Thesis novelty and impactful novelty are positively related to becoming a research faculty member or continuing researcher (all p < .001).These outcomes represent research faculty positions and continued research careers, respectively.Source: Results · p. 9
  • A 2SD increase in novelty (# new links) widens the gender-minority/majority faculty-probability gap from 3.5% to 9.5%; impactful novelty (uptake per new link) widens it from 4.3% to 15%.The widening persists over and above the distance between newly linked concepts.Source: Figure 4 · p. 10, Results · pp. 10–11

Discussion … Concept extraction from scientific text

Underrepresented scholars generate more scientific novelty, but their contributions receive less uptake and yield smaller career returns; they must innovate more for similar career success.

Discussion

  • Across demographic diversity measures, underrepresented groups produce more novelty (2-7), but women and non-white scholars’ novel contributions receive less uptake than those of dominant groups.The discussion describes this as a diversity-innovation paradox: higher novelty does not ensure equal adoption.Source: Discussion · p. 11
  • For gender minorities, lower uptake is partly explained by more distal linkages, which may be harder to integrate into prevailing field conversations.The discussion suggests that entering science from a new vantage can generate connections that are difficult to integrate into localized conversations.Source: Discussion · p. 11
  • Novelty and impactful novelty correspond with career success, but returns are lower for gender and racial minorities, especially for highly impactful contributions and beyond their distalness (8-13).At low impactful novelty, minorities and majorities are often rewarded similarly; the career discount grows for highly impactful novelty and holds over and above contribution distance.Source: Discussion · p. 11
  • Underrepresented groups must innovate more for similar career success, and the authors suggest their careers end prematurely despite their role in innovation, underscoring bias in hiring, evaluation, and publication.The paper presents a system-level account of how innovation differentially affects demographic groups’ scientific careers.Source: Discussion · p. 12

Data

  • The dataset contains 1,208,246 dissertations filed at U.S. doctorate-awarding universities from 1977–2015, covering about 86% of U.S. doctorates across disciplines.Metadata include candidate, year, university, thesis abstract, and primary advisor; the study follows recipients into subsequent academic and research careers.Source: Data · p. 12

Concept extraction from scientific text

  • Innovation is operationalized as combining topical-lexicon terms; Structural Topic Models identify latent themes, and FREX ranks terms using weighted frequency and exclusivity.FREX balances terms’ commonness and distinctiveness within a topic.Source: Concept extraction from scientific text · p. 13
  • Models fit at K=50–1000 plateaued at K=400, 500, and 600; sensitivity analyses across nine K and FREX settings yielded robust results.The nine scenarios combine three K values with three frequency-exclusivity weighting schemes.Source: Concept extraction from scientific text · p. 13
  • Main-text analyses use the 50/50 frequency-exclusivity scheme at K=500; inference uses 1982–2010 theses to allow five-year accumulation and uptake windows, with year fixed effects.All doctoral abstracts from 1977–2015 provide the semantic signal; the inferential window gives the latest graduates five years for their novelty to be taken up.Source: Concept extraction from scientific text · pp. 13–14

Outcome variable–Novelty and impactful novelty · Outcome variable–Distal novelty

The section operationalizes thesis novelty as meaningful new concept links, measures their uptake and semantic distance, and validates the distinction between distal and proximal links with expert coding. It also relates these measures to publication productivity and citations among publishing students.

Outcome variable–Novelty and impactful novelty

Outcome variable–Distal novelty

  • Mean = .426 distal novelty (Median = .419; SD = .118) averages cosine distances between newly linked concepts, distinguishing links across semantic clusters from proximal links.Concepts are embedded in a global co-occurrence space to estimate their semantic distance.Source: Outcome variable–Distal novelty · pp. 15–16
  • Average Cohen’s Kappa = .46 across three coders, whose assignments predict ~95% of true distal links, validating the automated distance measure against expert judgments.The validation used a random set of 100 links; true distal links had distance scores > .8.Source: Outcome variable–Distal novelty · p. 16
  • Distal links often connect concepts from different fields or form creative metaphors, while only 15–20% of links between distal concepts are hard to interpret substantively.Source: Outcome variable–Distal novelty · p. 16

Outcome variable–Careers · Main covariates

The study measures career outcomes using both a conservative research-faculty proxy and a broader indicator of continued research. It estimates gender and race from names and constructs discipline-year representation measures as covariates.

Outcome variable–Careers

  • Research faculty status (mean = .066) is defined conservatively as becoming a primary advisor of other PhD students at a PhD-granting US university.This captures transition from student to mentor with a lineage of students.Source: Outcome variable–Careers · p. 16
  • Continued research (mean = 0.319) is measured by publishing academically at least once within five years of the PhD or becoming research faculty.The ProQuest–Web of Science linkage follows publication output and includes researchers at non-PhD-granting institutions or abroad.Source: Outcome variable–Careers · pp. 16–17

Main covariates

  • Because ProQuest lacks direct gender and race reports, the study estimates these characteristics from names using census and Social Security data and a threshold algorithm trained on 20,264 scholars.It classifies advisees into White, Asian, and underrepresented racial minority categories and assigns gender.Source: Main covariates · p. 17
  • Race categories are aggregated into underrepresented racial minority because small proportions of finer-grained groups provide little statistical power.This category combines Hispanics, African Americans, Native Americans, and racial categories not captured by White or Asian.Source: Main covariates · pp. 18–19
  • The covariates include same-gender and same-race shares within a discipline-year (means = .576 and .625) and whether a student belongs to a gender or racial minority there.Gender minority mean = .336; racial minority mean = .246.Source: Main covariates · p. 18
  • Models of novelty use same-gender/race shares and White/non-White status, while career models use minority status within disciplines.Results are similar when career models use binary gender or race indicators.Source: Main covariates · p. 18
  • The analyses exclude unclassified gender and race cases, but conclusions are robust when including unknowns and when limiting the sample to names strongly associated with one race.Including unknowns does not qualitatively change the gender and race distinctions shown in Figure 2C and F.Source: Main covariates · p. 18
  • Name-based gender and race are coarse signals of how others may perceive individuals, not direct measures of self-identification.The authors note that names and individuals can carry varying degrees of gender and racial association.Source: Main covariates · p. 19

Confounding Factors … Diversity and Innovation

The section proposes that underrepresented groups’ outsider perspectives may help them form novel connections, and describes the study’s modeling, controls, and data resources. It also documents funding and supplementary materials.

Confounding Factors

  • The dissertation discipline classifier achieved 96% precision (NDISCIPLINE = 84); students were assigned to their first ProQuest-filing institution, and dissertation filing years ranged from 1977 to 2015.Source: Confounding Factors · p. 19

Analytical strategy

  • The study models novelty and impactful novelty with negative binomial regression, distal novelty with linear regression, and binary career outcomes with logistic regression, controlling for institution, discipline, and graduation year.These fixed effects account for differences in university prestige and resources, disciplinary cultures, and scholars’ time to make career transitions or gain recognition.Source: Analytical strategy · pp. 19–20
  • Models use institution-year doctorate weights to address ProQuest filing selectivity and generalize to the US PhD population; sample sizes are 808,375 for novelty, 345,257 for impactful novelty, and 630,971 for distal novelty.Impactful novelty is modeled among students with nonzero novelty and uptake, while distal novelty is modeled among students with nonzero novelty.Source: Analytical strategy · p. 20

Funding

  • The study was supported by National Science Foundation grants NSF #1633036 and NSF #1827477, and a Dutch Organization for Scientific Research grant NWO.The funding passage gives no NWO grant number.Source: Funding · p. 21

Data and materials availability

  • The study received protocol approval from Stanford, obtained ProQuest permission to analyze dissertation data, and provides its key-metrics code on GitHub.The dissertation corpus and Web of Science data can be requested from ProQuest and Clarivate Analytics, respectively.Source: Data and materials availability · p. 22

Supplementary Information for

  • The supplementary materials include text on innovation measurement, structural topic models, the PMI score, student gender and race, population coverage and weights, inferential models, and database linkage.The supplementary-information contents also list academic discipline.Source: Supplementary Information for · p. 27

Diversity and Innovation

  • Underrepresented groups’ outsider perspectives may help them notice overlooked relations and form novel connections, while inclusion adds variety to perspectives brought to scientific research.The passage frames this as a rationale for why underrepresented groups may contribute novelty, not as a reported empirical result.Source: Diversity and Innovation · p. 28

Measuring Innovation Through Citations, Keywords, and Text · Structural Topic Models for Concept Extraction · Structural Topic Models

The paper identifies scientific novelty through novel combinations of concepts in dissertation text, extending prior citation- and keyword-based approaches. It uses Structural Topic Models (STMs) to extract concepts and assess their use across a near-population corpus of dissertations.

Measuring Innovation Through Citations, Keywords, and Text

  • Novel combinations of concepts in scientific text provide the paper’s innovation measure, extending work that treats novel bibliographic recombinations as innovative.The authors argue that text-based concept combinations preserve explicit meaning and avoid limitations of citations and keywords.Source: Measuring Innovation Through… · pp. 28–29
  • Compared with citation- or keyword-based approaches, dissertation-language measures are less affected by indexing, journal guidelines, and varied reasons for citing work.A prior study used chemical entities in MEDLINE abstracts to avoid classification confounding, but that approach was specific to chemistry and prompted calls for finer-grained building blocks (7).Source: Measuring Innovation Through… · p. 29
  • The near-population dissertation corpus represents book-, publication-, and proceedings-oriented sciences, though concepts identified in dissertations may appear earlier in other corpora.The corpus provides insight into which dissertations are novel relative to others and which students show early innovative sparks.Source: Measuring Innovation Through… · pp. 29–30

Structural Topic Models

  • STMs model topic prevalence in approximately 1.2 million dissertation abstracts as a linear function of doctorate year, representing texts as mixtures of latent themes.The topics are weighted sets of co-occurring words, and each document receives a topic-mixture representation.Source: Structural Topic Models · p. 30
  • Including publication metadata in topic prevalence had little impact on the β distributions used for concept extraction, and STMs produced very similar β distributions to simpler LDA models.The authors retained STMs because they add information without reducing topic quality or interpretability; directly modeling β by publication year was computationally intractable.Source: Structural Topic Models · p. 31
  • Internal and external validation indices place the optimal topic count at approximately K = 400–600, with the main-text analysis using K = 500.The selected STM’s vocabulary weights identify concepts describing latent corpus dimensions; concepts are extracted from each topic’s highest FREX-score terms.Source: Structural Topic Models · p. 32
  • Results were robust across concept-extraction specifications emphasizing frequency, exclusivity, or both, and across K values of 400, 500, and 600.The paper presents the robustness results in Table S1.Source: Structural Topic Models · p. 32

Preprocessing Texts and Fitting STMs · Internal validation · External validation

The authors preprocess dissertation texts and fit STM models across topic counts, then validate topic selection internally and against student-provided fields and keywords; external validation points to an optimum around 500 topics.

Preprocessing Texts and Fitting STMs

Internal validation

  • Internal validation assesses topic coherence and exclusivity together because coherence alone can be inflated by fewer topics or common words.Coherence measures co-occurrence among high-probability words; exclusivity measures how distinct a topic’s words are from those in other topics.Source: Internal validation · p. 33
  • Figures S1-A and S1-B indicate a likely internal topic-count limit where coherence and exclusivity plateau.The authors seek a K at which neither metric improves nor declines as the number of topics changes.Source: Internal validation · pp. 33–34

External validation

  • External validation compares STM topic-mixture similarities for a constant random sample of 1,000 documents with relations based on student-provided dissertation fields and keywords.Document pairs are related in the STM network using cosine similarity, while shared field-and-keyword bigrams define relations in the comparison network.Source: External validation · p. 34
  • External-validation correlations follow internal-metric trends and decline beyond K = 500, supporting an optimum around 500 topics against user-labeled information.The comparison uses STM-based document relations and relations derived from dissertation fields and keywords.Source: External validation · p. 35

Consistency · The “Right” K · Concept Extraction With FREX

Topic assignments stabilize around K = 400–600, and key findings are qualitatively robust across K values and FREX weighting schemes; the main text uses K = 500 with frequency and exclusivity equally balanced.

Consistency

  • The authors compare each document’s highest-proportion topic assignment across consecutive K models using the Fowlkes–Mallows index to track clustering overlap.The index is the geometric mean of precision and recall; higher values indicate greater agreement between partitions.Source: Consistency · pp. 35–36
  • More than two-thirds of document-to-topic assignments are stable from K = 400, with overlap stabilizing between K = 400 and K = 600.Raw Fowlkes–Mallows scores rise steadily and improve only gradually after that range.Source: Consistency · p. 36

The “Right” K

  • Key results stay mostly qualitatively similar at K = 400, 500, or 600, including concept/link introduction and uptake measured with low, medium, and high FREX weights.The metrics appear to plateau around K = 500, which is used in the main text.Source: The “Right” K · p. 36

Concept Extraction With FREX

  • FREX scores combine a topic term’s weighted frequency and exclusivity to identify concepts that balance commonness with distinctiveness.The authors examine equal frequency/exclusivity weighting (50/50), frequency-heavy weighting (75/25), and exclusivity-heavy weighting (25/75).Source: Concept Extraction With FREX · p. 37
  • Sensitivity analyses find robust results for novelty, impactful novelty, and recognition across nine scenarios combining K = 400–600 with three FREX weightings.The scenarios use the top 500 FREX words per topic; the main text reports equal frequency and exclusivity weighting at K = 500.Source: Concept Extraction With FREX · p. 37

On Analyzing Abstracts Versus Full Texts · The PMI Score to Identify Meaningful Links · Student Gender and Race

The study uses dissertation abstracts to identify meaningful concept links and estimates students’ gender and race from names. These methodological choices rely on abstracts as useful summaries and are checked against available validation data.

On Analyzing Abstracts Versus Full Texts

  • The analysis uses abstracts because full texts are difficult to access and scale, while abstracts are easier to obtain and computationally cheaper.The authors conjecture that abstracts approximate the knowledge and concepts in full texts.Source: On Analyzing Abstracts Versus Full Texts · pp. 37–38
  • The authors assume that abstracts retain innovations no more or less for numerical minorities than for majorities.Because abstracts are about 10 sentences, concept co-occurrences are more likely to be substantively meaningful, while full-text analysis raises questions about the relevant distance between concepts.Source: On Analyzing Abstracts Versus Full Texts · p. 38

The PMI Score to Identify Meaningful Links

  • High PMI scores identify concept links that occur more often than chance, and a rank-based cutoff filters spurious recombinations.PMI compares the likelihood of a link occurring with the likelihoods of its individual concepts.Source: The PMI Score to Identify Meaningful… · p. 38
  • The PMI analysis retains links whose individual terms appear in at least 10 theses and considers the top 10 million links.The cutoff is intended to remove obviously meaningless links while preserving opportunities to identify novelty.Source: The PMI Score to Identify Meaningful… · pp. 38–39

Student Gender and Race

  • Gender and race are inferred from first and last names, respectively, using Social Security Administration first-name data and Census last-name data.The corpus lacks student gender and race records; the Census data cover 167,409 distinct last names representing over 95% of the US population.Source: Student Gender and Race · p. 39
  • The study reports a gender correlation of r = .91 and estimates that approximately 8.5% of cases have unknown genders.Source: Student Gender and Race · p. 40
  • Race classification accuracy varies substantially: it is 97.2% for white, 93.4% for Asian, 70.4% for Hispanic, and 9.9% for African and Native American cases.The authors report that many African and Native American cases were labeled white and supplement the name-based classifications with a second method using full names.Source: Student Gender and Race · p. 40
  • The supplemental method has precision of .83 for Hispanic and .74 for African American names, and results using a smaller, higher-precision race dataset are qualitatively similar.About 10.8% of cases remain unknown for race.Source: Student Gender and Race · p. 41

Academic Discipline · Population Coverage and Data Weights · Inferential Models

The methods infer missing academic disciplines, weight ProQuest data to account for its coverage of US doctorates, and model novelty, uptake, and career outcomes.

Academic Discipline

  • The discipline classifier uses dissertation subject categories, keywords, abstract topics, Word2Vec vectors, and the degree-granting university as features.Its features include NRC and ProQuest fields, a 100-topic LDA distribution, and average word vectors for keywords, fields, and title.Source: Academic Discipline · p. 42
  • 96% precision (NDISCIPLINE = 84) was achieved when inferring dissertation departments.Source: Academic Discipline · p. 42

Population Coverage and Data Weights

  • 86% of US doctorates were covered by ProQuest across three decades, against approximately 1.2 million doctorates awarded during 1977–2015.Source: Population Coverage and Data Weights · p. 42
  • For 1982–2010, survey weights scaled ProQuest university-year shares of PhD recipients to the corresponding National Science Foundation shares.The weighting accounts for possible selectivity in when universities filed theses in ProQuest.Source: Population Coverage and Data Weights · pp. 42–43

Inferential Models

  • Individual-level models estimate novelty or uptake counts, average distality of introduced links, and career success as faculty research or sustained research careers.Predictors include gender and race representation, with institution, discipline, and year as confounders.Source: Inferential Models · p. 43
  • For uptake per new link, negative binomial regression uses logged new-link counts as an offset to model a non-integer rate (35, 36).Source: Inferential Models · p. 44

Linking ProQuest to Web of Science · Supplementary References

The supplement describes a conservative linkage pipeline connecting ProQuest dissertations to Web of Science authors, then reports checks on topic models, novelty measures, and related analyses. It also documents the scope and validation of the supplementary measures.

Linking ProQuest to Web of Science

  • Because pre-2009 Web of Science data lacked high-precision canonical author identifiers, the authors linked its disambiguated clusters to Clarivate’s substantially accurate post-2009 disambiguated authors, including by shared full name and email address.Source: Linking ProQuest to Web of Science · pp. 44–45
  • 97% precision was inferred for links between pre- and post-2009 Web of Science author clusters using scholars’ self-labeled publications.Clarivate supplied the self-labeled publication set used to infer precision.Source: Linking ProQuest to Web of Science · p. 45
  • ProQuest dissertation-to-author matches were selected through a multi-step sieve using advisor links, institutional overlap, keyword matches, name similarity, and abstract or title similarity.The process used maximum bipartite matching so neither author could be linked to more than one author on the other side.Source: Linking ProQuest to Web of Science · p. 45
  • To reduce false matches, the pipeline excluded publication histories far from graduation, mostly pre-graduation publishing without advisor evidence, and weak matches based only on names and text.It excluded nearest publication dates more than 15 years after or 10 years before graduation.Source: Linking ProQuest to Web of Science · p. 46
  • Across K = 400–600, topic-model exclusivity rose and plateaued near K = 400, coherence declined, and external-correlation results stabilized after K = 400.Robustness analyses at K = 450 yielded the same results as K = 400.Source: Figure S1 · p. 47
  • Novelty and impactful-novelty distributions were qualitatively similar across K and FREX scenarios, with both measures’ stable time series beginning around 1982.Sensitivity analyses likewise showed mostly similar result patterns, with some variable results at lower K or very high exclusivity.Source: Figure S2 · p. 48, Table S2 · p. 52
  • The supplement illustrates topic concepts across fields, finds nearby embedding concepts semantically similar, and shows women and non-white students are usually numerical minorities across fields.Women became numerical majorities in a few field-years, while non-white students became racial majorities in very few cases.Source: Figure S3 · p. 49, Figure S4 · p. 50, Table S4 · p. 54
  • Novelty and impactful novelty were positively related to publication counts and accumulated citations in models controlling for discipline, graduation year, and PhD university.Source: Table S3 · p. 53

Related papers

Loading 1909.02063v2…