PaperlayerView exact revision on arXiv

Source-linked AI summary

The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research

Nur Ahmed, Muntasir Wahed

arXiv:2010.15581v1cs.CYcs.LG

TL;DR

AI research is increasingly viewed as less democratized amid AI’s pervasive and consequential deployment. The paper constructs a large paper dataset and applies generalized synthetic control to study this concern, finding that firms—especially large technology firms—increasingly contribute to AI research relative to other computer science areas.

  • Problem

    AI research is becoming less democratized as AI is increasingly deployed in consequential domains.

  • Method

    The paper constructs a novel dataset of 171,394 papers and uses generalized synthetic control to investigate these research-participation patterns.

  • Results

    Firms, particularly large technology firms, increasingly contribute to AI research relative to other computer science areas.

  • Takeaways & Limitations

    Large firms have an annual presence of 23 papers per conference.

  • Takeaways & Limitations

    Causal impact estimates require a strong parallel trend assumption.

Abstract

from arXiv · show

Increasingly, modern Artificial Intelligence (AI) research has become more computationally intensive. However, a growing concern is that due to unequal access to computing power, only certain firms and elite universities have advantages in modern AI research. Using a novel dataset of 171394 papers from 57 prestigious computer science conferences, we document that firms, in particular, large technology firms and elite universities have increased participation in major AI conferences since deep learning's unanticipated rise in 2012. The effect is concentrated among elite universities, which are ranked 1-50 in the QS World University Rankings. Further, we find two strategies through which firms increased their presence in AI research: first, they have increased firm-only publications; and second, firms are collaborating primarily with elite universities. Consequently, this increased presence of firms and elite universities in AI research has crowded out mid-tier (QS ranked 201-300) and lower-tier (QS ranked 301-500) universities. To provide causal evidence that deep learning's unanticipated rise resulted in this divergence, we leverage the generalized synthetic control method, a data-driven counterfactual estimator. Using machine learning based text analysis methods, we provide additional evidence that the divergence between these two groups - large firms and non-elite universities - is driven by access to computing power or compute, which we term as the "compute divide". This compute divide between large firms and non-elite universities increases concerns around bias and fairness within AI technology, and presents an obstacle towards "democratizing" AI. These results suggest that a lack of access to specialized equipment such as compute can de-democratize knowledge production.

Introduction

The paper examines whether modern AI research has become more concentrated because unequal access to compute advantages large firms and elite universities. Using extensive conference data and causal analysis, it finds divergence that crowds out less-elite universities and reflects a compute divide.

  • Motivation: AI research is increasingly concentrated because industry presence and unequal access to resources challenge the democratization of AI.The paper links this concern to modern AI’s computational intensity and the possibility that a small group shapes AI’s future.
  • Approach: The study analyzes 171,394 papers from 57 leading computer science conferences to test concentration and organizational participation since deep learning’s rise.The dataset covers AI and non-AI venues with broadly similar selectivity, prestige, and impact.
  • Findings: Global technology firms publish 44 additional papers annually per AI conference, while elite universities publish 40 additional papers relative to the counterfactual.Global technology firms average 23 papers per computer science conference, making the estimated increase substantial.
  • Findings: Mid-tier and lower-tier universities publish 14 and 5 fewer papers, respectively, while HBCUs and Hispanic-serving institutions are underrepresented in top AI venues.The affected university groups are QS ranked 201-300 and 301-500.
  • Mechanism: Firms increased AI participation through firm-only publications and collaboration primarily with elite universities, contributing to a compute divide with non-elite universities.Text analysis attributes part of the divergence to unequal access to computing power and finds firms publish more in deep learning than both elite and non-elite universities.
  • Implications: The findings suggest that modern AI knowledge production is concentrated among a small, less-diverse group, raising concerns for bias, fairness, and democratization.The paper argues that specialized equipment can produce “haves and have-nots” in a scientific field.

Relevant Literature

Modern AI research differs from the earlier general-purpose-computing era because compute usage and specialized hardware have accelerated sharply since 2012. This shift gives groups with specialized hardware a competitive advantage and makes research competition less equal.

  • Earlier era: Before 2012, researchers primarily used general-purpose hardware and competed mainly on algorithmic ideas under broadly similar computing conditions.Specialized-hardware investment was riskier because its payoff was uncertain.
  • Two computing eras: Since 2012, modern AI compute usage has doubled every 3.4 months rather than every two years under Moore’s law.The acceleration is associated with specialized hardware and unconventional processing units such as GPUs.
  • Modern era: Modern AI research uses specialized hardware, including GPUs, TPUs, and other dedicated systems, rather than relying primarily on general-purpose chips.Large firms began investing more in specialized hardware during the 2010s.
  • Competitive consequences: Researchers without access to specialized hardware face more difficult competition because compute can play an outsized role in determining which ideas perform better.The paper concludes that increased compute provides a competitive advantage to certain research groups.

The Role of Compute, Data, and Human Capital in Modern AI Research

Modern AI depends on specialized compute, proprietary data, and scarce human expertise, resources that large firms can more readily obtain than many universities. These resource differences help explain firms’ advantages and elite universities’ stronger position in AI research.

  • Human capital: Deep learning depends heavily on trained scientists because researchers still have limited understanding of why it works and the field has limited codification.This makes accumulated expertise important for producing new knowledge.
  • Human capital: Large firms can recruit and retain scarce AI talent, including faculty members from universities, because universities often cannot match industry compensation.The paper notes that large firms recruited 180 faculty members from North American universities between 2004 and 2018.
  • Compute and data: Large firms have advantages from access to compute and proprietary datasets, while elite and mid-tier universities lack comparable access to compute and large datasets.These resources are described as important ingredients for modern AI research and deep-learning training.
  • Compute and data: Specialized compute requires substantial investment in GPUs, cloud infrastructure, and related technologies, reinforcing the advantage of organizations able to make those investments.AlphaGo Zero’s training cost was estimated at $35 million, compared with a $90 million annual budget for Carnegie Mellon’s largest robotics lab.
  • Compute and data: Post-Moore computing benefits organizations that can design specialized chips and write software for them, helping large firms get ahead in AI competition.The paper identifies specialized chips and cloud computing as sources of advantage for large firms.

Research Equipment and the De-Democratization of Knowledge Production

The paper models knowledge production as depending on human and material inputs, arguing that deep learning increased the importance and cost of compute and data. It uses the generalized synthetic control method to examine whether this shift widened participation disparities.

  • Knowledge-production model: Knowledge production depends on knowledge, skills, materials, equipment, and effort.
  • Knowledge-production model: The model treats output as increasing or decreasing with changes in any input, while allowing organizations to obtain inputs through collaboration.A university may collaborate with a firm to use specialized equipment.
  • Research question: The study asks whether deep learning de-democratizes a scientific field, addressing the lack of empirical evidence on rising research costs and democratization.
  • De-democratization mechanism: Deep learning increased the importance of compute and data, thereby raising knowledge-production costs and barriers to entry.
  • De-democratization mechanism: Organizations with limited access to specialized equipment struggle to produce new knowledge, whereas organizations with access advance in knowledge production.
  • Predicted divergence: The paper argues that increased compute, data, and AI talent should lower knowledge production for non-elite universities but raise it for elite universities and large firms.
  • Empirical approach: The generalized synthetic control method estimates treatment effects by imputing post-intervention counterfactuals for treated AI conferences from control conferences.
  • Empirical approach: The method estimates the average treatment effect on treated conferences as the post-treatment mean difference between observed and counterfactual outcomes.

Identification Strategy

The identification strategy treats the unexpected 2012 ImageNet deep-learning breakthrough and GPU use as an exogenous shock. It compares AI conferences with non-AI computer science conferences using panel data and synthetic counterfactuals.

  • Identification: The ImageNet shock is used to estimate its causal impact on organizational participation in AI conferences.
  • ImageNet shock: The 2012 ImageNet result unexpectedly demonstrated that GPUs could unlock neural-network capabilities.
  • ImageNet shock: The study uses the unanticipated spread of GPU-based deep learning as an exogenous event correlated with increased compute availability.
  • Data: The dataset combines csrankings.org, Scopus, QS World University Rankings, US News rankings, and Fortune magazine data.
  • Data: The final sample contains 171,394 peer-reviewed articles from 57 conferences spanning 2000 to 2019.
  • Sample construction: The study excludes NAACL because it had fewer than six pre-2012 observations required for a reliable counterfactual.
  • Treatment and controls: AI conferences form the treated units, while non-AI conferences provide control units in the donor pool.
  • Treatment and controls: The control conferences are considered suitable because firms and universities may participate in both sets, and the conferences are similar in submissions and selectivity.

Variables

The variables measure conference-level publication output and organizational participation using affiliation classifications. The main explanatory variable marks whether a conference is an AI treatment unit after ImageNet 2012.

  • Outcome variable: The conference is the unit of analysis, and the primary dependent variable is the total number of papers associated with corresponding institutions.
  • Organizational classifications: Affiliations are classified using fuzzy string matching, regular expressions, and manual review of unclassified observations.
  • Measurement caveat: The authors note that affiliation classification can produce misclassification-related limitations.
  • Organizational classifications: Firm categories distinguish Fortune500 technology firms, Fortune500 non-technology firms, non-Fortune500 firms, and all firms.
  • University classifications: Large academic entities are consolidated under parent institutions, and QS university groups are constructed by rank bands including QS1-50 and QS201-300.
  • Independent variable: ImageNet2012 equals 1 for AI conferences from 2012 onward and 0 otherwise.
  • Control variable: TotalNumOfPaper controls for conference popularity and selectivity by capturing the total number of papers in a given year.

Summary Statistics

The dataset documents rising firm participation in major AI conferences after 2012, alongside pronounced concentration among large technology firms and selected universities. HBCU participation remains limited, while firms’ presence in non-AI conferences is comparatively stable.

  • Microsoft alone produced 3,302 top-AI-conference publications, exceeding the combined total of all HBCU- and HACU-affiliated institutions.
  • After 2012, firm participation increased across all ten major AI conferences, while non-AI conferences showed no consistent comparable pattern.
  • All firms increased annual AI-conference publication by 46 papers, with Fortune500Tech firms accounting for more than 350 additional papers over eight years.
  • Fortune500NonTech firms showed no discernible AI-research impact, whereas non-Fortune500 technology firms such as Baidu, Nvidia, Uber, and SenseTime increased participation.

estimates

Generalized synthetic-control estimates associate the 2012 ImageNet shock with a substantial increase in large technology firms’ AI participation. Firms expanded through both independent and university collaborations, especially with elite universities, while mid- and lower-tier universities declined relative to counterfactuals.

  • Firm participation: The treatment effect was smaller from 2012 to 2014 and became more salient after 2016, consistent with the reported timing of compute availability.
  • Collaboration strategies: Firm-only publications increased by 8 annually per conference, while joint firm–university publications increased by almost 42.
  • Collaboration strategies: Firms collaborated six times more with QS-ranked top-50 universities than with QS-ranked 301–500 universities combined.
  • University participation: QS top-50 universities gained more than 40 papers annually per conference, exceeding 320 additional papers per year per conference over eight years.
  • University participation: QS-ranked 201–300 and 301–500 universities published 8 and almost 6 fewer papers annually per conference, respectively, with the latter representing 25% fewer papers than counterfactuals.

method estimates

Placebo analyses and alternative affiliation measurements support the reported ImageNet-shock estimates, while the authors identify university-ranking classification as a limitation. The alternative ranking analysis produces qualitatively consistent results.

  • Placebo tests: The Software Engineering placebo showed no significant treatment effect after 2012, unlike the AI-conference result.
  • Placebo tests: Assigning the ImageNet shock to 2009 produced no discernible treatment effect between 2009 and 2012, with a p-value much higher than 0.05.
  • Robustness: The results remained robust when organizational participation was measured using weighted author affiliations.
  • Limitation: The authors identify possible idiosyncrasies in classifying elite and non-elite universities as a limitation of the estimates.
  • Robustness: Using the US News Global Ranking instead of the QS ranking, elite universities still gained 19 papers relative to the counterfactual.

method estimates (US News 2018 ranking)

The estimates use counterfactual and fixed-effects approaches to assess how the 2012 ImageNet shock and compute affected participation across university and firm groups. Results consistently show gains for large firms and elite universities alongside losses for mid- and lower-tier universities.

  • Robustness across rankings: 37 additional papers: universities ranked as elite in 2011 published more than the counterfactual after the ImageNet shock.Using the 2011 QS ranking avoids rankings being affected by the 2012 shock.
  • Heterogeneous university effects: Universities ranked 201-300 published approximately 14 fewer papers per year per conference than the counterfactual.Universities ranked 301-500 published 6 fewer papers than the counterfactual.
  • Robustness across rankings: The estimates are consistent across rankings, years, estimation methods, and falsification tests.The matrix completion estimator produced similar significance and effect sizes.
  • Compute effects: A one-standard-deviation increase of 41700.14 petaflops per second annually corresponded to 29 additional firm papers per year per conference.The firm-compute association is positive after the ImageNet shock.
  • Compute effects: For elite universities, the same 41700.14 increase corresponded to 4 additional papers per year per conference.Universities ranked 51-100 showed a positive impact, while those ranked 101-200 showed no noticeable impact.
  • Compute effects: Increased compute negatively affected non-elite universities ranked 101-200, 201-300, and 301-500, which each lost ground in AI research.The results suggest that increased compute helped large technology firms and elite universities more than mid- and lower-tier universities.

Appendix D).

Firms, especially Fortune500Tech firms, increased their share of deep-learning publications after 2012, while text analysis identifies distinct research emphases across organizational groups. Large firms and elite universities show greater engagement with compute-intensive deep learning than non-elite universities.

  • Publication shares: Firms’ share of deep-learning publications increased steadily and more sharply after 2012, driven particularly by Fortune500Tech firms.For most other groups, the share of deep-learning papers remained comparatively stable.
  • Text-analysis method: TF-IDF analysis compares research terms across Fortune500GlobalTech, QS1-50, and QS301-500 papers.The analysis preprocesses abstracts, calculates group-specific term scores, and normalizes them across terms.
  • Research focus: All three groups focus largely on state-of-the-art research aimed at achieving the best result on a benchmark.The groups nevertheless differ in approaches, datasets, and focus areas.
  • Research focus: Firms have higher presence in terms associated with convolutional neural networks, deep learning, long short-term memory, and recurrent neural networks.The pattern is consistent with firms’ focus on deep learning and commercial applications.
  • Compute-intensive research: GPU usage is higher among large firms and top-tier universities than among non-elite universities.Non-elite universities show greater emphasis on support vector machines, feature selection, Bayesian methods, computational cost, and computational complexity.
  • Compute-intensive research: The text analysis is consistent with large firms and elite universities having an advantage in deep-learning and compute-intensive research relative to non-elite universities.A similar NeurIPS analysis produced qualitatively similar results.

Diversity and Innovation Outcome in AI research

The paper links concentration of AI research among firms and elite universities to concerns about diversity, bias, and research direction. It argues that compute and other resource barriers may justify policy interventions and broader access to data and infrastructure.

  • Diversity and bias: Prior research links limited diversity and representation to biased datasets, models, and outcomes affecting minorities and vulnerable populations.Examples include racial disparities in automated speech recognition and gender-related gaps in AI technologies.
  • Diversity and research direction: AI is increasingly shaped by large technology firms and elite universities, both of which have documented diversity problems.The paper connects this concentration to questions about the future direction and inclusiveness of AI research.
  • Open questions: The paper calls for further research on how divergence between elite and non-elite universities could affect AI technologies.This remains an open question rather than an established outcome in the paper.
  • Policy implications: The paper identifies government intervention as potentially necessary to reduce the compute divide.It presents this as a policy implication of the study’s findings.
  • Policy implications: Publicly released datasets could benefit resource-constrained organizations, including non-elite universities and startups.The paper proposes publicly owned data as one way governments can support participation in AI research.
  • Concentration and entry barriers: High costs for compute, datasets, and human capital can raise entry barriers and concentrate power among established firms.The paper also notes that compute costs can discourage some academic research and accelerate movement from academia to industry.

Conclusion

Using 171,394 papers from 57 major computer science conferences, the paper finds that deep learning’s rise increased AI research concentration among large technology firms and elite universities. It attributes part of the divergence to unequal compute access and calls for coordinated efforts to address the compute divide.

  • Central findings: AI is increasingly shaped by large technology firms and elite universities following deep learning’s unanticipated GPU-driven rise since 2012.The paper describes these actors as increasingly central to AI knowledge production.
  • Study scope: 171,394 peer-reviewed papers from 57 major computer science conferences form the study’s empirical basis.The dataset is used to examine organizational participation in AI research.
  • Central findings: Hundreds of mid-tier and lower-tier universities have been crowded out of the AI research space.The paper reports a marked difference in AI knowledge production between elite and non-elite universities.
  • Compute divide: Uneven access to computing power partly explains the divergence between firms and universities, which the paper terms the compute divide.Machine-learning-based text analysis provides additional evidence for this explanation.
  • Implications: The paper argues that lack of access to specialized resources can de-democratize a scientific field.This extends the study’s contribution from AI participation to knowledge production more broadly.
  • Contribution: The paper documents that firms are increasing their research presence in AI, contrary to recent evidence of declining corporate research participation.This contribution concerns corporate science as well as unequal access to compute.

Appendix B

Appendix B describes the conference panel data and documents a post-2012 shift toward compute-intensive, deep-learning-oriented AI research. It also shows that AI conferences diverged from non-AI conferences after 2012.

  • Data structure: The panel comprises observations from 57 conferences, including biannual conferences such as ICCV, ECCV, and IJCAI.Each box in the panel-data structure represents one observation.
  • Conference participation: AI conferences diverged from non-AI conferences only after 2012 in the AllFirms measure.The comparison covers AI and non-AI conferences over time.
  • Research focus: Since 2012, deep-learning-related keywords became widely used at NeurIPS, while Bayesian methods, support vector machines, and feature selection became less popular.Convolutional and recurrent neural networks, long short-term memory, and generative adversarial networks were barely used or absent before 2012 but prominent afterward.
  • Research focus: AAAI shows a similar post-2012 pattern, with wider use of deep learning and declining popularity of traditional methods.The comparison uses normalized TF-IDF scores before and after 2012.
  • Compute intensity: Taken together, the NeurIPS and AAAI evidence indicates that AI research has become more deep-learning dependent and therefore more compute-intensive since 2012.The appendix presents this conclusion from the two conference-level keyword analyses.
Loading 2010.15581v1…