Source-linked AI summary
The AI Index 2021 Annual Report
Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, Yoav Shoham, Jack Clark, Raymond Perrault
TL;DR
The report addresses the challenge of evaluating AI progress as models develop novel capabilities and benchmarks. It expands and calibrates AI data while using temporal Shapley values to assess contributions over time; AI publications rose substantially, though newer benchmarks need more longitudinal data.
Problem
Novel model capabilities and zero- to few-shot performance are difficult to define and measure with existing benchmarks.
Method
The report expands and calibrates AI data and measures each system’s contribution to state-of-the-art performance over time using temporal Shapley values.
Results
AI publications grew nearly 12 times between 2000 and 2019, while AI journal publications were 5.4 times higher in 2020 than in 2000.
Takeaways & Limitations
Tracking AI progress requires new benchmarks for emerging capabilities and continued measurement as systems are tested across them.
Takeaways & Limitations
The relatively new benchmarks require a couple of years of testing before temporal progress graphs can be generated.
Abstract
from arXiv · showhide
Welcome to the fourth edition of the AI Index Report. This year we significantly expanded the amount of data available in the report, worked with a broader set of external organizations to calibrate our data, and deepened our connections with the Stanford Institute for Human-Centered Artificial Intelligence (HAI). The AI Index Report tracks, collates, distills, and visualizes data related to artificial intelligence. Its mission is to provide unbiased, rigorously vetted, and globally sourced data for policymakers, researchers, executives, journalists, and the general public to develop intuitions about the complex field of AI. The report aims to be the most credible and authoritative source for data and insights about AI in the world.
INTRODUCTION TO THE 2021 AI INDEX REPORT
The 2021 AI Index expanded its data, external calibration, and HAI connections while documenting AI’s growth, applications, and unresolved measurement and ethics challenges.
- The report expanded its data, collaborated with more external organizations to calibrate it, and deepened connections with Stanford HAI.
- AI hiring and private investment grew during 2020 despite COVID-19, while virtual conferences increased participation.
- Drug design and discovery received more than USD 13.8 billion in private AI investment in 2020, 4.5 times higher than in 2019.
- In 2019, 65% of graduating North American AI PhDs entered industry, up from 44.4% in 2010.
- China led the United States in AI journal citations in 2020, while the United States retained more heavily cited AI conference papers.
- AI ethics lacks measurement benchmarks and consensus, and researchers and civil society rated it more important than industrial organizations.
Organizations
The report’s organizations include academic, industry, government, nonprofit, and conference-community contributors spanning AI research, education, policy, and data production.
- Contributors represent Stanford, Harvard, industry, government, nonprofits, research centers, universities, and AI-focused organizations.
- The listed organizations contribute expertise across AI research, education, corporate representation, benchmarks, molecular synthesis, and policy.
- The report also acknowledges contributors to conferences, data visualization, graphic design, editing, and website development.
CHAPTER 1: RESEARCH & DEVELOPMENT
AI research activity expanded across publications, citations, conference participation, and preprints, with institutional patterns differing across countries and regions.
- 34.5%: AI journal publications grew from 2019 to 2020, compared with 19.6% growth from 2018 to 2019.
- Academic institutions produced the largest share of peer-reviewed AI papers everywhere, while second-place origins differed across the United States, China, and the European Union.
- In 2020, China surpassed the United States in the global share of AI journal citations, while the United States maintained more cited AI conference papers.
- Conference attendance across nine AI conferences almost doubled in 2020 as major conferences moved online during COVID-19.
- AI-related arXiv publications grew more than sixfold, from 5,478 in 2015 to 34,736 in 2020.
- AI publications represented 3.8% of worldwide peer-reviewed scientific publications in 2019, up from 1.3% in 2011.
CHAPTER 2: TECHNICAL PERFORMANCE
AI capabilities advanced rapidly in generation, vision, language, reasoning, healthcare, and biology, while established benchmarks increasingly struggled to measure progress.
- Generative AI can produce text, audio, and images that humans struggle to distinguish from non-synthetic outputs in some constrained applications.
- Computer vision performance is flattening on some major benchmarks even as training resources and deployed technologies continue to expand.
- NLP advances have outpaced evaluation benchmarks, with systems reaching human-level performance on SuperGLUE after earlier progress exceeded GLUE’s scope.
- New evolving-benchmark metrics attribute groups’ performance over time to individual systems and are applied to theorem proving and Boolean satisfiability.
- Machine learning has reshaped healthcare and biology, including AlphaFold’s protein-folding breakthrough and improved chemical-synthesis planning.
CHAPTER 3: THE ECONOMY
AI investment and hiring continued to grow during the COVID-19 pandemic, although the United States experienced a decline in AI job postings. Investment increasingly concentrated in fewer startups, while industry attention to AI ethics remained limited.
- USD 13.8 billion was invested in “Drugs, Cancer, Molecular, Drug Discovery” in 2020, 4.5 times the 2019 amount.
- AI hiring continued to grow across sampled countries in 2020 despite the pandemic, with Brazil, India, Canada, Singapore, and South Africa showing the highest growth from 2016 to 2020.
- 9.3% more private AI investment was recorded in 2020 than in 2019, even as the number of newly funded companies fell for the third consecutive year.The 2020 increase exceeded the 5.7% increase from 2018 to 2019.
- Industry efforts to address AI ethics remained limited, particularly regarding equity, fairness, and privacy risks.
- 27% of surveyed businesses increased AI investment during the pandemic, while less than a fourth decreased it and half reported no effect.
- U.S. AI job postings fell 8.2%, from 325,724 in 2019 to 300,999 in 2020, marking the first decrease in the share of postings in six years.
CHAPTER 4: AI EDUCATION
AI education expanded, while the field’s talent pipeline increasingly connected to industry and international graduates. Specialized offerings varied by level, with robotics and automation prominent in degree programs and machine learning in short courses.
- Undergraduate AI course offerings increased 102.9% and graduate offerings increased 41.7% over the previous four academic years.
- 65.7% of new North American AI PhDs entered industry in 2019, up from 44.4% in 2010, while academia’s share fell from 42.1% to 23.7%.
- AI-related PhDs rose from 14.2% of U.S. computer science PhDs in 2010 to around 23% in 2019.
- AI faculty departures from North American universities to industry declined from 42 in 2018 to 33 in 2019.
- International students comprised 64.3% of new North American AI PhDs in 2019; 81.8% of foreign graduates stayed in the United States.
- In the European Union, robotics and automation dominated specialized bachelor’s and master’s programs, while machine learning dominated specialized short courses.
CHAPTER 5: ETHICAL CHALLENGES OF AI APPLICATIONS
Ethics-related AI research and news attention increased, but ethics remained a relatively small presence in major AI conference paper titles.
- The number of AI conference papers with ethics-related keywords in their titles has grown since 2015.
- Despite that growth, the average number of ethics-keyword paper titles at major AI conferences remained low over the years.
- The five most-covered 2020 AI ethics topics included the European Commission’s AI white paper, Timnit Gebru’s dismissal, and IBM’s exit from facial recognition.
CHAPTER 6: DIVERSITY IN AI
Gender, racial, and inclusion disparities remained substantial in AI, even as participation in some diversity-focused activities grew. Survey evidence also documents discrimination and harassment as barriers to participation.
- Female AI PhD graduates averaged less than 18% of North American graduates, while female tenure-track CS faculty comprised just 16% at several universities worldwide.
- Among new U.S. resident AI PhD graduates in 2019, 45% were white, 22.4% Asian, 3.2% Hispanic, and 2.4% African American.
- Black or African American and Hispanic computing PhDs averaged only 3.1% and 3.3%, respectively, over the previous decade.
- Black in AI workshop attendance and submissions were 2.6 times higher in 2019 than in 2017, while accepted papers were 2.1 times higher.
- More than 40% of Queer in AI members surveyed reported discrimination or harassment at work or school, and almost half viewed limited inclusiveness as an obstacle.
CHAPTER 7: AI POLICY AND NATIONAL STRATEGIES
AI policy activity expanded across national and intergovernmental levels by the end of 2020, while U.S. congressional attention also reached a historic high.
- More than 30 countries and regions had published national AI strategies by December 2020, following Canada’s first strategy in 2017.
- The launch of GPAI and the OECD AI Policy Observatory and Network of Experts on AI promoted intergovernmental cooperation to support AI for all.
- AI mentions by the U.S. 116th Congress exceeded those of the 115th Congress by more than threefold across legislation, committee reports, and CRS reports.
- CHAPTER 1: Research & Development: The report frames R&D as fundamental to AI progress and examines publications, citations, patents, preprints, geographic contributions, conferences, and GitHub activity.Its data sources include Elsevier/Scopus, Microsoft Academic Graph, arXiv, and Nesta.
CHAPTER HIGHLIGHTS
AI activity expanded across publications, patents, preprints, conferences, regions, and institutional sectors. China gained prominence in journal and conference output and citations, while the United States retained an advantage in cited conference papers and academic-corporate collaboration.
- Publications: AI journal publications increased 34.5% from 2019 to 2020, compared with 19.6% growth from 2018 to 2019.
- Publications: Peer-reviewed AI publications grew nearly 12 times from 2000 to 2019, reaching 3.8% of all worldwide peer-reviewed publications.
- Regions: East Asia & Pacific held the largest regional share of peer-reviewed AI publications since 2004, while South Asia and sub-Saharan Africa grew eightfold and sevenfold from 2009 to 2019.
- Institutional affiliation: The United States produced more than twice as many hybrid academic-corporate AI publications as the European Union, while corporate affiliation was its leading nonacademic source.
- Geographic leadership: China held 18.0% of AI journal publications in 2020 and surpassed the United States in journal-citation share, 20.7% to 19.8%.
- Preprints and conferences: AI-related arXiv publications grew from 5,478 in 2015 to 34,736 in 2020, while virtual conferences nearly doubled attendance across nine conferences.
25,719 IROS
The report surveys AI’s expanding research and software ecosystem while highlighting both rapid technical progress and growing ethical concerns. It also documents increasing corporate participation in AI research and the prominence of major open-source libraries.
- 25,719 IROS: Corporate representation increased across all 10 major AI conferences, extending the compute divide in deep learning research.The cited discussion links unequal compute resources with inequality and notes that large technology firms tend to be less diverse than smaller institutions.
- 25,719 IROS: GitHub stars are used to measure the popularity of AI programming libraries.The report describes GitHub as a platform for uploading, commenting on, and downloading software, where stars express user interest.
- 25,719 IROS: TensorFlow was the most popular AI software library, followed by Keras in 2020, while PyTorch was increasingly popular excluding TensorFlow.Keras is built on TensorFlow 2.0, and the report identifies PyTorch as another library gaining popularity.
- Technical Performance: The Technical Performance chapter covers computer vision, language, speech, concept learning, and theorem proving using quantitative benchmarks and qualitative insights.Its measurements include common benchmarks and prize challenges, complemented by evidence from academic papers.
- Technical Performance: Wider AI deployment accompanies growing concerns about algorithmic bias and the ethical challenges of capabilities such as image and video synthesis.The report frames these concerns alongside the increasing ease and breadth of AI system deployment.
CHAPTER HIGHLIGHTS
The report documents broad AI progress across perception, language, reasoning, biology, and applied systems, while noting that some benchmarks and evaluation methods are becoming inadequate.
- Generative everything: Generative models increasingly produce text, audio, and images that humans struggle to distinguish from authentic outputs in constrained applications.This progress is also driving investment in generative-model detection technologies.
- The industrialization of computer vision: Computer vision performance has advanced substantially, but some major benchmarks are flattening as companies use increasingly large computational resources.The report suggests harder benchmarks are needed to continue testing progress.
- Natural Language Processing: NLP capabilities have improved rapidly enough that technical advances are beginning to outpace the benchmarks used to evaluate them.Large language models have also been deployed in economically significant applications such as search.
- New analyses on reasoning: New evolving-benchmark and credit-attribution analyses are applied to automated theorem proving and Boolean satisfiability to study progress beyond fixed leaderboards.These analyses assess performance over time and attribute contributions to individual systems.
- Healthcare and biology: Machine learning is reshaping healthcare and biology through protein-folding advances, chemical synthesis planning, and faster COVID-19 drug discovery.The report highlights AlphaFold, molecular representations, and PostEra’s machine-learning-based techniques.
- Computer vision: Image recognition has become more affordable and broadly applicable as underlying technology advances, while deep learning has dominated competition leaderboards.ImageNet training time fell from 6.2 minutes in December 2018 to 47 seconds in July 2020, alongside increased accelerator use.
- Computer vision: Scissors recognition improved by 129.2%, from 6.6% in 2019 to 15.22% in 2020, despite remaining the 10th hardest activity.The report identifies this as the greatest improvement among all activities.
CHAPTER HIGHLIGHTS
AI investment, hiring, and labor demand generally expanded through 2020 despite the COVID-19 pandemic, although U.S. AI job postings declined and industry efforts on AI ethics remained limited.
- USD 13.8 billion was invested privately in drugs, cancer, molecular, and drug discovery in 2020, 4.5 times more than in 2019.
- Private AI investment rose 9.3% from 2019 to 2020 while the number of newly funded companies fell for the third consecutive year.
- Industry efforts to address AI ethics remained limited, with equity, fairness, and privacy receiving comparatively little attention.
- U.S. AI job postings fell 8.2%, from 325,724 in 2019 to 300,999 in 2020, the first decline in the country’s share in six years.
- AI hiring increased across sample countries in 2020, with Brazil, India, Canada, Singapore, and South Africa showing the highest growth from 2016 to 2020.
U.S. AI Labor Demand: By Skill Cluster
U.S. AI labor demand grew across skill clusters from 2013 to 2020, with machine learning and artificial intelligence-related jobs showing the fastest increases before a broad 2020 decline in share.
- U.S. AI Labor Demand: By Skill Cluster: Machine learning-related AI jobs rose from 0.1% to 0.5% of total U.S. jobs between 2013 and 2020.
- U.S. AI Labor Demand: By Skill Cluster: The machine learning and artificial intelligence skill clusters experienced the fastest growth in U.S. online AI job postings over the period.
- U.S. AI Labor Demand: By Skill Cluster: In 2020, the share of AI jobs among overall U.S. job postings decreased across all skill clusters.
- U.S. AI Labor Demand: By Skill Cluster: Artificial intelligence-related AI jobs increased from 0.03% to 0.3% of total U.S. jobs between 2013 and 2020.
U.S. Labor Demand: By Industry
U.S. AI labor demand varied substantially by industry in 2020, with information and professional services leading while agriculture experienced the largest recent increase.
- U.S. Labor Demand: By Industry: Information had the highest share of AI job postings in 2020 at 2.8%, followed by professional, scientific, and technical services at 2.5%.
- U.S. Labor Demand: By Industry: Agriculture, forestry, fishing, and hunting recorded the biggest increase in AI job-posting share from 2019 to 2020, rising by almost 1 percentage point.
- U.S. Labor Demand: By Industry: Agriculture, forestry, fishing, and hunting had a 2.1% share of AI job postings in 2020.
U.S. Labor Demand: By State
U.S. AI labor demand differed by state in 2020: the District of Columbia had the highest posting share, while California had the largest total volume.
- U.S. Labor Demand: By State: The District of Columbia had the highest share of AI job postings in 2020, at 1.88%.
- U.S. Labor Demand: By State: California had the highest number of AI job postings, with 63,433 positions.
- U.S. Labor Demand: By State: Six states besides Washington, D.C., recorded AI job-posting shares above 1% in 2020, compared with five the previous year.
- U.S. Labor Demand: By State: California had more AI job postings than Texas, New York, and Virginia combined.
AI SKILL PENETRATION
AI skill penetration measures how prevalent AI skills are across occupations using LinkedIn profile, job, and location data.
- AI skill penetration is the average share of AI skills among each occupation’s top 50 skills.
- The metric uses LinkedIn data on members’ listed skills, positions held, and position locations.
- The measure estimates how prevalent AI skills are across occupations.
Global Comparison
Across the sampled countries and industries, AI investment and adoption expanded in 2020, while hiring growth continued despite the COVID-19 pandemic. Adoption patterns varied by region, industry, function, and capability, but organizations reported limited attention to AI risks and ethics.
- Global Comparison: India’s relative AI skill penetration was 2.83 times the global average, followed by the United States at 1.99 times.
- Global Comparison: USD 67.9 billion was invested globally in AI in 2020, up 40% from 2019, while M&A investment increased 121.7%.
- Global Comparison: Private AI investment rose 9.3% in 2020 even as the number of funded companies fell for the third consecutive year.
- Global Comparison: Over 50% of respondents reported AI adoption in at least one business function, with developed Asia-Pacific countries leading in adoption.
- Global Comparison: AI adoption varied by industry, function, and capability, with service operations, product and service development, and marketing and sales among the most common functions.
- Global Comparison: Only a minority of companies acknowledged AI risks or reported mitigation efforts, while cybersecurity remained the only risk considered relevant by a majority.