The AI Index 2021 Annual Report

COVID AND AI

• The report describes mixed COVID-19 effects: AI hiring and private investment grew in 2020, and virtual conferences may have increased AI research participation.It also describes machine-learning techniques used by an AI startup to accelerate COVID-related drug discovery.Source: COVID AND AI · p. 2

CHANGES FOR THIS EDITION

• Following a 2020 survey of more than 140 readers, the report expanded technical analysis, added diversity and ethics data while noting information gaps, and strengthened cross-country economic, bibliometric, and AI-strategy comparisons.Source: CHANGES FOR THIS EDITION · p. 2

PUBLIC DATA AND TOOLS

• The report offers public data and charts, a revamped Global AI Vibrancy Tool comparing up to 26 countries across 22 indicators, and a companion report on AI measurement issues.The interactive tool supports preference-based country comparisons, policy priorities, and identification of AI centers of excellence in advanced and emerging economies.Source: PUBLIC DATA AND TOOLS · p. 2

TOP 9 TAKEAWAYS

• Drug discovery received over USD 13.8 billion in private AI investment in 2020, 4.5 times 2019; 65% of North American AI PhDs entered industry in 2019, versus 44.4% in 2010.Source: TOP 9 TAKEAWAYS · p. 4
• AI systems can compose text, audio, and images so convincingly that people struggle to distinguish synthetic from nonsynthetic outputs in some constrained applications.Source: TOP 9 TAKEAWAYS · p. 4
• In 2019, 45% of new U.S. resident AI PhDs were white, compared with 2.4% African American and 3.2% Hispanic; 64.3% of North American AI PhDs were international students.Of foreign graduates, 81.8% stayed in the United States and 8.6% took jobs outside it.Source: TOP 9 TAKEAWAYS · p. 4
• China led the United States in AI journal citations, while the United States had more and more-heavily cited AI conference papers; AI mentions more than tripled in the 116th Congress versus the 115th.Source: TOP 9 TAKEAWAYS · p. 4
• Surveillance-related image classification, face recognition, video analysis, and voice identification advanced significantly in 2020; AI ethics lacked benchmarks and consensus, and researchers and civil society valued it more than industry.Source: TOP 9 TAKEAWAYS · p. 4

AI Ethics Lab

• Cansu Canca, Yasemin Usta, Rediet Abebe, and Hassan Kane are listed with AI Ethics Lab and Black in AI.Source: AI Ethics Lab · p. 7

Burning Glass Technologies

• Burning Glass Technologies lists Layla O’Kane, Bledi Taska, and Zhou Zhou.Source: Burning Glass Technologies · p. 7

Computing Research Association

• The Computing Research Association, Elsevier, and Intento are represented by named contributors Andrew Bernat, Susan Davidson, Clive Bastin, Jörg Hellwig, Sarah Huggett, Mark Siebert, Grigory Sapunov, and Konstantin Savenkov.Source: Computing Research Association · p. 7

National Institute of Standards and Technology

• The section lists contributors associated with Nesta, NetBase Quid, and Queer in AI.Names include Joel Klinger, Juan Mateos-Garcia, Kostas Stathoulopoulos, Zen Ahmed, Scott Cohen, Julie Kim, and Raphael Gontijo Lopes.Source: National Institute of Standards and… · p. 7, NetBase Quid · p. 7

ActivityNet

• Named contributors are credited for ActivityNet, AI Brain Drain and Faculty Departure, Automated Theorem Proving, and the Boolean Satisfiability Problem.Source: ActivityNet · p. 8, AI Brain Drain and Faculty Departure · p. 8, Automated Theorem Proving · p. 8

Boolean Satisfiability Problem

• The listed topics also include corporate representation at AI research conferences, conference attendance, ethics at AI conferences, and visual question answering.Contributors are identified for these areas, including researchers affiliated with universities, Google, Stanford, IBM Research–Europe, and Georgia Tech/FAIR.Source: Boolean Satisfiability Problem · p. 8, Conference Attendance · p. 8, Ethics at AI Conferences · p. 8

CHAPTER 1: RESEARCH & DEVELOPMENT

• AI journal publications grew 34.5% from 2019 to 2020, versus 19.6% from 2018 to 2019.The report notes the later growth rate was much higher.Source: CHAPTER 1: RESEARCH & DEVELOPMENT · p. 10
• Academic institutions produced the highest proportion of peer-reviewed AI papers in every major country and region; corporate research ranked second in the United States (19.2%), while government did in China (15.6%) and the European Union (17.2%).Source: CHAPTER 1: RESEARCH & DEVELOPMENT · p. 10
• In 2020, China surpassed the United States in share of AI journal citations worldwide, while the United States consistently had more cited AI conference papers over the last decade.China briefly overtook the United States in overall AI journal publications in 2004, then the United States retook the lead in 2017.Source: CHAPTER 1: RESEARCH & DEVELOPMENT · p. 10
• After most major AI conferences moved online in response to COVID-19, attendance across nine conferences almost doubled in 2020.Source: CHAPTER 1: RESEARCH & DEVELOPMENT · p. 10
• AI-related arXiv publications grew more than sixfold in six years, from 5,478 in 2015 to 34,736 in 2020.Source: CHAPTER 1: RESEARCH & DEVELOPMENT · p. 10
• AI publications made up 3.8% of peer-reviewed scientific publications worldwide in 2019, up from 1.3% in 2011.Source: CHAPTER 1: RESEARCH & DEVELOPMENT · p. 10

Generative everything

• Generative AI can produce text, audio, and images that people struggle to distinguish from real outputs in some constrained applications.This has prompted investment in detection technologies, with the DeepFake Detection Challenge data indicating how well computers distinguish outputs.Source: CHAPTER 2: TECHNICAL PERFORMANCE · p. 10

The industrialization of computer vision

• Computer vision performance is flattening on some large benchmarks as training compute grows and deployment technologies mature.The report says this suggests the community needs harder benchmarks; object-detection frameworks for analyzing video still frames are among the maturing technologies.Source: CHAPTER 2: TECHNICAL PERFORMANCE · p. 10

NLP outruns its evaluation metrics

• NLP advances have outpaced evaluation benchmarks, with systems reaching human-level performance on SuperGLUE after earlier progress overshot GLUE.Google and Microsoft have deployed BERT in their search engines, while large language models have also been developed by companies including Microsoft and OpenAI.Source: CHAPTER 2: TECHNICAL PERFORMANCE · p. 11

New analyses on reasoning

• New AI Index analyses allow reasoning performance to be measured on evolving benchmarks and credit to be attributed to individual systems over time.The analyses are applied to Automated Theorem Proving and Satisfiability of Boolean formulas.Source: CHAPTER 2: TECHNICAL PERFORMANCE · p. 11

Machine learning in healthcare and biology

• Machine learning has reshaped healthcare and biology, including AlphaFold’s protein-folding breakthrough and ML applications to chemical synthesis and COVID-related drug discovery.Scientists use ML representations of chemical molecules to improve synthesis planning, and PostEra used ML-based techniques to accelerate COVID-related drug discovery during the pandemic.Source: CHAPTER 2: TECHNICAL PERFORMANCE · p. 11

Private AI investment by sector

• “Drugs, Cancer, Molecular, Drug Discovery” received more than USD 13.8 billion in private AI investment in 2020, 4.5 times its 2019 amount.Source: CHAPTER 3: THE ECONOMY · p. 11

AI hiring

• Brazil, India, Canada, Singapore, and South Africa had the highest AI hiring growth from 2016 to 2020, and hiring continued to grow across sample countries in 2020.Source: CHAPTER 3: THE ECONOMY · p. 11

Private AI investment

• Private AI investment rose 9.3% in 2020 from 2019, versus 5.7% from 2018 to 2019, while newly funded companies declined for the third consecutive year.Source: CHAPTER 3: THE ECONOMY · p. 11

Responsible AI

• McKinsey survey findings indicate limited industry action on AI ethics: equity and fairness received comparatively little attention, while fewer companies viewed privacy risks as relevant and mitigation efforts did not change from 2019.Source: CHAPTER 3: THE ECONOMY · p. 11

AI investment and the pandemic

• Half of McKinsey survey respondents said the pandemic did not affect their AI investment, 27% reported increasing it, and fewer than a fourth reported decreasing it.Source: CHAPTER 3: THE ECONOMY · p. 11

U.S. AI job postings

• 8.2% fewer AI jobs were posted in the United States in 2020 than in 2019, falling from 325,724 to 300,999—the first decrease in its share of AI job postings in six years.Source: CHAPTER 3: THE ECONOMY · p. 11

CHAPTER 4: AI EDUCATION

• Undergraduate practical-AI courses rose 102.9% and graduate courses 41.7% over four academic years, according to a 2020 AI Index survey.The survey measured courses teaching skills to build or deploy a practical AI model.Source: CHAPTER 4: AI EDUCATION · p. 12
• Industry-bound new AI PhDs in North America rose from 44.4% in 2010 to 65.7% in 2019, an increase of 48%; academia-bound fell from 42.1% to 23.7%, a drop of 44%.Source: CHAPTER 4: AI EDUCATION · p. 12
• AI-related PhDs grew from 14.2% of U.S. CS PhDs in 2010 to around 23% in 2019, while networking, software engineering, programming languages, and compilers declined relative to 2010.AI and Robotics/Vision specializations saw a substantial increase.Source: CHAPTER 4: AI EDUCATION · p. 12
• AI faculty departures from North American universities to industry fell from 42 in 2018 to 33 in 2019 after a two-year increase.Of the 2019 departures, 28 were tenured and five untenured; from 2004 to 2019, Carnegie Mellon had 16 departures, Georgia Tech 14, and the University of Washington 12.Source: CHAPTER 4: AI EDUCATION · p. 12
• International students made up 64.3% of new AI PhDs in North America in 2019, up 4.3% from 2018; 81.8% of foreign graduates stayed in the U.S. and 8.6% took jobs abroad.Source: CHAPTER 4: AI EDUCATION · p. 12
• In the EU, most specialized AI academic offerings were at master’s level; robotics and automation led specialized bachelor’s and master’s programs, while machine learning dominated short courses.Source: CHAPTER 4: AI EDUCATION · p. 12

• Ethics-keyword paper titles submitted to AI conferences have grown since 2015, but their average number at major conferences remains low.Source: CHAPTER 5: ETHICAL CHALLENGES OF AI… · p. 12
• The five most-covered 2020 AI ethics news topics were the European Commission’s AI white paper, Timnit Gebru’s dismissal from Google, the UN ethics committee, the Vatican’s ethics plan, and IBM’s exit from facial recognition.Source: CHAPTER 5: ETHICAL CHALLENGES OF AI… · p. 12

• For more than a decade, women averaged less than 18% of North American AI PhD graduates, while female faculty made up 16% of tenure-track CS faculty at surveyed universities.Source: CHAPTER 6: DIVERSITY IN AI · p. 13
• In 2019, U.S. resident AI PhD graduates were 45% white, 22.4% Asian, 3.2% Hispanic, and 2.4% African American.Source: CHAPTER 6: DIVERSITY IN AI · p. 13
• White non-Hispanic graduates averaged 62.7% of computing PhDs over 10 years, compared with 3.1% for Black or African American and 3.3% for Hispanic graduates.Source: CHAPTER 6: DIVERSITY IN AI · p. 13
• By 2019, Black in AI workshop attendees and submitted papers were 2.6 times higher than in 2017, while accepted papers were 2.1 times higher.Source: CHAPTER 6: DIVERSITY IN AI · p. 13
• In Queer in AI’s 2020 membership survey, almost half viewed lack of inclusiveness as an obstacle to becoming an AI/ML practitioner, and over 40% reported workplace or school discrimination or harassment.Source: CHAPTER 6: DIVERSITY IN AI · p. 13

CHAPTER 7: AI POLICY AND NATIONAL STRATEGIES

• More than 30 countries and regions had published national AI strategies by December 2020, after Canada published the first in 2017.Source: CHAPTER 7: AI POLICY AND NATIONAL… · p. 13
• The 2020 launch of GPAI and the OECD AI Policy Observatory and Network of Experts on AI promoted intergovernmental cooperation to support AI development for all.Source: CHAPTER 7: AI POLICY AND NATIONAL… · p. 13
• AI mentions in legislation, committee reports, and CRS reports by the 116th U.S. Congress exceeded three times those of the 115th, making it the most AI-focused session in history.Source: CHAPTER 7: AI POLICY AND NATIONAL… · p. 13

By Institutional Affiliation

• Academic institutions supplied the largest share of peer-reviewed AI publications across China, the EU, and the United States, while the next-largest source differed by region.Government was the next-largest source in China (15.6%) and the EU (17.2%), while corporate affiliations led in the United States (19.2%). From 2015 to 2019, the United States produced more than twice as many co-authored academic-corporate AI publications as the EU, with China third.Source: CHAPTER HIGHLIGHTS · p. 17, By Institutional Affiliation · p. 21, Academic-Corporate Collaboration · p. 23

PEER-REVIEWED AI PUBLICATIONS

• Peer-reviewed AI publications grew nearly 12-fold from 2000 to 2019, increasing from 0.82% to 3.8% of worldwide publications.East Asia & Pacific held the largest regional share from 2004; between 2009 and 2019, South Asia and sub-Saharan Africa grew eightfold and sevenfold, respectively. China led the world share in 2019 after overtaking the EU in 2017, and its output was 3.5 times its 2014 level, versus 2 times for the EU and 2.75 times for the United States.Source: Overview · p. 18, By Region1 · p. 18, Figure 1.1.2 · p. 19

Overview

• AI journal publications were 5.4 times their 2000 level in 2020 and grew 34.5% from 2019, versus 19.6% growth the previous year.Their share of all global publications rose 0.4 percentage points in 2020, compared with an average annual increase of 0.03 percentage points over the prior five years.Source: Overview · p. 25, Figure 1.1.7a · p. 25, Figure 1.1.7b · p. 25

AI JOURNAL PUBLICATIONS

• In 2020, East Asia & Pacific held the largest AI journal share at 26.7%, while China led major powers in journal publications (18.0%) and citations (20.7%).Europe & Central Asia had 13.3% and North America 14.0% of journal publications; China’s citation share surpassed the United States (19.8%) for the first time. South Asia and Middle East & North Africa AI journal publication counts grew sixfold and fourfold over the previous decade.Source: By Region · p. 26, Figure 1.1.8 · p. 26, By Geographic Area · p. 27

AI conference publications

• AI conference publications quadrupled from 2000 to 2019, but 2019 output was only 1.09 times the 2010 level; China surpassed the United States in global publication share in 2019.East Asia & Pacific accounted for more than 27% of conference publications in 2020, and North America’s share reached 21.7%; the United States had 40.1% of 2020 conference citations, ahead of China (11.8%) and the EU (10.9%). Most major conferences went virtual during COVID-19, and attendance across nine almost doubled.Source: CHAPTER HIGHLIGHTS · p. 17, Overview · p. 28, By Region · p. 29

NUMBER of AI PATENT PUBLICATIONS, 2000-20

• AI patent publications increased from 21,806 in 2000 to 101,876 in 2019, while their share of all patents rose from around 2% in 2000 to 2.9% in 2020.The 2020 patent dataset had country or regional affiliations for only 8%, so the report excluded geographic AI patent shares from its main analysis.Source: Overview · p. 31, NUMBER of AI PATENT PUBLICATIONS,… · p. 31

ARXIV PUBLICATIONS

• AI-related arXiv publications grew more than sixfold, from 5,478 in 2015 to 34,736 in 2020.North America’s share declined from 41.6% in 2017 to 36.3% in 2020, while East Asia & Pacific’s share rose from 17.3% in 2015 to 26.5% in 2020.Source: ARXIV PUBLICATIONS · p. 32, By Region · p. 32, Figure 1.1.16 · p. 32

By Geographic Area

• China was catching up with the United States in arXiv AI-related publication counts, while the EU’s share remained largely unchanged.Source: By Geographic Area · p. 33, Figure 1.1.18a · p. 33

By Field of Study

• From 2015 to 2020, arXiv AI publications grew fastest in Robotics (cs.RO, 11 times) and Machine Learning in computer science (cs.LG, 10 times).In 2020, cs.LG and cs.CV accounted for 32.0% and 31.7%, respectively, of AI-related arXiv publications; from 2019 to 2020, cs.RO and cs.CL led growth at 35.8% and 35.4%, respectively.Source: By Field of Study · p. 34, Figure 1.1.19 · p. 34

Deep Learning Papers on arXiv

• Deep-learning publications in arXiv Computer Science and statistics Machine Learning categories grew almost sixfold over the last five years, based on topic modeling of paper abstracts.Researchers from Nesta used a topic-modeling algorithm to identify deep-learning papers by analyzing abstracts.Source: Deep Learning Papers on arXiv · p. 35, Figure 1.1.20 · p. 35

CONFERENCE ATTENDANCE

• In 2020, attendance across nine AI conferences almost doubled; IROS’s extended virtual access helps explain its high attendance, while IJCAI was absent from the 2020 charts.Organizers said virtual formats enabled broader global attendance, but exact counts were difficult to measure; AAMAS, KR, and ICPAS used different attendance measures, and ICPAS’s 450 attendees was an estimate.Source: CONFERENCE ATTENDANCE · p. 36

Corporate Representation at AI Research Conferences

• All 10 major AI conferences show upward corporate representation, which the authors link to unequal compute access and concerns about bias and fairness.Big tech firms have more resources to design AI products, but tend to be less diverse than smaller or less elite institutions.Source: Corporate Representation at AI… · p. 38, Figure 1.2.3 · p. 38

GITHUB STARS

• GitHub stars measure library popularity; TensorFlow is most popular, Keras ranks second in 2020, and PyTorch is becoming increasingly popular outside the TensorFlow comparison.Keras is built on TensorFlow 2.0; TensorFlow was developed by Google and publicly released in 2017, while PyTorch was created by Facebook.Source: GITHUB STARS · p. 39, Figure 1.3.1 · p. 40, Figure 1.3.2 · p. 40

ACCESS THE PUBLIC DATA

• Chapter 2 surveys technical progress in computer vision, language, speech, concept learning, and theorem proving using benchmark and prize-challenge measurements and qualitative insights from academic papers.Source: ACCESS THE PUBLIC DATA · p. 43
• Wider and easier AI deployment accompanies growing concerns about algorithmic bias and ethical challenges from capabilities such as image and video synthesis.Source: ACCESS THE PUBLIC DATA · p. 43

Deepfake Detection Challenge (DFDC)

• Deepfake detection log loss dropped by around 0.5 during the December 2019–March 2020 challenge, as researchers sought to counter malicious synthetic-face uses.The challenge used a public dataset of around 100,000 clips; lower log loss indicates more accurate predictions.Source: CHAPTER HIGHLIGHTS · p. 44, Deepfake Detection Challenge (DFDC) · p. 53, Figure 2.1.8 · p. 53

CHAPTER HIGHLIGHTS

• The highlights report NLP progress outpacing benchmarks, new evolving-benchmark analyses for symbolic reasoning, and machine-learning advances in protein folding and COVID-era drug discovery.They also describe computer vision performance flattening on some large benchmarks as companies invest more compute and deployed technologies mature; BERT was deployed in search, and SuperGLUE systems reached human-level performance.Source: CHAPTER HIGHLIGHTS · p. 44

ImageNet: Top-1 Accuracy

• 90.2% top-1 accuracy was achieved by a Google Brain model in January 2021, while ImageNet top-5 performance rose from around 85% in 2013 to almost 99% in 2020.Recent top-1 systems use extra training data, such as social-media photos, for pre-training. The report notes that the top-5 human error baseline is a rough estimate.Source: ImageNet: Top-1 Accuracy · p. 46, Figure 2.1.1 · p. 47, Figure 2.1.2 · p. 47

ImageNet: Training Time

• 47 seconds was the ImageNet training time in July 2020, down from 6.2 minutes in December 2018; one training cost fell from about USD 1,100 to about USD 7.43.Training-time variance between MLPerf entries also shortened, while competitors used more accelerator chips; frontier systems transitioned from GPUs to Google TPUs.Source: ImageNet: Training Time · p. 48, Figure 2.1.3 · p. 49, Figure 2.1.5 · p. 49

Harder Tests Beyond ImageNet

• ImageNet Adversarial tests robustness to natural confounders and persistent misclassifications, while ImageNet-C and ImageNet-Rendition test visual corruptions and generalization to illustrations.Because these benchmarks are relatively new, the report says temporal progress graphs must wait until the community tests a range of systems over the next couple of years.Source: Harder Tests Beyond ImageNet · p. 50

GAN PROGRESS ON FACE GENERATION

• By 2018, synthetic face generation had become difficult for humans to assess further, while broader generated categories and new evaluation methods were needed to track progress.STL-10 results show progress in convincing synthetic images. FID uses model-derived quantitative data; human-rating approaches such as HYPE may matter more as models improve, but are slower and more expensive.Source: Figure 2.1.6 · p. 51, STL-10: Fréchet Inception Distance… · p. 51, GAN PROGRESS ON FACE GENERATION · p. 52

Common Objects in Context (COCO): Keypoint Detection Challenge

• COCO keypoint detection reached 80.8% average precision after roughly 33% improvement in four years; DensePose accuracy rose from 56% in 2018 to 72% in 2019.Keypoint detection localizes body joints while detecting people or objects; DensePose estimates correspondences between image pixels and a 3D body template.Source: Figure 2.1.9 · p. 54, Common Objects in Context (COCO):… · pp. 54–55, Figure 2.1.10 · p. 55

Cityscapes

• Mean IoU on Cityscapes pixel-level semantic labeling increased by 35% from 2014 to 2020, with a significant progress boost in 2016 and 2017 as residual networks were adopted.Cityscapes supports semantic understanding of urban environments, which is important for autonomous vehicles.Source: Figure 2.1.11 · p. 56, Cityscapes · p. 56

EMBODIED VISION

• Embodied-vision research is moving beyond passive image and video tasks toward interactive systems developed in simulation; meanwhile, commercial cloud MT systems with pretrained models grew from 8 in 2017 to 28 in 2020.YOLO variants track performance against inference latency for real-world trade-offs; simulators support navigation and manipulation, and SEAN-EP enables remote human demonstrations. Google and Microsoft began deploying BERT in search in 2019.Source: EMBODIED VISION · p. 57, You Only Look Once (YOLO) · p. 60, §2.3 · p. 62

ActivityNet: Hardest Activity

• ActivityNet temporal-localization mean average precision among submissions grew by 140% in five years, and rock-paper-scissors improved 129.2%, from 6.6% in 2019 to 15.22% in 2020.The task localizes start and end times and recognizes activities in 600-hour untrimmed videos; drinking coffee remained the hardest activity in 2020. Activity understanding still focuses largely on short events, though longer-term understanding is gaining traction.Source: ActivityNet: Temporal Action… · p. 58, Figure 2.2.1 · p. 58, ACTIVITY RECOGNITION · p. 58

National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT)

• FRVT false non-match rates on mug-shot-type and visa photos fell from close to 50% to a fraction of a percent in 2020.FRVT independently evaluates commercial and prototype face-recognition technologies used for civil and governmental tasks.Source: National Institute of Standards and… · p. 61, Figure 2.2.4 · p. 61

SuperGLUE

• SuperGLUE’s DeBERTa score of 90.3 exceeded the 89.8 human-baseline average, though not on every task, suggesting the benchmark may need harder tests.SuperGLUE replaced GLUE with more challenging, diverse tasks; its score averages performance across the suite.Source: SuperGLUE · p. 62, Figure 2.3.1 · p. 62

SQuAD

• SQuAD 1.1 rose from an F1 score of 67.75 in August 2016 to 95.38 in 2020, while SQuAD 2.0 took 10 months to surpass human performance and reached 93.01 in 2020.SQuAD 1.1 surpassed the human score of 91.22 in September 2018; SQuAD 2.0 added over 50,000 unanswerable questions to test whether systems recognize when no answer exists.Source: SQuAD · p. 63, Figure 2.3.2 · p. 63

GPT-3

• GPT-3 has 175 billion parameters and can perform some tasks with zero or few examples, sometimes outperforming task-specific state-of-the-art models, but it can produce biased or inaccurate text and is costly to train.Across 42 benchmarks, average accuracy increased with model size in zero-, one-, and few-shot settings, with few-shot accuracy rising faster than zero-shot; examples of exceptions include SuperGLUE and SQuAD2.Source: GPT-3 · pp. 65–66

Visual Question Answering (VQA) Challenge

• VQA reached 76.4% accuracy in 2020, closer to the 80.8% human baseline and 1.1% above 2019, while VCR’s Q->AR score rose from 44 in 2018 to 70.5 in 2020.VQA answers natural-language questions about images; VCR additionally requires a rationale, and its 2020 score remained below the human QA->R score of 85.Source: Visual Question Answering (VQA)… · p. 67, Figure 2.4.1 · p. 67, Visual Commonsense Reasoning (VCR) Task · p. 68

Transcribe Speech: LibriSpeech

• The LibriSpeech Test Clean–Test Other performance gap narrowed from more than seven points in late 2015 to less than one in 2020, while VoxCeleb systems reached 99.4% speaker-attribution ability.The shrinking LibriSpeech gap indicates improved robustness but suggests the benchmark may be saturating; VoxCeleb systems still struggle with accents and very large speaker sets.Source: Transcribe Speech: LibriSpeech · p. 69, Speaker Recognition: VoxCeleb · p. 69, Figure 2.5.1a · p. 70

The Race Gap in Speech Recognition Technology

• 19 errors per 100 words for white speakers versus 35 for Black speakers; Black men exceeded 40, with gaps similar across companies.Researchers tested leading speech-to-text services using interview recordings from 42 white and 73 Black speakers.Source: Figure 2.5.3 · p. 71, Figure 2.5.4 · p. 71, The Race Gap in Speech Recognition… · p. 71

BOOLEAN SATISFIABILITY PROBLEM

• SAT solver performance improved sharply in 2019–2020: the 2019 median beat every prior-year top solver, and the 2020 median nearly matched 2019’s top solver.Temporal Shapley analysis found Kissat’s 2020 contribution highest among solvers apart from the first year, and concluded that novel techniques and more efficient implementations contributed equally to SAT’s state of the art.Source: BOOLEAN SATISFIABILITY PROBLEM · pp. 72, 74, Figure 2.6.1 · p. 73, Figure 2.6.2 · p. 73

AUTOMATED THEOREM PROVING (ATP)

• ATP systems solved an increasing fraction of test problems, but progress has slowed since 2015, with no recent rapid advances or breakthroughs.Axiom selection and search choice remain key challenges; machine learning approaches to both have grown, and recent competition results suggest they may be a game-changer.Source: AUTOMATED THEOREM PROVING (ATP) · pp. 74–75, Figure 2.6.3 · p. 75

Synthesis Planning

• Top-1 accuracy grew by 15.6% in 2020 from 2017; the latest molecular transformer scored 92% on top-1 accuracy in November 2020.Synthesis planning approaches use reaction text as machine translation or represent reactions as graph edits; they can score candidate molecules and produce routes to help chemists prioritize lab validation.Source: Figure 2.7.1 · p. 76, Synthesis Planning · p. 76, MOLECULAR SYNTHESIS · p. 76

COVID-19 AND DRUG DISCOVERY

• Moonshot received over 10,000 submissions from 365 contributors in four months; PostEra designed synthetic routes in under 48 hours, versus three to four weeks for human chemists.After the crowdsourcing phase ended toward late August 2020, the effort shifted to optimizing lead compounds and preparing for animal testing.Source: Figure 2.7.2 · p. 77, COVID-19 AND DRUG DISCOVERY · p. 77

ALPHAFOLD AND PROTEIN FOLDING

• AlphaFold and AlphaFold 2 won the last two CASP competitions with scores based partly on deep learning, whereas earlier winners typically used physics-based models.The comparison concerns median GDT scores for difficult freemodelling proteins.Source: Figure 2.7.3 · p. 78, ALPHAFOLD AND PROTEIN FOLDING · p. 78

EXPERT HIGHLIGHTS

• Experts most often named AlphaFold and GPT-3 as 2020’s most impressive AI advances and forecast pretrained models as a defining trend for 2021.Responses also highlighted transformer dominance across machine translation, NLP, and vision, as well as language generation’s research impact and remaining failings.Source: What was the single most impressive AI… · p. 79, What single trend will define AI in… · p. 79

ACCESS THE PUBLIC DATA

• The chapter examines AI’s economic relationship through worldwide AI labor demand, private investment, company adoption, robot installations, and corporate earnings mentions.It draws on LinkedIn, Burning Glass Technologies, CapIQ, Crunchbase, Quid, McKinsey, IFR, and Prattle data.Source: ACCESS THE PUBLIC DATA · p. 82

CHAPTER HIGHLIGHTS

• Private AI investment rose 9.3% in 2020 as newly funded companies fell for a third year; Drugs, Cancer, Molecular, Drug Discovery attracted more than USD 13.8 billion, 4.5 times its 2019 amount.The overall 9.3% increase exceeded 2019’s 5.7% increase.Source: CHAPTER HIGHLIGHTS · p. 83
• Despite growing calls to address AI ethics, companies’ efforts remained limited, with equity and fairness receiving comparatively little attention and privacy-risk mitigation unchanged.Fewer companies in 2020 viewed personal or individual privacy risks as relevant than in 2019.Source: CHAPTER HIGHLIGHTS · p. 83
• Half of McKinsey survey respondents said the pandemic had no effect on their company’s AI investment, 27% reported an increase, and less than a fourth reported a decrease.Source: CHAPTER HIGHLIGHTS · p. 83

AI HIRING

• Across 14 sampled countries, AI hiring in 2020 averaged 2.2 times its 2016 rate, with Brazil’s hiring index growing more than 3.5 times.Hiring continued to grow across the sampled countries during the pandemic; Brazil, India, Canada, Singapore, and South Africa had the highest growth from 2016 to 2020.Source: CHAPTER HIGHLIGHTS · p. 83, Figure 3.1.1 · p. 84, AI HIRING · p. 84

AI LABOR DEMAND

• Burning Glass compiled AI labor-demand data from postings on more than 45,000 online job sites, aggregating postings, removing duplicates, and extracting information from posting text.Its 2020 coverage added job sites, so the report cautions against direct comparison with 2019 report figures.Source: AI LABOR DEMAND · p. 86

Global AI Labor Demand

• More than five times as large on average, the share of AI job postings in six countries grew from 2013 to 2020; Singapore’s share grew 13.5 times.Burning Glass expanded its job-site coverage in 2020, so these figures should not be directly compared with the 2019 report.Source: Figure 3.1.3 · p. 86, Global AI Labor Demand · p. 86, AI LABOR DEMAND · p. 86
• The United States’ AI job postings fell 8.2%, from 325,724 in 2019 to 300,999 in 2020, and its share of postings recorded its first decline in six years.The United States was the only one of the six countries covered by Burning Glass to record a decrease in its share of AI job postings from 2019 to 2020.Source: CHAPTER HIGHLIGHTS · p. 83, Figure 3.1.3 · p. 86, Global AI Labor Demand · p. 86

U.S. AI Labor Demand: By Skill Cluster

• In 2020, the share of AI jobs among all U.S. job postings decreased across every skill cluster.Source: U.S. AI Labor Demand: By Skill Cluster · p. 87
• Machine-learning and artificial-intelligence jobs grew fastest, rising from 0.1% to 0.5% and 0.03% to 0.3% of total U.S. jobs, respectively, from 2013 to 2020.Each skill cluster groups related AI skills; for example, the neural-network cluster includes deep learning and convolutional neural networks.Source: Figure 3.1.4 · p. 87, U.S. AI Labor Demand: By Skill Cluster · p. 87

• Agriculture, forestry, fishing, and hunting had the biggest increase in AI job-posting share from 2019 to 2020, rising by almost 1 percentage point.Source: U.S. Labor Demand: By Industry · p. 88, Figure 3.1.6 · p. 89
• Information and professional, scientific, and technical services had consistently dominated U.S. demand for AI jobs.Source: U.S. Labor Demand: By Industry · p. 88
• Information (2.8%), professional, scientific, and technical services (2.5%), and agriculture, forestry, fishing, and hunting (2.1%) had the highest AI job-posting shares in 2020.Source: Figure 3.1.5 · p. 88, U.S. Labor Demand: By Industry · p. 88

U.S. Labor Demand: By State

• D.C. had the highest AI-posting share (1.88%), while California led in total postings (63,433).Figure 3.1.7 plots state shares against total AI job postings.Source: U.S. Labor Demand: By State · p. 90, Figure 3.1.7 · p. 90
• Six states besides D.C.—Washington, Virginia, Massachusetts, California, New York, and Maryland—had over 1% of job postings in AI, compared with five the prior year.Source: U.S. Labor Demand: By State · p. 90
• California’s 63,433 AI postings exceeded the next three states combined: Texas (22,539), New York (18,580), and Virginia (17,718).Source: U.S. Labor Demand: By State · p. 90

AI SKILL PENETRATION

• LinkedIn data on members’ listed skills, positions held, and position locations underlies the occupation-level measure.The metric averages the share of AI skills among each occupation’s top 50 skills.Source: AI SKILL PENETRATION · p. 91

Global Comparison

• India had the highest relative AI skill penetration at 2.83 times the global average, followed by the United States (1.99), China (1.40), Germany (1.27), and Canada (1.13).China and India had less than 40% LinkedIn workforce coverage, so their results may provide a less complete picture than other sampled countries.Source: Figure 3.1.8 · p. 91, Global Comparison · p. 91

Global Comparison: By Industry

• India had the highest relative AI skill penetration across the five leading industries, while Germany and Israel showed specialization in selected sectors.The five industries were education, finance, hardware and networking, manufacturing, and software and IT; Germany exceeded global-average penetration in hardware and networking and manufacturing, and Israel in manufacturing and education.Source: Figure 3.1.9 · p. 92, Global Comparison: By Industry · p. 92

CORPORATE INVESTMENT

• Global AI investment rose 40% to USD 67.9 billion in 2020, while private AI startup funding exceeded USD 40 billion but grew 9.3% as funded-startup counts fell.M&A made up most total investment and increased 121.7% from 2019; the number of funded AI startups declined for a third year, continuing a sharp drop from its 2017 peak.Source: Figure 3.2.1 · p. 93, CORPORATE INVESTMENT · p. 93, Global Trend · p. 94

Regional Comparison

• U.S. private AI investment led countries in 2020 at over USD 23.6 billion, while Drugs, Cancer, Molecular, Drug Discovery led focus areas at over USD 13.8 billion, 4.5 times its 2019 amount.China’s 2020 private investment was less than half the U.S. level, although the report notes substantial Chinese public AI R&D investment; gaming and education investment also rose, driven by U.S. and South Korean startups and a Chinese online education platform, respectively.Source: Regional Comparison · p. 95, Figure 3.2.6 · p. 97, Focus Area Analysis · p. 97

Global Adoption of AI

• Over 50% of survey respondents reported AI adoption in at least one business function, with developed Asia-Pacific leading regions and high tech and telecom leading industries.Adoption varied by industry and function; across industries, commonly adopted capabilities included other machine learning techniques, robotic process automation, and computer vision, while physical robotics and autonomous vehicles were associated with manufacturing and distribution industries and NLP with text-heavy data sectors.Source: Figure 3.3.1 · p. 98, Global Adoption of AI · p. 98, AI Adoption by Industry and Function · p. 99

Consideration and Mitigation of Risks from Adopting AI

• Cybersecurity was the only AI-adoption risk considered relevant by a majority, while half of respondents said COVID-19 did not affect AI investment and 27% increased it.Most changes in reported risks from 2019 were not statistically significant; companies gave comparatively little attention to equity and fairness, and fewer viewed privacy as a risk in 2020.Source: Consideration and Mitigation of Risks… · p. 101, The Effect of COVID-19 · p. 103, Figure 3.3.7 · p. 103

INDUSTRIAL ROBOT INSTALLATIONS

• Worldwide industrial robot installations fell 12% in 2019 to 373,240, ending six years of growth, while North America’s regional decline was 16%.Trade tensions and difficulties in automotive and electrical/electronics contributed to the global decline; China still had more industrial robots than the other four major country markets combined, and installations measure AI-ready infrastructure, not robots’ actual AI use.Source: Figure 3.3.8 · p. 104, Global Trend · p. 104, Regional Comparison · p. 105

EARNINGS CALLS

• AI mentions in corporate earnings calls were twice mentions of big data, cloud, and machine learning combined in 2020, though AI mentions fell 8.5% from 2019.The report’s following chapter examines AI education through a survey of higher-education institutions, computing-research graduate statistics, and European education data.Source: Figure 3.3.11 · p. 106, EARNINGS CALLS · p. 106, ACCESS THE PUBLIC DATA · p. 109

UNDERGRADUATE AI COURSE OFFERINGS

• Practical AI courses grew 102.9% at the undergraduate level and 41.7% at the graduate level across the surveyed universities in four academic years.AI-focused faculty grew 59.1%, from 105 in AY 2016–17 to 167 in AY 2019–20.Source: UNDERGRADUATE AI COURSE OFFERINGS · p. 111, Figure 4.1.1 · p. 111, GRADUATE AI COURSE OFFERINGS · p. 113

Undergraduate Courses That Teach AI Skills

• Almost 60% more students enrolled in or attempted to enroll in introductory AI and ML courses over four academic years.Enrollment in the European Union rose 165% over that period, while U.S. enrollment dipped in the last academic year; six of eight surveyed U.S. universities reported decreases, attributed by some to pandemic leaves or course changes.Source: UNDERGRADUATE AI COURSE OFFERINGS · p. 111, Undergraduate Courses That Teach AI… · p. 112

NEW CS PHDS BY SPECIALTY

• In 2019, AI/ML was the most popular U.S. CS PhD specialty at 22.8%, while AI/ML-focused graduates had grown 77% since 2010.More than 28,000 CS undergraduates completed degrees in North America in 2019, around three times the 2010 number; AI/ML and robotics/vision PhD graduates both reached all-time highs in 2019.Source: CS UNDERGRADUATE GRADUATES IN NORTH… · p. 114, NEW CS PHDS BY SPECIALTY · p. 115, NEW CS PHDS WITH AI/ML AND… · p. 117

Industry vs. Academia

• The share of new North American AI PhDs entering industry rose from 44.4% in 2010 to 65.7% in 2019, while the share entering academia fell from 42.1% to 23.7%.International students made up 64.3% of new North American AI PhDs in 2019; among foreign AI PhD graduates in the United States, 81.8% stayed for employment and 8.6% took jobs outside the country.Source: Industry vs. Academia · p. 118, Figure 4.2.5a · p. 118, AI PHDS · p. 119

AI OFFERINGS IN EU27

• The EU27 had 1,032 specialized AI programs, with the vast majority at master’s level; Germany offered the most programs overall.The Netherlands, France, and Sweden followed Germany, while France had the most AI master’s programs.Source: AI OFFERINGS IN EU27 · p. 120, Figure 4.3.1 · p. 120

By Content Taught in AI-Related Courses

• Robotics and automation were most common in specialized EU bachelor’s and master’s programs, while machine learning dominated short courses.AI ethics and AI applications each accounted for about 14% of curriculum on average; across the countries considered, the United States offered the most specialized AI programs, while the EU27 was a close second for AI master’s programs.Source: By Content Taught in AI-Related Courses · p. 121, Figure 4.3.2 · p. 121, INTERNATIONAL COMPARISON · p. 122

AI Brain Drain and Faculty Departure

• North American AI faculty departures to industry fell from 42 in 2018 to 33 in 2019, while research found departures chilled student-founded AI startups, especially those involving PhD students.The negative impact intensified when departing professors were replaced by faculty from lower-ranked schools or untenured AI professors; Carnegie Mellon had 16 departures from 2004 to 2019, followed by Georgia Tech with 14 and the University of Washington with 12.Source: AI Brain Drain and Faculty Departure · p. 123, AI Brain Drain and Faculty Departure… · p. 124, Figure 4.4.2 · p. 124

ACCESS THE PUBLIC DATA

• The report found little quantitative data for assessing how broader societal discussions about AI relate to technology development, making measurable indicators a research challenge.It notes that translating qualitative ethical arguments into quantitative data is important for policymakers, who can more readily manage what they can measure.Source: ACCESS THE PUBLIC DATA · p. 127

5.1 AI PRINCIPLES AND FRAMEWORKS

• 117 AI-principles documents appeared from 2015 to 2020, with research and professional organizations among the earliest publishers and private companies producing the most; critics say these principles are often vague and non-binding.The report identifies privacy, accountability, transparency, and explainability as common themes; 2018 was a high-water mark for ethics-principle releases by technology companies and government agencies.Source: §5.1 · p. 129

5.2 GLOBAL NEWS MEDIA

• In 2020, AI ethics guidance and frameworks received 21% of coverage, while the European Commission white paper led individual stories at 5.9%, followed by the Gebru dismissal at 3.5%, UN committee at 2.7%, Vatican plan at 2.6%, and IBM exit at 2.5%.Research and education and facial recognition each received 20% of coverage.Source: §5.2 · p. 131, Figure 5.2.1 · p. 131

5.3 ETHICS AT AI CONFERENCES

• Ethics-keyword paper-title submissions more than doubled from 2015 to 2020, but average matches across six major AI conferences remained low; from 2020, NeurIPS required Broader Impacts statements.The report also notes a proliferation of conferences and workshops focused on responsible AI.Source: §5.3 · p. 132, Figure 5.3.1 · p. 133, Figure 5.3.2 · p. 133

5.4 ETHICS OFFERINGS AT HIGHER EDUCATION INSTITUTIONS

• Among 18 surveyed departments, 11 hosted AI-ethics keynotes or panels and 7 offered standalone AI-ethics courses.Some universities also offered general computer-science ethics courses or embedded ethics modules.Source: §5.4 · p. 134, Figure 5.4.1 · p. 134

ACCESS THE PUBLIC DATA

• Sparse public demographic data limits analysis of AI workforce diversity, while the chapter notes that underrepresentation risks reinforcing inequalities and unjust outcomes.The statistics omit some dimensions, including disability and intersectional diversity, and do not capture the daily challenges or structural problems faced by marginalized groups.Source: ACCESS THE PUBLIC DATA · p. 137

CHAPTER HIGHLIGHTS; BLACK IN AI

• Black in AI workshop participation grew: 2019 attendance and submitted-paper counts were 2.6 times 2017 levels, and accepted papers were 2.1 times higher; Queer in AI survey respondents reported barriers and discrimination.Almost half of Queer in AI members surveyed in 2020 viewed lack of inclusiveness as an obstacle, and more than 40% reported discrimination or harassment at work or school.Source: CHAPTER HIGHLIGHTS · p. 138, BLACK IN AI · p. 146, Figure 6.2.4 · p. 146

WOMEN IN ACADEMIC AI SETTINGS; WOMEN IN THE AI WORKFORCE

• Female AI and CS PhD graduates averaged 18.3% over the past decade, and women made up 16.1% of surveyed tenure-track faculty focused on AI; women’s AI-skills penetration was lower in most examined countries.India, South Korea, Singapore, and Australia were closest to gender equity in AI-skills penetration among the 12 countries examined.Source: WOMEN IN ACADEMIC AI SETTINGS · p. 139, Figure 6.1.1 · p. 139, Figure 6.1.2 · p. 139

NEW AI PHDS IN THE UNITED STATES BY RACE/ETHNICITY; CS TENURE-TRACK FACULTY BY RACE/ ETHNICITY

• Among new U.S.-resident AI PhDs in 2019, 45.6% were white and 22.4% Asian, while Black and Hispanic representation was 2.4% and 3.2%; among new computing PhDs, Black and Hispanic shares averaged 3.1% and 3.3%.Across 15 universities, tenure-track faculty were approximately 67.0% white, 14.3% Asian, 0.6% Black or African, and 0.8% Hispanic, Latino, or Spanish; comparisons of 2019 AI and CS PhDs were limited by unknown cases.Source: NEW AI PHDS IN THE UNITED STATES BY… · pp. 144–145, Figure 6.2.1 · p. 144, Figure 6.2.2 · p. 145

BLACK IN AI

• 18 of 73 universities completed the Black in AI survey, a 24.7% response rate; 2020 data were excluded because they were affected by the pandemic.Source: BLACK IN AI · p. 146

Demographics Breakdown

• About 31.5% of QAI respondents identify as gay; 37.0% as cis male, 26.1% as cis female, and students average 41.7% of respondents.Trans female and male respondents account for 5.0% and 2.5%, respectively; gender queer, gender fluid, nonbinary, and other identities are also reported.Source: Demographics Breakdown · p. 147, Figure 6.3.1 · p. 147, Figure 6.3.2 · p. 148

Experience as Queer Practitioners

• 81.4% cite a lack of role models and 70.9% a lack of community as major career obstacles; more than 40% report discrimination or harassment at work or school.Almost half also view lack of inclusiveness as an obstacle, and around 9.7% report discrimination or harassment on more than five occasions.Source: Experience as Queer Practitioners · pp. 149–150, Figure 6.3.4 · p. 149, Figure 6.3.5 · p. 150

CHAPTER HIGHLIGHTS

• In 2020, the Global Partnership on AI and OECD AI Policy Observatory and Network of Experts on AI promoted intergovernmental cooperation to support AI development for all.Source: CHAPTER HIGHLIGHTS · p. 154
• The 116th U.S. Congress was the most AI-focused congressional session in history, with AI mentions in legislation and reports more than triple those of the 115th Congress.Source: CHAPTER HIGHLIGHTS · p. 154

7.1 NATIONAL AND REGIONAL AI STRATEGIES

• More than 30 countries and regions had published AI strategy documents by December 2020, after Canada published the first national AI strategy in 2017.The report defines an AI strategy as a policy document supporting AI development while maximizing its benefits for society.Source: CHAPTER HIGHLIGHTS · p. 154, §7.1 · p. 155

National AI Strategies and Human Rights

• Among 30 states and two regional strategies, privacy was the most commonly mentioned human right, followed by equality and nondiscrimination, but few strategies deeply assessed AI’s human-rights impacts.The report found little specificity on how human rights should be protected in AI, unlike the detail given to economic competitiveness and innovation.Source: National AI Strategies and Human Rights · p. 164, Table 7.1.1 · p. 164

7.2 INTERNATIONAL COLLABORATION ON AI

• International AI initiatives use global organizations, partnerships, and bilateral agreements, with AI ethics a prominent intergovernmental focus; UNESCO’s expert group sent a revised draft ethics recommendation to member states for comment.The EU High-Level Expert Group also presented trustworthy-AI ethics guidelines, policy and investment recommendations, and an assessment checklist.Source: §7.2 · p. 165, Ad Hoc Expert Group (AHEG) for the… · p. 166, Working Groups · p. 166

FEDERAL BUDGET FOR NON-DEFENSE AI R&D

• FY 2021 civilian agencies budgeted USD 1.5 billion for AI R&D, almost 55% above the FY 2020 request; FY 2020 funding was USD 973.5 million, or USD 1.1 billion with appropriations and transfers.These figures exclude substantial Department of Defense and intelligence-sector investments withheld for national-security reasons.Source: Figure 7.3.1 · p. 167, FEDERAL BUDGET FOR NON-DEFENSE AI R&D · p. 167

U.S. DEPARTMENT OF DEFENSE AI R&D BUDGET REQUEST

• USD 5.0 billion was requested for unclassified DOD AI R&D programs in FY 2021, consistent with the prior year’s enacted amount, but the request was not a final budget.DOD’s own accounting gave USD 841 million, illustrating how estimates vary with what spending is counted as AI-related; appropriations could also raise the request.Source: Figure 7.3.2 · p. 168, U.S. DEPARTMENT OF DEFENSE AI R&D… · p. 168

Contract Spending by Department and Agency

• In FY 2020, DOD led federal agencies in AI-related contracts with USD 1.4 billion; its USD 3.9 billion total for 2001–2020 exceeded the other 44 agencies’ combined USD 2.9 billion.NASA and the Department of Homeland Security followed DOD in FY 2020, with USD 139.1 million and USD 112.3 million, respectively.Source: Contract Spending by Department and… · p. 169

Total Contract Spending

• USD 1.8 billion in unclassified AI-related federal contracts in FY 2020 was more than 25% above FY 2019 and over six times FY 2015 spending, yet represented 0.25% of federal contracts.The government spent USD 682 billion on contracts overall in FY 2020.Source: Figure 7.3.3 · p. 169, Total Contract Spending · p. 169

U.S. Congressional Record

• The 116th Congress was the most AI-focused U.S. session, with 173 AI-related legislative pieces, 70 committee reports, and 243 CRS reports; AI and ML mentions rose in U.S. and U.K. proceedings but fell in Canada.The session’s AI mentions in legislation, committee reports, and CRS reports were more than triple those of the 115th Congress.Source: U.S. Congressional Record · p. 172, Parliamentary Proceedings · p. 172

CENTRAL BANKS

• AI mentions across 16 central banks peaked at 1,020 in 2019 before declining sharply in 2020 as communications focused on COVID-19’s economic effects.The Federal Reserve, Norges Bank, and European Central Bank had the most aggregated mentions over the prior five years.Source: CENTRAL BANKS · p. 174

U.S. AI POLICY PAPERS

• U.S. AI policy papers focused mainly on innovation and technology, international affairs and security, and industry and regulation; ethics-related topics were largely secondary.Physical sciences, energy and environment, humanities, and democracy received the least attention.Source: U.S. AI POLICY PAPERS · p. 176, Figure 7.4.7 · p. 176

Nuances specific to methodology

• Elsevier considers post-1995 Scopus papers more reliable, while its AI-paper collection was conducted internally without AI Index involvement.The report notes that AI boundaries are difficult to define and calls for greater methodological transparency to support reproducibility.Source: Nuance · p. 179, Nuances specific to methodology · p. 180

AI Training Set

• Elsevier’s AI training set comprises approximately 1,500 publications randomly selected from seven million and checked against an expert-verified gold set.The training set identifies publications by their Scopus identifiers.Source: AI Training Set · p. 180

Methodology

• MAG equally distributes paper credit among unique regions, but 30% of journal and conference affiliation data lacks a country or region.Papers with multiple regional affiliations are divided equally among those regions.Source: Methodology · p. 181, Notes About the MAG Data · p. 181

Notes About the MAG Data

• In 2020, the top 20 AI conferences presented 103,000 papers (13.7% of all AI conference papers) and received 7.15 million citations (47% of all such citations).Some 2020 conference papers were not properly tagged; data is known to be missing for ICCV and NAACL.Source: Notes About the MAG Data · p. 181

MICROSOFT ACADEMIC GRAPH: PATENT DATA CHALLENGE

• MAG patent counts can misstate regional or geographic AI patent shares because affiliations may be inaccurate and multiple filings can represent one invention.MAG uses patent family IDs to count related filings once, but over-conflation is more noticeable for patents than scholarly articles.Source: MICROSOFT ACADEMIC GRAPH: PATENT DATA… · p. 182, By Region · p. 182

AI Journal Publications (OECD Definition)

• OECD’s broader AI definition includes machine-learning fields and subtopics, unlike MAG’s AI-only definition; under the original dataset, only 25% of 2020 AAAI conference publications were included.Charts using the OECD definition show overall trends very similar to those under the main definition.Source: Citation · p. 184, AI Journal Publications (OECD… · p. 184

Nuance

• arXiv categories are author-selected rather than automatically assigned, and participation may amplify over-indexed community representation in growing subcategories.Authors identify the primary category, and AI or machine-learning work may also be categorized under other subfields or keywords.Source: Nuance · p. 192

Source

• ImageNet results use top-1 accuracy from each paper’s latest version on the validation set, but rankings can vary with evaluation set, metric, and crop method.For papers from 2014 onward, reported results use a single model and single-crop testing; the three earliest models use ensembles.Source: Source · p. 194
• The report surveys ImageNet top-1 and top-5 progress using selected papers, distinguishing scores with and without extra training data.The listed papers supply top-5 scores in s311.p4 and s311.p5, and top-1 scores in s311.p6 and s311.p7.Source: Source · pp. 194–195

IMAGENET: TRAINING TIME

• ImageNet training-time trends come from studies optimizing time to a standard accuracy, while MLPerf is identified as the most reliable updated compute source after DAWNBench’s March 2020 deprecation.DAWNBench measures training time and cost as well as inference latency and cost; MLPerf results are sourced from its 2018, 2019, and 2020 training releases.Source: IMAGENET: TRAINING TIME · p. 196, Note · p. 196, Source · p. 196

COCO: KEYPOINT DETECTION

• Other vision progress sources include the COCO keypoints leaderboard, DensePose challenge and arXiv materials, ActivityNet temporal-localization trends, and surveyed YOLO versions.The source describes YOLO mAP increasing with model size alongside decreasing video FPS.Source: COCO: KEYPOINT DETECTION · p. 196, COCO: DENSEPOSE ESTIMATION · p. 196, ACTIVITYNET: TEMPORAL LOCALIZATION TASK · p. 197

VISUAL QUESTION ANSWERING (VQA)

• VQA data includes more than 250,000 images and 1.1 million questions, while VCR evaluates question answering, rationale selection, and their combined accuracy.VQA questions have ten concise open-ended answers each, and ten humans answered each question; VoxCeleb spans more than 7,000 speakers and over a million utterances.Source: Methodology · p. 198, VISUAL COMMONSENSE REASONING (VCR) · p. 199, VOXCELEB · p. 200

Methodology

• Papers and code tracking follows selected arXiv categories and major ML conferences, but misses some paywalled application areas; state-of-the-art tables combine scraped, hand-annotated, and user-contributed data.The extraction process searches paper sources and GitHub, cross-checks links, and lets the community correct errors or add missing values.Source: Methodology · p. 198, Process of Extracting Dataset at Scale · p. 198

Nuances about FRVT evaluation metrics

• FRVT Wild results have identity-label errors and difficult convergence, contain test data only, and should be interpreted separately for 1:1 verification and 1:N identification.Fast search algorithms matter especially for 1:N, whereas speed is not a factor in 1:1.Source: Nuances about FRVT evaluation metrics · p. 199

Results

• Temporal Shapley accounts for solver introduction time, while ATP progress is indicated by when TPTP problems are first solved.Shapley value averages a solver’s marginal performance contribution across solver sets (Fréchette et al. (2016)); temporal Shapley addresses the disadvantage of comparing later solvers with earlier ones (Kotthoff et al. (2018)). TPTP difficulty ratings track how many ATP systems solve a problem, and historical ratings identify its first solve.Source: Results · p. 201, §2 · pp. 201–202

AI Skills Penetration

• LinkedIn AI skills penetration measures AI-group skills among an entity’s 50 most representative skills, while country and industry comparisons benchmark penetration across overlapping occupations.The 50 skills are selected using TF-IDF weighting; global comparisons use average penetration across the same occupations in a country sample.Source: AI Skills Penetration · p. 203, Global Comparison · p. 203, Global Comparison: By Industry · p. 204

LinkedIn AI Hiring Index

• LinkedIn’s AI hiring index tracks monthly hiring of identified AI talent and indexes rates against 2016; China and India are included despite LinkedIn coverage below 40%.Eligible countries generally require at least 40% labor-force coverage and 10 AI hires in a month, so China and India may be less fully represented.Source: Country Sample · p. 203, LinkedIn AI Hiring Index · p. 204

Measuring Demand for AI

• Burning Glass analyzes postings from over 45,000 job sites, and classifies a posting as an AI job when it requests at least one skill from its AI taxonomy.The taxonomy covers over 17,000 skills; for this report, all listed skills were treated as AI skills.Source: Job Posting Data · p. 204, Measuring Demand for AI · p. 205

Job Posting Data

• After adjustment, Burning Glass captured roughly 85% of JOLTS postings in 2016, and its data represent over 85% of total labor demand.Burning Glass counts new postings, whereas JOLTS counts active postings; the estimate adjusts for that difference.Source: Job Posting Data · p. 205

Search, Data Sources, and Scope

• NetBase Quid analyzes company descriptions and related text across 3.6 million profiles, combining CapIQ and Crunchbase data and visualizing up to 7,000 relevant search results.Company information is updated weekly; Boolean searches can be filtered by region, investment, operating status, organization type, and founding year.Source: Search, Data Sources, and Scope · p. 206, Data · p. 206, Methodology · p. 206

Methodology

• McKinsey’s June 2020 online survey received 2,395 responses, including 1,151 respondents from organizations reporting AI adoption; responses were weighted by nations’ shares of global GDP.Respondents’ answers are limited by their perceptions of their organizations’ AI adoption.Source: Methodology · p. 207, Note · p. 207

Nuance

• The IFR indicator counts industrial robots installed by country, but the share running AI software and AI’s contribution to their use are unclear.Industrial robots are defined by ISO 8373:2012.Source: Methodology · p. 207, Nuance · p. 207

Nuances

• CRA Taulbee surveys doctoral departments rather than students, and its reported new hires may overstate the number of genuinely new hires.The survey’s AI specialization categories changed over time; in 2018, 29% of new assistant professors came from another academic institution, though some may have been non-tenure-track faculty.Source: Methodology · p. 208, Nuances · p. 208

Methodology

• The education study uses text-mining and an expert-validated, expanded keyword list to identify programs, group their content areas, and classify them as specialized or broad.The keyword approach draws on AI-journal terms, industrial terms, and topic modeling, and adds coverage of subdomains and transversal areas such as AI ethics.Source: Methodology · p. 210
• Because Studyportals mainly tracks English-language programs, country comparisons may be uneven and the study may show only part of the inclusion of advanced digital skills in bachelor’s programs.The database includes over 207,000 programs from 3,700 universities in more than 120 countries; bachelor’s offerings are expected to be more affected than master’s programs.Source: Methodology · p. 210

Visualization in Quid Software

• Quid’s news analysis searched global news from 2020 for combinations of AI-technology and ethics terms, then selected 10,000 relevant stories and visualized de-duplicated articles.Its indexed news and blogs are English-language sources from LexisNexis.Source: Network · p. 211, News Dataset Data Source · p. 211, Visualization in Quid Software · p. 211

Conference and Public Venue - Sample

• The ethics-in-AI measure counts papers matching classical, trending, or ethics keywords in titles or abstracts, and category percentages can overlap.The sampled AI venues include AAAI, IJCAI, ICML, and NIPS; robotics venues include IEEE Transactions on Robotics, ICRA, and IROS; and the CS sample includes mainstream computing venues.Source: Methodology · p. 212, Conference and Public Venue - Sample · p. 213

AI Skills Penetration

• AI skill penetration measures an entity’s AI-skill intensity as the share of its top 50 representative skills in the AI skill group, after TF-IDF re-weighting LinkedIn members’ self-added skill frequencies from 2015–2020.Source: AI Skills Penetration · p. 214

Global Comparison: By Gender

• A country’s relative AI skill penetration compares its rate with a benchmark while controlling for occupations; gender measures decompose the female and male labor pools and compare rates across the same occupations.For example, relative rates of 2 for women and 1.9 for men imply women’s rate is about 5% higher in that country.Source: Global Comparison: By Gender · p. 214

Methodology

• BGOV used contract and defense-budget data to identify AI-related government spending, including 305 unique AI- and machine-learning-related budget activities worth USD 5.0 billion in FY 2021.The contract-spending model used government product-service codes and more than 100 AI-related keywords and acronyms.Source: Methodology · p. 215

Organizations

• The policy-paper review tracked thought leadership from 36 organizations across think tanks, policy institutes and academia; civil society, associations and consortiums; and industry and consultancy.Source: Organizations · p. 216

OVERVIEW

• The tool covers 22 metrics: 14 for research and development, 6 for the economy, and 2 for gender diversity under the inclusion pillar.It is available as a web tool for data-driven national policy strategy.Source: OVERVIEW · p. 218

Only per capita metrics

• Users can choose equal, absolute-only, or per-capita-only weighting, adjust individual metric weights, and switch between global comparisons and national country deep dives.The national view uses scaled 0–100 country-metric values, where 100 is the highest value in the global distribution for that metric.Source: OVERVIEW · p. 219, All weights to midpoint · p. 219, Only absolute metrics · p. 219

Methodology

• The methodology harmonizes country and time data, normalizes indicators to 0–100, averages country-indicator values by year, and builds weighted measures for available pillars.Countries were selected using a 73% average data-coverage threshold for 2015–2020, with Russia and South Korea added manually despite not meeting it.Source: Source · p. 219, Normalization · p. 219, Coverage and Nuances · p. 219

RESEARCH AND DEVELOPMENT INDICATORS

• The 14 research and development indicators include country-level AI conference and journal publications, patents, and deep-learning papers, with total and per-capita measures represented.Source: RESEARCH AND DEVELOPMENT INDICATORS · p. 220

ECONOMY INDICATORS

• The economy indicators cover relative AI skill penetration, AI hiring, private AI-startup investment, and AI company counts, including per-capita measures for investment and companies.Source: ECONOMY INDICATORS · p. 221