Artificial Intelligence Index Report 2024

Top 10 Takeaways

• AI exceeds human performance on some image-classification, visual-reasoning, and English-understanding benchmarks, but trails on competition-level mathematics, visual commonsense reasoning, and planning.Source: Top 10 Takeaways · p. 5

2. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable

• Industry produced 51 notable machine-learning models in 2023, compared with academia’s 15, while industry-academia collaborations produced a record 21.Source: §2 · p. 5

3. Frontier models get way more expensive. According to AI Index estimates, the training costs

• Estimated compute costs reached $78 million for GPT-4 and $191 million for Gemini Ultra, illustrating the rising cost of training state-of-the-art models.Source: §3 · p. 5

5. Robust and standardized evaluations for LLM responsibility are seriously lacking.

• Responsible-AI reporting lacks standardization: leading developers test models against different benchmarks, complicating systematic comparisons of their risks and limitations.Source: §5 · p. 5

6. Generative AI investment skyrockets. Despite a decline in overall AI private investment last

• Generative AI funding nearly octupled to $25.2 billion in 2023, while labor studies found faster task completion and higher-quality work but warned that inadequate oversight can reduce performance.Studies also indicated potential to narrow skill gaps between lower- and higher-skilled workers.Source: §6 · p. 5

8. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance

• AI advances are enabling scientific applications such as AlphaDev’s more efficient algorithmic sorting and GNoME’s materials discovery, while U.S. AI-related regulations reached 25 in 2023, up 56.3% year over year.The U.S. total was one regulation in 2016.Source: §8 · p. 6, §9 · p. 6

10. People across the globe are more cognizant of AI’s potential impact—and more nervous.

• Public concern is rising: 66% expect AI to dramatically affect their lives in three to five years, 52% feel nervous about AI products and services, and 52% of Americans feel more concerned than excited.The respective comparison points were 60% the prior year, a 13-percentage-point rise in nervousness since 2022, and 37% of Americans in 2022.Source: §10 · p. 6

8. Closed LLMs significantly outperform open ones. On 10 select AI benchmarks, closed models

• Closed models outperformed open models by a median performance advantage of 24.2%; among newly released models, 65.7% were open-source, versus 44.4% in 2022 and 33.3% in 2021.The number of models released was more than double the amount released in 2022.Source: §2 · p. 14, §8 · p. 16

1. Robust and standardized evaluations for LLM responsibility are seriously lacking.

• Different responsible-AI benchmarks hinder systematic model-risk comparisons, while limited disclosure of training data and methods impedes research on robustness and safety; LLM outputs may also raise copyright questions.The report also notes that popular LLMs may reproduce copyrighted material, making the legal status of such outputs a central question.Source: §1 · p. 17, §6 · p. 17

Chapter 3: Responsible AI (cont’d)

• AI incident reports reached 123 in 2023, a 32.3 percentage point increase from 2022; researchers also found political bias in ChatGPT, while deepfakes and less-obvious prompts expose risks.The report describes fake content as easy to generate and disseminate, and notes that apparently nonsensical prompts can elicit harmful LLM behavior; scholars disagree on how to weigh immediate risks against existential threats.Source: §2 · p. 17, Chapter 3: Responsible AI (cont’d) · p. 18

1. Generative AI investment skyrockets. Despite a decline in overall AI private investment last year,

• Generative AI funding nearly octupled to $25.2 billion in 2023; U.S. AI investment reached $67.2 billion, nearly 8.7 times China’s, as surveyed firms reported cost reductions and revenue increases.The United States’ investment rose 22.1% from 2022, while investment in China and the European Union, including the United Kingdom, fell 44.2% and 14.1%, respectively. The survey found 59% reported revenue increases and a 10 percentage point rise in respondents reporting decreased costs.Source: §1 · p. 19, §2 · p. 19, §4 · p. 19

6. AI organizational adoption ticks up. A 2023 McKinsey report reveals that 55% of organizations now

• AI use reached 55% of organizations in 2023, while studies found it can improve worker speed and output quality, and China’s share of global industrial robot installations rose to 52.4% by 2022.In the United States, AI-related job postings fell from 2.0% of all postings in 2022 to 1.6% in 2023; Fortune 500 earnings calls mentioned AI 394 times in 2023, with generative AI the most frequent theme.Source: §6 · p. 19, §2 · p. 19, §9 · p. 20

3. Highly knowledgeable medical AI has arrived.

• In 2023, AI applications included AlphaDev for more efficient algorithmic sorting and GNoME for materials discovery; GPT-4 Medprompt reached 90.2% accuracy, while FDA-approved AI-related devices rose more than 45-fold since 2012.Source: §1 · p. 21, §3 · p. 21, §4 · p. 21

2. The migration of AI PhDs to industry continues at an accelerating pace.

• Industry attracted 70.7% of new AI PhDs in 2022 versus 20.0% entering academia, while U.S. and Canadian CS master’s and PhD graduates slightly declined since 2018.The share of new AI PhDs joining industry rose 5.3 percentage points over the past year; international CS graduates also declined in 2022, especially at the master’s level.Source: §2 · p. 22, §4 · p. 22

1. The number of AI regulations in the United States sharply increases.

• AI-related regulations in the United States grew to 25 in 2023 from one in 2016, while global legislative proceedings mentioning AI rose from 1,247 in 2022 to 2,175 in 2023.The European Union reached a deal on the AI Act, the U.S. President signed an Executive Order on AI, and 21 U.S. regulatory agencies issued AI regulations in 2023, up from 17 in 2022.Source: §4 · p. 24, §2 · p. 24, §5 · p. 24

1. People across the globe are more cognizant of AI’s potential impact—and more nervous.

• Public concern and awareness increased: 66% expected AI to dramatically affect their lives within three to five years, 52% of Americans felt more concerned than excited, and 63% knew of ChatGPT.Only 34% expected AI to boost the economy, while younger respondents were more optimistic about AI’s effects on entertainment than baby boomers (59% versus 40%).Source: §1 · p. 26, §5 · p. 26, §2 · p. 26

1. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable

• Industry produced 51 notable machine learning models in 2023, compared with academia’s 15; industry-academia collaborations produced 21 models, a new high.Source: §1 · p. 30

2. More foundation models and more open foundation models. In 2023, a total of 149 foundation

• Newly released models were more than twice the 2022 amount; 65.7% were open-source, compared with 44.4% in 2022 and 33.3% in 2021.Source: §2 · p. 30
• GPT-4’s estimated compute training cost was $78 million, while Gemini Ultra’s was $191 million.Source: §2 · p. 30
• U.S.-based institutions originated 61 notable AI models in 2023, compared with 21 from the European Union and 15 from China.Source: §2 · p. 30

6. China dominates AI patents. In 2022, China led global AI patent origins with 61.1%, significantly

• GitHub AI projects grew from 845 in 2011 to approximately 1.8 million in 2023, rising 59.3% that year; project stars more than tripled to 12.2 million.Stars rose from 4.0 million in 2022 to 12.2 million in 2023.Source: §6 · p. 30

Total Number of AI Publications1

• AI publications nearly tripled from approximately 88,000 in 2010 to more than 240,000 in 2022, with a 1.1% rise in the final year.In 2022, there were roughly 230,000 AI journal articles and 42,000 conference submissions; since 2015, conference publications rose 2.6-fold and journal publications 2.4-fold. Publication counts use revised CSET methods and are available only through 2022, so the report advises caution.Source: §6 · p. 30, Total Number of AI Publications1 · p. 31, By Type of Publication · p. 32

By Field of Study

• Machine learning publications grew nearly sevenfold since 2015; in 2022, computer vision, pattern recognition, and process management had 21,309, 19,841, and 12,052 publications, respectively.Source: By Field of Study · p. 33, Figure 1.1.3 · p. 33

By Sector

• The academic sector produced 81.1% of AI publications in 2022 and remained the leading global research source over the past decade.Industry participation was most significant in the United States, followed by the European Union plus the United Kingdom, and China.Source: By Sector · p. 34, Figure 1.1.5 · p. 35

AI Journal Publications

• AI journal publications grew approximately 2.4 times since 2015 and increased 4.5% from 2021 to 2022, while conference publications reached 41,174 in 2022, up 30.2% year over year.Conference publications rose from 22,727 in 2020 to 31,629 in 2021 and have more than doubled since 2010; journal and conference publications grew at comparable rates since 2015.Source: AI Journal Publications · p. 36, Figure 1.1.6 · p. 36, AI Conference Publications · p. 37

Overview

• AI patent grants increased 62.7% from 2021 to 2022, after growing 56.1% over 2010–14.The count of granted AI patents has increased more than 31 times since 2010.Source: §5 · p. 30, Overview · p. 38, Figure 1.2.1 · p. 38

By Filing Status and Region

• By 2022, 67.4% of AI patent filings were ungranted, up from 42.2% in 2015; ungranted patents numbered 128,952 versus 62,264 granted.Since 2015, most filings have not been granted, and the gap has widened.Source: By Filing Status and Region · p. 39
• China accounted for 61.1% of granted AI patents in 2022, versus 20.9% for the United States, whose share fell from 54.1% in 2010.East Asia and the Pacific held 75.2% of granted patents and North America 21.2%; South Korea led per-capita grants at 10.3 per 100,000, followed by Luxembourg (8.8) and the United States (4.2). Singapore, South Korea, and China had the greatest per-capita increases from 2012 to 2022.Source: By Filing Status and Region · pp. 41–43, Figure 1.2.5 · p. 42, Figure 1.2.6 · p. 43

Overview

• The model dataset selects entries for state-of-the-art advances, historical significance, or high citation rates, but may omit models.The report says the dataset can still reveal trends in relative terms.Source: Overview · p. 45

Sector Analysis

• Since 2014, industry has led notable-model releases; in 2023 it produced 51 versus academia’s 15, while collaborations reached a record 21.The report says the industry-academia gap narrowed slightly but that the trend largely persisted.Source: Sector Analysis · p. 46, Figure 1.3.1 · p. 46

National Affiliation

• In 2023, the United States produced 61 notable models, China 15, and France 8; the EU and UK together surpassed China for the first time since 2019.The report attributes models to researchers’ institutional affiliations; models with authors from multiple countries can be double-counted.Source: National Affiliation · p. 47, Figure 1.3.3 · p. 47

Parameter Trends

• Model parameter counts have risen sharply since the early 2010s, with high-parameter models particularly prominent in industry.The report links rising counts to task complexity, data availability, hardware improvements, and the demonstrated efficacy of larger models; it says companies can bear the associated training costs.Source: Parameter Trends · p. 49

Compute Trends

• Training compute for notable AI models has risen exponentially, especially in the last five years.The report notes that models requiring more computation often have larger environmental footprints, and companies typically have more access to compute than academic institutions.Source: Compute Trends · p. 50, Figure 1.3.6 · p. 50
• By 2024, high-quality language data could be depleted; low-quality language data may be exhausted within two decades, and image data by the late 2030s to mid-2040s.The projections use historical growth in foundation-model training data, with compute-based projections adjusting that growth for expected compute availability.Source: Compute Trends · p. 52
• Models predominantly trained on synthetic data can undergo model collapse, losing underlying data distributions and producing a narrower range of outputs.The passages describe this as a limitation of synthetic-data training.Source: Compute Trends · p. 52
• As training generations increase, generated-image quality declines under both fully synthetic training and synthetic augmentation.The report also notes that image models trained only on synthetic-data cycles, or with insufficient real human data, experience a significant quality drop.Source: Compute Trends · p. 54, Figure 1.3.10 · p. 54
• Synthetic-data images showed higher FID scores, lower precision, and lower recall; synthetic augmentation loops degraded less than fully synthetic loops, but both had diminishing returns with further training.The measures indicate lower similarity to real images, reduced realism or quality, and decreased diversity, respectively.Source: Compute Trends · p. 54, Figure 1.3.12 · p. 55

Foundation Models

• The section uses Stanford’s Ecosystem Graphs, which track datasets, models, and applications, to study foundation-model trends over time.Source: Foundation Models · p. 56

Model Release

• Foundation models increased more than twofold since 2022 and nearly 38-fold since 2019; in 2023, 65.8% were open access, 18.8% had no access, and 15.4% had limited access.The share released with open access has risen significantly since 2021.Source: Model Release · pp. 56–57, Figure 1.3.13 · p. 56, Figure 1.3.14 · p. 57

Organizational Affiliation

• Industry produced 72.5% of foundation models in 2023, compared with 18.8% from academia; Google led 2023 releases with 18, while it led releases since 2019 with 40.Meta released 11 and Microsoft 9 models in 2023; UC Berkeley led academic institutions with 3. Since 2019, OpenAI followed Google with 20 releases, Tsinghua had 7, and Stanford had 5.Source: Organizational Affiliation · pp. 58–60, Figure 1.3.16 · p. 59, Figure 1.3.17 · p. 60

National Affiliation

• The United States originated 109 foundation models in 2023, versus China’s 20, and led cumulative releases since 2019 with 182, ahead of China’s 30 and the United Kingdom’s 21.National affiliation is assigned according to the country where a contributing researcher’s institution is headquartered.Source: National Affiliation · pp. 61–62, Figure 1.3.18 · p. 61, Figure 1.3.20 · p. 62

Training Cost

• In 2023, estimated training costs were $78 million for GPT-4 and $191 million for Gemini Ultra, compared with around $900 for the 2017 Transformer.RoBERTa Large cost around $160,000 to train in 2019; models with greater computational training needs cost substantially more.Source: Training Cost · pp. 63, 65, Figure 1.3.21 · p. 64, Figure 1.3.23 · p. 65

Conference Attendance

• Conference attendance rose 6.7% from 2022 to 2023, and annual attendance had increased by around 50,000 since 2015.NeurIPS drew approximately 16,380 participants in 2023; NeurIPS, ICML, ICCV, and AAAI attendance increased year over year, while CVPR, ICRA, ICLR, and IROS declined slightly. Attendance figures warrant caution because virtual and hybrid attendance is difficult to measure.Source: Conference Attendance · pp. 66–67, Figure 1.4.1 · p. 66

Projects

• GitHub AI projects grew from 845 in 2011 to approximately 1.8 million in 2023, including a 59.3% increase in the last year.Source: Projects · p. 69, Figure 1.5.1 · p. 69
• GitHub AI-project identification and geographic mapping methods changed, so this edition’s data may not align perfectly with earlier report editions.The updated classification incorporates generative AI keywords, and geographic mapping uses GitHub server-side data rather than OECD mapping based on self-reported data.Source: Projects · p. 69
• In 2023, the United States accounted for 22.9% of GitHub AI-project contributions, India 19.0%, and the EU and UK 17.9%; the U.S. share has declined since 2016.Source: Projects · p. 70, Figure 1.5.2 · p. 70
• 12.2 million stars in 2023, up from 4.0 million in 2022, more than tripled for AI-related GitHub projects.The report says this increase, together with a previously reported rise in projects, underscores accelerating growth in open-source AI software development.Source: Figure 1.5.3 · p. 71, Projects · p. 71
• 10.5 million GitHub stars went to U.S.-located projects in 2023, while all major sampled regions saw year-over-year increases in stars.Source: Projects · p. 72, Figure 1.5.4 · p. 72

ACCESS THE PUBLIC DATA

• AI has surpassed human performance on some image-classification, visual-reasoning, and English-understanding benchmarks, but trails on competition-level mathematics, visual commonsense reasoning, and planning.Source: ACCESS THE PUBLIC DATA · p. 77

2. Here comes multimodal AI. Traditionally AI systems have been limited in scope, with language models

• GPT-4 and Gemini generate fluent prose and show strong language understanding, while multimodal systems process images and text and sometimes audio.Source: §2 · p. 77, §2.2 · p. 85

5. Human evaluation is in. With generative models producing high-quality text, images, and more, benchmarking

• Benchmarking increasingly incorporates human evaluations such as Chatbot Arena; PaLM-E and RT-2 make robots more flexible and able to ask questions for real-world interaction.Source: §5 · p. 77

7. More technical research in agentic AI. Creating AI agents, systems capable of autonomous operation

• Autonomous agents are improving, mastering complex games such as Minecraft and tackling real-world tasks such as online shopping and research assistance.Source: §7 · p. 77

8. Closed LLMs significantly outperform open ones. On 10 select AI benchmarks, closed models

• Closed models outperformed open ones, with a median performance advantage of 24.2%.Source: §8 · p. 77

Timeline: Significant Model Releases

• SynthID watermarks remain detectable after image alterations, and Mistral 7B surpasses Llama 2 13B; AI-created specialized data supports harder tasks.The data includes specialized material for image segmentation and 3D reconstruction, and can enhance current capabilities while enabling future algorithmic improvements.Source: §3 · p. 77, Timeline: Significant Model Releases · p. 79

State of AI Performance

• As of 2023, AI surpassed human baselines on some benchmarks but still lagged on complex visual commonsense and competition-level math tasks.Source: State of AI Performance · p. 81, Figure 2.1.162 · p. 81

3. Harder benchmarks emerge. AI models have reached performance saturation on established benchmarks

• Established benchmarks such as ImageNet stagnated near saturation, prompting a shift toward harder coding, reasoning, and agentic-behavior tests, including SWE-bench, MMMU, and AgentBench.Many benchmarks improve soon after introduction, then show little or no improvement in recent years.Source: §3 · p. 77, AI Index Benchmarks · pp. 82–84

HELM: Holistic Evaluation of Language Models

• 0.96 was GPT-4’s January 2024 mean win rate atop the aggregate HELM leaderboard, although different models led different task categories.HELM evaluates models across diverse scenarios, including reading comprehension, language understanding, and mathematical reasoning.Source: Figure 2.2.3 · p. 86, Figure 2.2.4 · p. 86, HELM: Holistic Evaluation of Language… · p. 86

MMLU: Massive Multitask Language Understanding

• 90.0% was Gemini Ultra’s January 2024 MMLU score, exceeding the 89.8% human baseline and improving 14.8 percentage points since 2022.The score was the highest reported on MMLU, a benchmark spanning 57 subjects.Source: Figure 2.2.6 · p. 87, MMLU: Massive Multitask Language… · p. 87

Chatbot Arena Leaderboard

• With over 200,000 votes, the Chatbot Arena leaderboard ranked OpenAI’s GPT-4 Turbo as users’ most preferred model in early 2024.Users compared anonymous model responses and voted for their preferred generations.Source: Chatbot Arena Leaderboard · p. 88, Figure 2.2.8 · p. 89

TruthfulQA

• Despite the observation that larger models tend to be less truthful, GPT-4 (RLHF) achieved TruthfulQA’s highest performance thus far in early 2024, scoring 0.6—nearly three times higher than a GPT-2-based model tested in 2021.Source: TruthfulQA · p. 90

HaluEval

• ChatGPT fabricated unverifiable information in approximately 19.5% of responses, and many LLMs struggled to detect hallucinations across evaluation tasks.The HaluEval benchmark contains over 35,000 hallucinated and normal samples; detection tasks included question answering, knowledge-grounded dialogue, and text summarization.Source: HaluEval · p. 92, Figure 2.2.12 · p. 93

HumanEval

• 96.3% was the leading HumanEval score, achieved by a GPT-4 variant (AgentCoder), 11.2 percentage points above the highest 2022 score.HumanEval comprises 164 challenging handwritten programming problems.Source: Figure 2.3.2 · p. 94, HumanEval · p. 94

SWE-bench

• Claude 2 solved only 4.8% of SWE-bench problems, while the top-performing 2023 model exceeded the best 2022 model by 4.3 percentage points.SWE-bench comprises 2,294 real GitHub issues and tests software engineering tasks requiring coordinated changes and complex reasoning.Source: Figure 2.3.4 · p. 95, §2.3 · p. 95, SWE-bench · p. 95

2.4 Image Computer Vision and Image Generation

• By 2024, Midjourney could create strikingly realistic Harry Potter images, while HEIM introduced a benchmark assessing text-to-image models across 12 deployment-relevant aspects.The report says most people struggle to distinguish AI-generated images from real human faces; HEIM aspects include image-text alignment and image quality.Source: §2.4 · p. 96, HEIM: Holistic Evaluation of… · p. 97

HEIM: Holistic Evaluation of Text-to-Image Models

• Human evaluators rate models, a crucial feature because automated metrics struggle to assess various aspects of images accurately.The passage specifically situates this point in evaluating aesthetics.Source: HEIM: Holistic Evaluation of… · p. 97
• DALL-E 2 ranks highest for human-evaluated image-text alignment, while Dreamlike Photoreal ranks highest for image quality, aesthetics, and originality.HEIM reports no single model excels across all criteria.Source: HEIM: Holistic Evaluation of… · p. 97, Figure 2.4.3 · p. 97, Figure 2.4.4 · p. 97
• HEIM evaluates image-generation models across 12 aspects, including alignment, quality, originality, reasoning, bias, fairness, robustness, multilinguality, and efficiency.The list also includes aesthetics, knowledge, and toxicity.Source: HEIM: Holistic Evaluation of… · p. 97
• MVDream achieves Inception Score (IS) and CLIP scores comparable to those in its training set, indicating high-quality generated images.The report notes implications for creative industries, where 3D content creation is traditionally time-consuming and labor-intensive.Source: HEIM: Holistic Evaluation of… · p. 98

Instruction-Following; VisIT-Bench

• GPT-4V scored 1,349 Elo on VisIT-Bench, marginally surpassing its human reference score across 592 challenging instructions in about 70 categories.VisIT-Bench assesses vision-language instruction-following, the capacity to interpret text-based directives about images.Source: Figure 2.4.9 · p. 99, VisIT-Bench · p. 99, Instruction-Following · p. 99

Editing; EditVal

• EditVal evaluates text-guided image editors across more than 13 edit types and 19 object classes, and its benchmark shows performance improvements since 2021 on varied editing tasks.The benchmark was applied to eight leading methods, including SINE and Null-text.Source: EditVal · p. 100, Figure 2.4.11 · p. 100

Segmentation

• Segment Anything uses AI alongside humans to create large datasets for training better systems, while Skoltech3D provides multiview 3D surface-reconstruction scenes captured across 100 viewpoints and 14 lighting conditions.The report describes Skoltech3D as a major improvement over existing 3D reconstruction datasets.Source: Segmentation · pp. 105, 107
• RealFusion generates complete 360-degree object models from single images by using existing 2D image generators to create multiple views, outperforming 2021 Shelf-Supervised across a wide range of objects.The method addresses the limited information often available in a single image for full reconstruction.Source: Segmentation · p. 108, Figure 2.4.21 · p. 108

2.5 Video Computer Vision and Video Generation

• Video analysis performs tasks across videos rather than single images, and video generation creates videos from text or images.The report notes that video generation has traditionally trailed image generation because of greater complexity and smaller training datasets.Source: §2.5 · pp. 109, 111

UCF101

• 36: W.A.L.T-XL recorded this FVD16 score on UCF101, more than halving the previous year’s state-of-the-art score.UCF101 is an action-recognition dataset with 101 categories that has also been used to benchmark video generators.Source: UCF101 · p. 109

2.5 Video Computer Vision and Video Generation

• Emu Video generates an image from text and then a video from both text and image, outperforming previously released state-of-the-art methods in human-evaluated quality and text faithfulness.The comparison measures the proportion of cases in which evaluators preferred Emu Video over the compared method.Source: Figure 2.5.6 · p. 111, §2.5 · p. 111

General Reasoning; Multimodal Understanding and Reasoning Benchmark for Expert AGI

• 59.4%: Gemini Ultra leads all MMMU subject categories with this overall score, while the benchmark is presented as a demanding test of AI perception, knowledge, and reasoning.The report uses MMMU validation-set results, as recommended by the benchmark authors.Source: Multimodal Understanding and Reasoning… · p. 112, Figure 2.6.212 · p. 114

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

• 65%: PhD-level experts’ GPQA accuracy compared with around 34% for nonexperts and 41.0% for GPT-4 on the main test set.GPQA contains 448 difficult multiple-choice questions crafted by subject-matter experts.Source: GPQA: A Graduate-Level Google-Proof… · p. 115, Figure 2.6.5 · p. 115

GSM8K

• 97%: GPT-4 Code Interpreter led GSM8K, improving 4.4% over the previous year’s state of the art and 30.4% from 2022.GSM8K comprises approximately 8,000 grade-school math word problems requiring multistep arithmetic solutions.Source: GSM8K · p. 117, Figure 2.6.9 · p. 118

MATH

• 84.3%: a GPT-4-based model solved MATH problems in 2023, up from 6.9% when the benchmark was first released.MATH contains 12,500 challenging competition-level mathematics problems.Source: MATH · p. 119

PlanBench

• About 34%: GPT-4 generated correct, cost-optimal PlanBench plans, versus about 6% for I-GPT-3.The one-shot test covered 600 Blocksworld problems.Source: PlanBench · p. 120, Figure 2.6.12 · p. 120

Visual Commonsense Reasoning (VCR)

• 7.93%: AI performance on the VCR challenge increased between 2022 and 2023, though AI has yet to outperform humans on the task.VCR evaluates both answer selection and the rationale for that answer using the Q->AR score.Source: Visual Commonsense Reasoning (VCR) · p. 121, Figure 2.6.14 · p. 121

Moral Reasoning

• AI’s growing use in healthcare and judicial systems makes robust moral reasoning important for navigating ethical principles and considerations.Source: Moral Reasoning · p. 122

MoCa

• GPT-4 showed the greatest agreement with human moral sentiments among surveyed models, while no model perfectly matched human moral systems.Larger models such as GPT-4 and Claude showed greater alignment than GPT-3; the study used a discrete agreement metric, and moral alignment has no universally agreed principles.Source: MoCa · p. 122, Figure 2.6.16 · p. 123

BigToM

• BigToM was developed to evaluate LLMs’ social and causal reasoning and was rated by human evaluators as superior to existing theory-of-mind benchmarks.It comprises 25 controls and 5,000 model-generated evaluations, testing forward belief, forward action, and backward belief; causal reasoning also concerns explaining outputs and updating conclusions.Source: BigToM · p. 124, Causal Reasoning · p. 124
• GPT-4’s BigToM performance neared but did not surpass human levels, matching humans on belief tasks and slightly exceeding them on forward action.GPT-4 also outperformed GPT-3.5, indicating that LLM performance on theory-of-mind benchmarks was trending upward.Source: BigToM · p. 124, Figure 2.6.18 · p. 125
• On causal-relationship discernment, GPT-4 scored 96% accuracy, 13 percentage points above the previous year’s best, and outperformed prior covariance-based AI models described as effective causal reasoners.Source: BigToM · p. 126

2.7 Audio

• MusicGen outperformed Riffusion, Moûsai, and MusicLM across generative music metrics, and human evaluators favored its overall quality.It had lower FAD and KL scores and a higher CLAP score; these indicate more plausible generation, better alignment with reference music, and greater adherence to text descriptions, respectively.Source: §2.7 · p. 129, Figure 2.7.2 · p. 130

AgentBench

• GPT-4 scored 4.01 on AgentBench, ahead of Claude 2’s 2.49; the study also found 2023 LLMs outperforming earlier versions in agentic settings.The researchers attributed struggles on some benchmark subsections to limits in long-term reasoning, decision-making, and instruction-following.Source: AgentBench · p. 131, Figure 2.8.2 · p. 132

2.8 Agents

• Voyager collected 3.3 times more unique items, traveled 2.3 times farther, and reached key milestones 15.3 times faster than previous models in Minecraft.The GPT-4-based agent remembers plans, adapts to new settings, and transfers knowledge in an open-ended environment where prior systems struggled to learn continuously.Source: §2.8 · p. 133

MLAgentBench

• On MLAgentBench, some agents exceeded 80% on ogbn-arxiv, while all scored 0% on BabyLM; GPT-4 consistently delivered the best results.The benchmark evaluates AI research agents across 15 varied research tasks, and results varied substantially by task.Source: MLAgentBench · p. 134, Figure 2.8.5 · p. 134

PaLM-E

• PaLM-E combines robotics and language modeling to address real-world manipulation and knowledge tasks, and its largest model has 562B parameters.It is trained on diverse visual-language and robotics data.Source: PaLM-E · p. 135
• PaLM-E outperformed prior methods on embodied VQA and planning, beat PaLI and CLIP-FT at failure detection, and demonstrated visual, language, mathematical, chain-of-thought, and multi-image reasoning.The report presents PaLM-E as evidence that language modeling and text data can improve robotics performance.Source: PaLM-E · p. 135

2.10 Reinforcement Learning

• Newly released foundation models reporting RLHF increased from none in 2021 to seven in 2022 and 16 in 2023.The Index tracked publicly reported use, so models using RLHF without reporting it may be missing.Source: §2.10 · p. 138, Figure 2.10.1 · p. 138

2.11 Properties of LLMs

• Emergent abilities appeared more pronounced with discontinuous metrics than continuous ones, and researchers observed them on only five of 39 BIG-bench benchmarks.The findings challenge the view that scaling inevitably produces new, unpredictable behaviors.Source: §2.11 · p. 141, Figure 2.11.1 · p. 142
• GPT-4’s June 2023 version performed worse than its March version by 42 percentage points on code generation, 16 on sensitive questions, and 33 on certain math tasks.The study also found that GPT-4’s instruction-following ability diminished over time, potentially explaining broader performance declines.Source: §2.11 · p. 143, Figure 2.11.2 · p. 144

Closed vs. Open Model Performance

• Closed models outperformed open models on all 10 selected benchmarks, with a median advantage of 24.2%, ranging from 4.0% on GSM8K to 317.7% on AgentBench.The report groups fully closed and limited-access models as closed; the benchmark data were collected in early January 2024.Source: §2.11 · p. 146, Figure 2.11.4 · p. 146, Figure 2.11.5 · p. 147

Prompting

• Prompt crafting can improve LLM performance without model changes; OPRO iteratively generates prompts from problem descriptions and prior solutions to improve algorithmic performance.Source: Prompting · pp. 148, 150
• ORPO achieves significantly greater accuracy than step-by-step prompting or an empty starting point on virtually all 23 BIG-bench Hard tasks.Figure 2.12.4 presents the benchmark comparison.Source: Prompting · p. 150

Fine-Tuning

• Fine-tuning further trains or adjusts models on smaller datasets, boosting overall performance, task-specific capabilities, and behavioral control.The report describes fine-tuning as an increasingly popular method of enhancing LLMs.Source: Fine-Tuning · p. 151

Attention

• Optimization methods target efficiency, including memory use, to make LLMs more accessible, while disclosure of inference’s environmental costs remains insufficient.The passages note that LLM training demands substantial computational resources and that inference costs may be a greater concern.Source: Attention · pp. 152, 154

Training

• Llama 2 70B released approximately 291.2 tonnes of carbon—nearly 291 times a round-trip traveler’s emissions—while GPT-3 training reportedly released 502 tonnes.Meta notes that it offsets all carbon emissions from Llama 2 training.Source: Figure 2.13.1 · p. 154, Training · p. 154
• Emissions generally rose with model size, especially in the Llama 2 series, but smaller models on carbon-intensive grids could still emit heavily; some estimates suggest declines over time.Major developers such as OpenAI, Google, Anthropic, and Mistral do not report training emissions, though Meta does.Source: Figure 2.13.2 · p. 155, Training · pp. 155–156

Inference

• Image generation has a much higher carbon footprint than text classification across 1,000 inferences.Research on inference emissions is scant, and per-query emissions can accumulate across thousands or millions of daily queries.Source: Figure 2.13.4 · p. 156, Inference · p. 156

Positive Use Cases

• AI supports environmental efforts through thermal energy system management, improved pest control strategies, and better urban air quality.The report showcases these applications as recent cases.Source: Positive Use Cases · p. 157

ACCESS THE PUBLIC DATA

• The responsible AI chapter examines privacy and data governance, transparency and explainability, security and safety, fairness, and AI’s potential impact on elections and political processes.The chapter analyzes metrics, research, and benchmarks in its four responsible AI areas.Source: ACCESS THE PUBLIC DATA · p. 160

1. Robust and standardized evaluations for LLM responsibility are seriously lacking.

• Responsible-AI reporting lacks standardization, and developers test models against different benchmarks, complicating systematic comparisons of leading models’ risks and limitations.Source: §1 · p. 161

2. Political deepfakes are easy to generate and difficult to detect. Political deepfakes are already

• AI deepfake detection methods show varying accuracy, CounterCloud demonstrates how easily fake content can be disseminated, and LLMs can be induced to behave harmfully through less obvious prompts such as repeating random words.Source: §2 · p. 161

5. LLMs can output copyrighted material. Multiple researchers have shown that the generative outputs

• Popular LLMs may output copyrighted material, and whether such outputs constitute copyright violations is a central legal question.Source: §5 · p. 161

6. AI developers score low on transparency, with consequences for research. The newly

• The Foundation Model Transparency Index finds that developers lack transparency, particularly about training data and methods, hindering research into AI systems’ robustness and safety.Source: §6 · p. 161

AI Incidents

• 123 AI incidents were reported in 2023, a 32.3% increase from 2022, and reported incidents have grown more than twentyfold since 2013.The report notes that growth may reflect greater AI use and awareness, as well as improved reporting; earlier incidents may have been underreported.Source: AI Incidents · p. 164

Risk Perception

• Privacy and data-governance risks lead organizations’ concerns globally, while fairness risks were selected by 20% of North American respondents, 31% in Asia, and 34% in Europe.Source: Risk Perception · p. 166, Figure 3.1.5 · p. 166

Risk Mitigation

• Organizations in most regions have operationalized at least one mitigation measure for relevant risks; some organizations in Europe (18%), North America (17%), and Asia (25%) have operationalized more than half of the measures asked about.Source: Risk Mitigation · p. 167, Figure 3.1.6 · p. 167

Overall Trustworthiness

• DecodingTrust evaluates LLMs across responsible-AI dimensions including bias, adversarial robustness, privacy, and machine ethics, with higher trustworthiness scores indicating more reliable models.Source: Overall Trustworthiness · p. 168
• GPT-4 improved over GPT-3.5 on standard benchmarks but remained more vulnerable to jailbreak prompts, partly because of stronger instruction following; Claude 2.0 was rated safest.The study also highlights biased outputs and leakage of private information from training datasets and conversation histories.Source: Overall Trustworthiness · p. 168, Figure 3.1.7 · p. 168

Tracking Notable Responsible AI Benchmarks

• Virtually all tracked responsible-AI benchmarks received more citations in 2023 than in 2022, with TruthfulQA citations rising especially sharply.The benchmarks include measures of truthfulness, toxicity, and bias in model outputs.Source: Tracking Notable Responsible AI… · p. 169, Figure 3.1.8 · p. 169

Reporting Consistency

• Inconsistent benchmark reporting complicates model comparisons and may reflect benchmark saturation, new tests without reporting standards, or selective reporting; the report calls for consensus on tests.These are presented as possible explanations for developers’ diverse benchmark choices.Source: Reporting Consistency · p. 171
• Among five developers, no responsible-AI benchmark set was universally accepted, TruthfulQA was used by at most three, and one developer reported none.All developers reported additional nonstandardized internal capability and safety tests.Source: Figure 3.1.10 · p. 171, Reporting Consistency · p. 171

3.2 Privacy and Data Governance

• LLM privacy challenges include obtaining informed consent for massive-scale data collection, balancing system utility with privacy, and preventing re-identification of anonymized data.Users may be unaware of how their data is used or how extensively it is collected.Source: §3.2 · p. 172

Academia

• 213 privacy and data-governance submissions appeared at the selected AI conferences in 2023, compared with 92 in 2022 and 39 in 2019.The report describes privacy and data governance as an increasing topic of research interest.Source: Academia · p. 173, Figure 3.2.1 · p. 173

Industry

• Less than 0.6% of surveyed companies fully operationalized all six data-governance mitigations; 90% operationalized at least one, 10% none, and the average was 2.2 of six.These are companies surveyed about risks relevant to their AI adoption.Source: Industry · p. 174

Extracting Data From LLMs

• This subsection introduces a November 2023 study investigating whether sensitive training data, including scraped personal information, can be extracted from LLMs.The passage frames the study around extractable memorization but supplies no findings from it.Source: Extracting Data From LLMs · p. 175
• Prompts that divert models from standard dialogue can extract training data from open and closed models, including PII, NSFW content, literature, and UUIDs.Although most deviations produce nonsensical outputs, some disclose data; researchers also found new attacks that overcome ChatGPT guardrails.Source: Extracting Data From LLMs · p. 175

Foundation Models and Verbatim Generation

• Without protective filters, models frequently reproduce training text; efficient n-gram checks can prevent perfect verbatim output, but models may still approximate memorized text.Memorization rates differ across datasets, and blocking extended verbatim outputs is proposed to reduce exposure of copyrighted material and personal information.Source: Foundation Models and Verbatim… · p. 177
• Prompts to Midjourney v6 produced images closely resembling film content, including with generic prompts, underscoring persistent copyright concerns around generative AI.The reported examples included images resembling movie characters and toys, and the problem persisted despite attempts to frame indirect prompts to avoid infringement.Source: Foundation Models and Verbatim… · p. 178

Auditing Privacy in AI Models

• A new privacy-auditing method estimates privacy loss in one training run, approximates multiple single-data-point runs, and reduces computational demands with minimal performance impact.It adds multiple independent data points simultaneously and assesses privacy by testing which points were used in training.Source: Auditing Privacy in AI Models · p. 179

3.3 Transparency and Explainability

• Advanced models are difficult to explain, even for developers, and their complexity can trade greater performance for lower interpretability than simpler alternatives.This creates a choice between high-performing opaque models and more transparent, potentially less precise models.Source: §3.3 · p. 180

Academia

• Transparency and explainability submissions to major academic conferences more than tripled from 2019, reaching a record-high 393 explainability-related submissions in 2023.The conferences included AAAI, FAccT, AIES, ICML, ICLR, and NeurIPS.Source: Figure 3.3.1 · p. 181, Academia · p. 181

Industry

• Organizations averaged 1.43 of four transparency measures, only 8% fully operationalized more than half, and less than 0.7% fully operationalized all measures.While 44% considered transparency and explainability relevant to their AI strategies, 12% had operationalized none and 88% reported at least one measure.Source: Industry · p. 182

The Foundation Model Transparency Index

• Meta’s Llama 2 and BigScience’s BLOOMZ were the most transparent models, but the mean score was 37%; open models averaged 51.3% versus 30.9% for closed models.The index assesses transparency across dimensions including development resources, algorithmic design, and downstream applications, using developers’ publicly accessible disclosures.Source: Figure 3.3.4 · p. 183, The Foundation Model Transparency Index · p. 183
• The Foundation Model Transparency Index uses over 100 indicators to assess model transparency and categorizes models by access level to show how they align with different access schemes.The report presents an aggregate transparency overview alongside the access-level breakdown.Source: The Foundation Model Transparency Index · p. 184, Figure 3.3.5 · p. 184

Neurosymbolic Artificial Intelligence (Why, What, and How)

• Neurosymbolic AI combines deep learning with symbolic reasoning for clearer insights and direct knowledge modification, aiming to mitigate hallucinations and incorrect reasoning despite symbolic methods’ performance limitations.The approach seeks to give models an explicit knowledge model of the world.Source: Neurosymbolic Artificial Intelligence… · p. 185

3.4 Security and Safety

• Security and safety debates encompassed present harms and potential extreme risks, while AI’s dual-use potential complicated security measures and systems could amplify sophisticated, adaptable, hard-to-detect cyberattacks.The chapter distinguishes protecting AI components from misuse-related harms and inherent system risks such as reliability concerns.Source: §3.4 · p. 186

Academia

• Security and safety submissions to select academic conferences increased 70.4% since 2019, despite a decline in submissions since 2022.Figure 3.4.1 presents the 2019–23 submission trend.Source: Academia · p. 187, Figure 3.4.1 · p. 187

Industry

• 45% of surveyed organizations considered reliability risks relevant to AI adoption, while the global average was 2.16 of six reliability measures fully implemented.The survey also found that 12% had no reliability measures fully operationalized.Source: Industry · p. 188, Figure 3.4.2 · p. 188

Do-Not-Answer: A New Open Dataset for Comprehensive Benchmarking of LLM Safety Risks

• Researchers created one of the first comprehensive open-source LLM safety datasets, evaluating six prominent models with a risk taxonomy spanning mild to severe risks.The evaluated models were GPT-4, ChatGPT, Claude, Llama 2, Vicuna, and ChatGLM2.Source: Do-Not-Answer: A New Open Dataset for… · p. 191
• Most tested models produced some harmful content; GPT-4 and ChatGPT were prone to discriminatory or offensive output, Claude to misinformation, and ChatGLM2 had the most violations.The harmful-response analysis is shown across risk categories and models in Figure 3.4.7, with total violations by model in Figure 3.4.8.Source: Do-Not-Answer: A New Open Dataset for… · p. 191, Figure 3.4.7 · p. 191, Figure 3.4.8 · p. 192

Universal and Transferable Attacks on Aligned Language Models

• A universal attack used automatically generated suffixes to make aligned LLMs produce unsafe content across models; the method is called Greedy Coordinate Gradient (GCG).The research contrasts this approach with prior attacks that often required substantial human intervention and were specific to particular models.Source: Universal and Transferable Attacks on… · p. 193
• Automated attacks can transfer across closed and open models, and unintelligible, non-human-readable prompts expose a gap in red-teaming focused on interpretable prompts.The reported models include ChatGPT, Bard, Claude, Llama-2-Chat, and Pythia.Source: Universal and Transferable Attacks on… · p. 193

MACHIAVELLI Benchmark

• MACHIAVELLI uses 134 choose-your-own-adventure games with over half a million social decision scenarios to assess ethical behavior and reward trade-offs.Models may compromise ethical choices to maximize game scores, but strategies can mitigate this trade-off; the benchmark targets traits including deception, morality, and power-seeking.Source: MACHIAVELLI Benchmark · p. 195

Academia

• 212 fairness and bias papers were submitted to selected academic conferences in 2023, up 25.4% from 2022 and almost fourfold since 2019.The report frames fairness as difficult to define and measure, and surveys its study and deployment in academia and industry.Source: Academia · p. 197, Figure 3.5.1 · p. 197, §3.5 · p. 197

Industry

• 29% of surveyed organizations considered fairness risks relevant to AI adoption; the share was 34% in Europe and 20% in North America.European organizations reported this risk most frequently, and North American organizations least frequently.Source: Industry · p. 198
• 1.97 out of five fairness measures were adopted on average, indicating limited comprehensive integration despite most companies fully implementing at least one measure.The report found no significant regional variation in implementation rates.Source: Industry · p. 198

(Un)Fairness in AI and Healthcare

• All four tested LLMs showed some race-based medical bias, with responses varying by question; Claude consistently gave problematic answers on one query, while most models did so on another.The study cautions against medical use because LLMs sometimes perpetuated debunked myths.Source: (Un)Fairness in AI and Healthcare · p. 199

Social Bias in Image Generation Models

• 90.8% accuracy: BiasPainter detects bias automatically, while evaluations found image-generation models biased across dimensions, particularly age and race compared with gender.BiasPainter is a testing framework for social bias in image-generation models; its automatic detection accuracy improved on previous methods.Source: Social Bias in Image Generation Models · p. 200, Figure 3.5.6 · p. 200

Measuring Subjective Opinions in LLMs

• LLM responses aligned closely with people in Western countries, with non-Western opinions notably underrepresented in shared responses.The study used GlobalOpinionQA to compare model answers with cross-country responses to survey statements.Source: Measuring Subjective Opinions in LLMs · p. 201

LLM Tokenization Introduces Unfairness

• The same sentence can require up to 15 times more tokens across languages; Portuguese needs about 50% more than English, while Shan needs 15 times more.The report links tokenization variability to differences in how efficiently languages are processed by GPT-4.Source: LLM Tokenization Introduces Unfairness · p. 203
• Up to four times higher inference costs and longer processing times can burden users of languages that require more tokens than English.Because models have fixed context windows, those languages also consume proportionally more available context, potentially reducing service quality.Source: LLM Tokenization Introduces Unfairness · p. 203

Generating Disinformation

• During Slovakia’s close 2023 election, likely AI-manipulated audio spread during a quiet period, while Meta policy did not cover audio manipulations.Progressive Slovakia ultimately lost to SMER by a slim margin.Source: Generating Disinformation · p. 206

Dissemination of Fake Content

• CounterCloud demonstrated an automated disinformation pipeline that scraped articles, generated counter-articles and fake engagement, and posted them on X for around $400.The counter-articles could include images and audio summaries and were attributed to a fake journalist.Source: Dissemination of Fake Content · p. 207

Detecting Deepfakes

• Five deepfake detection approaches, including newer methods, showed significant performance declines on datasets they had not seen during training.The study noted similarities between seen and unseen datasets as a possible basis for future broadly generalizable detectors.Source: Detecting Deepfakes · p. 208, Figure 3.6.4 · p. 208
• 10.7 percentage points was the maximum accuracy difference across racial subgroups; detectors performed worst on dark skin and best on Caucasian faces.The cited training data, FaceForensics++, was not equally balanced by race and gender.Source: Detecting Deepfakes · p. 209

LLMs and Political Bias

• ChatGPT showed systematic bias favoring Democrats in the United States and Labour in the U.K., correlating positively with Democrat versions and negatively with Republican versions.The report notes concern that such bias could influence users’ political views and stances.Source: LLMs and Political Bias · p. 210
• Research involving 529 individuals suggests that people generally have difficulty reliably detecting audio deepfakes.Source: LLMs and Political Bias · p. 211
• 73% of the time, listeners correctly detected audio deepfakes.The authors expect detection accuracy to decline as audio-generation methods improve; more convincing deepfakes could enable political manipulation, defamation, and a “liar’s dividend.”Source: LLMs and Political Bias · p. 211, Figure 3.6.7 · p. 211
• AI-based video surveillance of voters could undermine election integrity.Source: LLMs and Political Bias · p. 212
• Voter authentication is already highly feasible and carries significant risk; researchers assess political AI uses by technological readiness, risk, and deployment visibility.Source: LLMs and Political Bias · p. 212

ACCESS THE PUBLIC DATA

• The chapter uses data from Lightcast, LinkedIn, Quid, McKinsey, Stack Overflow, and IFR to examine AI labor, investment, adoption, economic impact, and robot installations.Its scope includes labor demand, hiring, skills, talent availability, generative AI investment, corporate adoption, and projected economic impact.Source: ACCESS THE PUBLIC DATA · p. 215

• Nearly octupling from 2022 to $25.2 billion, generative AI funding surged as OpenAI, Anthropic, Hugging Face, and Inflection reported substantial fundraising rounds.Source: §1 · p. 216

2. Already a leader, the United States pulls even further ahead in AI private investment.

• In 2023, U.S. AI investment reached $67.2 billion, nearly 8.7 times China’s; U.S. investment rose 22.1% while China’s fell 44.2% and EU/UK’s 14.1%.The United States’ increase was measured against 2022; private AI investment in China and the European Union, including the United Kingdom, declined over the same period.Source: §2 · p. 216

9. The data is in: AI makes workers more productive and leads to higher quality work.

• AI-related positions fell from 2.0% of U.S. job postings in 2022 to 1.6% in 2023, while studies found AI can speed tasks and improve output quality, but poor oversight can diminish performance.Studies also found potential for AI to bridge the skill gap between lower- and higher-skilled workers. The job-posting decline was attributed to fewer listings from leading AI firms and a lower proportion of tech roles at those firms.Source: §2 · p. 216, §9 · p. 217

4. AI decreases costs and increases revenues. A new McKinsey survey reveals that 42% of surveyed

• 42% of surveyed organizations reported AI-related cost reductions and 59% reported revenue increases; the share reporting lower costs rose 10 percentage points year over year.The survey covered implementation of AI, including generative AI.Source: §4 · p. 216

5. Total AI private investment declines again, while the number of newly funded AI

• Global private AI investment declined for a second year, while newly funded AI companies rose to 1,812, up 40.6% from the previous year.The decline was less sharp than the decrease from 2021 to 2022.Source: §5 · p. 216

6. AI organizational adoption ticks up. A 2023 McKinsey report reveals that 55% of organizations now

• Organizational AI use reached 55%, up from 50% in 2022 and 20% in 2017; China’s share of global industrial robot installations grew from 20.8% in 2013 to 52.4% in 2022.China had been the leading installer of industrial robots since surpassing Japan in 2013.Source: §6 · p. 216

8. Greater diversity in robotic installations. In 2017, collaborative robots represented a mere 2.8% of all

• Collaborative robots rose from 2.8% of new industrial robot installations in 2017 to 9.9% in 2022, while service robot installations increased across application categories except medical robotics.The report characterizes this as a growing emphasis on human-facing robot roles.Source: §8 · p. 217

10. Fortune 500 companies start talking a lot about AI, especially generative AI.

• AI appeared in 394 Fortune 500 earnings calls in 2023, nearly 80% of companies, up from 266 calls in 2022; generative AI was the most-mentioned theme, at 19.7% of calls.Since 2018, AI mentions in Fortune 500 earnings calls had nearly doubled.Source: §10 · p. 217

4.1 What’s New in 2023: A Timeline

• The 2023 timeline highlights AI dealmaking, including BioNTech’s $680 million InstaDeep acquisition, Databricks’ $1.3 billion MosaicML acquisition, and Thomson Reuters’ $650 million Casetext acquisition.The timeline also records funding rounds, product launches, and other AI-related developments during the year.Source: Jan. 10 · p. 218, June 26 · p. 220, June 29 · p. 220

Global AI Labor Demand

• In 2023, AI skills appeared in 1.6% of U.S. job postings, down from 2.0% in 2022; the United States, Spain, and Sweden led globally at 1.6%, 1.4%, and 1.3%.Most countries saw the share of postings requiring AI skills decrease from 2022 to 2023, although many had more AI-related postings over the prior five years.Source: Figure 4.2.1 · p. 223, Global AI Labor Demand · p. 223

U.S. AI Labor Demand by Skill Cluster and Specialized Skill

• Machine learning led U.S. AI skill demand at 0.7%, ahead of artificial intelligence at 0.5% and natural language processing at 0.2%.Generative AI was the only tracked AI-related skill cluster to gain market share year over year, growing by more than a factor of 10; Python also rose notably among specialized skills over the decade.Source: U.S. AI Labor Demand by Skill Cluster… · pp. 224–225, Figure 4.2.2 · p. 224
• 15,410 U.S. job postings cited generative AI, 4,669 cited large language modeling, and 2,841 cited ChatGPT; their shares of generative AI postings were 60.0%, 18.2%, and 11.1%, respectively.These figures describe 2023 postings.Source: U.S. AI Labor Demand by Skill Cluster… · pp. 226–227, Figure 4.2.4 · p. 226, Figure 4.2.5 · p. 227

U.S. AI Labor Demand by Sector

• In 2023, information led U.S. sectors in AI-posting share at 4.6%, followed by professional, scientific, and technical services at 3.3% and finance and insurance at 2.9%.Nearly every sector had a lower proportion of AI job postings than in 2022, except public administration and educational services.Source: U.S. AI Labor Demand by Sector · p. 228, Figure 4.2.6 · p. 228

U.S. AI Labor Demand by State

• California had 70,630 AI job postings and Texas 36,413; California accounted for 15.3% of U.S. AI postings, followed by Texas at 7.9% and Virginia at 5.3%.By AI share of all state job postings, the District of Columbia led at 2.7%, followed by Delaware at 2.4%.Source: U.S. AI Labor Demand by State · pp. 229–230, Figure 4.2.9 · p. 230
• From 2022 to 2023, the share of total AI-related postings declined in Washington, California, New York, and Texas; since 2019, California’s share fell steadily while Texas’s rose slightly.Source: U.S. AI Labor Demand by State · pp. 230–231

AI Hiring

• In 2023, relative AI hiring rates were highest in Hong Kong at 28.8%, followed by Singapore and Luxembourg at 18.9% each.AI hiring picked up from early 2023 in Australia, Canada, Singapore, and India; LinkedIn coverage is below the stated threshold in India and South Korea, so findings for them warrant caution.Source: Figure 4.2.124 · p. 232, AI Hiring · p. 232

AI Skill Penetration

• From 2015 to 2023, India had the highest AI skill penetration rate at 2.8, followed by the United States at 2.2 and Germany at 1.9.Source: AI Skill Penetration · p. 234
• U.S. AI skill penetration was 2.2 times the global average; across sampled countries, men had higher relative AI skill penetration than women.Among women, the highest reported relative rates were India (1.7), the United States (1.2), and Israel (0.9).Source: AI Skill Penetration · pp. 234–235

AI Talent

• In 2023, AI talent concentration was highest in Israel (1.1%), Singapore (0.9%), and South Korea (0.8%); since 2016, the largest increases were in India (263%), Cyprus (229%), and Denmark (213%).Luxembourg, Switzerland, and the United Arab Emirates reported the greatest incoming AI talent migration; Israel, India, and South Korea recently saw declining net migration. Concentration estimates may be affected by LinkedIn coverage and should be used cautiously.Source: AI Talent · pp. 236, 238

4.2 Jobs Chapter 4: Economy

• The average global salary for a cloud infrastructure engineer was $105,000, compared with $185,000 in the United States; U.S. salaries were higher across all surveyed positions.In both the global and U.S. results, senior executives were the highest-compensated roles, followed by engineering managers; respondents were professional developers, not exclusively AI workers.Source: §4.2 · p. 240

Corporate Investment

• Global corporate AI investment declined to $189.2 billion in 2023, down approximately 20% from 2022, while mergers and acquisitions fell 31.2%.Despite the recent decline, AI-related investment increased thirteenfold over the past decade.Source: Figure 4.3.1 · p. 242, Corporate Investment · p. 242

Global Trends

• Global private AI investment fell 7.2% for a second year, while generative AI drew $25.2 billion—nearly nine times its 2022 investment—and 99 new startups received funding.Generative AI accounted for over a quarter of private AI investment in 2023.Source: Figure 4.3.2 · p. 243, Global Trends · pp. 243, 244, 246

Regional Comparison by Funding Amount

• The United States invested $67.2 billion in AI in 2023, roughly 8.7 times China’s $7.8 billion and 17.8 times the United Kingdom’s $3.8 billion.Across 2013–23, the United States also led with $335.2 billion, ahead of China ($103.7 billion) and the United Kingdom ($22.3 billion).Source: Figure 4.3.8 · p. 247, Regional Comparison by Funding Amount · pp. 247–248
• From 2022 to 2023, U.S. AI private investment rose 22.1%, while investment fell 44.2% in China and 14.1% in the European Union plus the United Kingdom.The U.S. generative AI investment gap over the EU plus UK widened from about $1.9 billion in 2022 to $21.1 billion in 2023.Source: Figure 4.3.10 · p. 249, Regional Comparison by Funding Amount · pp. 249–250

Regional Comparison by Newly Funded AI Companies

• The United States led regions in newly funded AI companies in 2023 with 897, followed by China with 122 and the United Kingdom with 104.Source: Regional Comparison by Newly Funded AI… · p. 251
• Over 2013–23, the United States had around 3.8 times as many newly funded AI companies as China and 7.6 times as many as the United Kingdom.Since 2022, new AI company counts increased significantly in the United States, European Union, and United Kingdom, while China saw a slight year-over-year decrease.Source: Regional Comparison by Newly Funded AI… · pp. 252–253, Figure 4.3.13 · p. 252

Focus Area Analysis

• In 2023, private AI investment was highest in AI infrastructure/research/governance ($18.3 billion), NLP and customer support ($8.1 billion), and data management and processing ($5.5 billion).Investment in infrastructure/research/governance and data management/processing grew from 2022, while medical and healthcare and NLP/customer support investment had declined since peaking in 2021.Source: Focus Area Analysis · p. 254, Figure 4.3.15 · p. 254
• The United States outpaced China and the European Union and United Kingdom in nearly all AI investment focus areas, but China led facial recognition and nearly matched U.S. semiconductor investment.Facial-recognition investment was $130 million in China versus $90 million in the United States; semiconductor investment was $630 million in China versus $790 million in the United States.Source: Focus Area Analysis · p. 256, Figure 4.3.17 · p. 257

Industry Adoption

• The industry-adoption analysis draws on McKinsey’s 2023 survey of 1,684 respondents, which newly included detailed questions about generative AI adoption and AI-related hiring.Respondents represented varied regions, industries, company sizes, functional areas, and tenures.Source: Industry Adoption · p. 258

Adoption of AI Capabilities

• 55% of surveyed organizations had adopted AI in at least one function in 2023, up from 50% in 2022 and 20% in 2017.AI adoption increased in every surveyed region; Europe rose by 9 percentage points, Greater China by 7, and North America remained the leader.Source: Adoption of AI Capabilities · pp. 258, 265, Figure 4.4.1 · p. 258
• Contact-center automation was the most common surveyed AI use case by function (26%); across industries, the most embedded technologies were NL text understanding, robotic process automation, and virtual agents (30% each).Tech, media, and telecom had the highest reported adoption in product/service development (44%), while data engineers were the most frequently hired AI-related role (36%).Source: Adoption of AI Capabilities · pp. 259, 260, 261, 263, Figure 4.4.2 · p. 259
• Respondents most often reported AI-related cost savings in manufacturing (55%) and service operations (54%), while manufacturing led revenue gains (66%), followed by marketing and sales (65%) and risk (64%).Source: Adoption of AI Capabilities · p. 264

Adoption of Generative AI Capabilities

• Generating initial drafts of text documents was the most frequent reported generative AI application (9%), followed by personalized marketing, text summarization, and image/video creation (8% each).Most reported leading generative AI use cases were in marketing and sales.Source: Adoption of Generative AI Capabilities · p. 266
• Generative AI adoption was 33% across regions; its leading organizational applications were marketing and sales (14%), product and/or service development (13%), and service operations (10%).The report says general AI remained more prevalent overall than generative AI.Source: Adoption of Generative AI Capabilities · pp. 267–268, Figure 4.4.10 · p. 267

Preference

• Among professional developers, 56.0% reported using GitHub Copilot, 83.3% favored ChatGPT as an AI search tool, and 53.1% regularly used AWS.Tabnine and AWS’s cloud-platform competitors are not included here; AWS is described as the most commonly used cloud platform.Source: Preference · pp. 269–270, Figure 4.4.12 · p. 269, Figure 4.4.13 · p. 269

Workflow

• Developers reported regular AI use for code writing (82.6%), debugging assistance (48.9%), and documentation (34.4%), while 55.2% expressed interest in AI for code testing.Only 23.9% currently used AI for code testing.Source: Workflow · p. 270, Figure 4.4.15 · p. 270
• Among professional developers, 76.1% viewed AI development tools favorably or very favorably, while 42.2% reported moderate or high trust in their output accuracy.Only 3.2% expressed unfavorable opinions; 27.2% reported some level of distrust or high distrust.Source: Workflow · p. 271

AI’s Labor Impact

• Copilot users completed tasks 26% to 73% less time; consultants with GPT-4 improved productivity 12.2%, speed 25.1%, and quality 40.0%.The Copilot meta-review compared users of Microsoft Copilot or GitHub Copilot with workers without AI access; consultant results were compared with a control group.Source: AI’s Labor Impact · pp. 272–273, Figure 4.4.19 · p. 272, Figure 4.4.20 · p. 273
• Lower-skilled consultants improved 43.0%, versus 16.5% for higher-skilled consultants, narrowing but not eliminating the performance gap.Higher-skilled workers using AI still performed better than lower-skilled AI users, but the disparity was markedly lower than without AI.Source: AI’s Labor Impact · p. 275
• AI assistance improved résumé-review accuracy, but overreliance can impair worker performance.The supplied passage does not quantify the accuracy improvement.Source: AI’s Labor Impact · p. 276
• Recruiters using “good AI” performed 1.08 points worse than those using error-prone “bad AI”; the study theorizes complacency reduced scrutiny.The passages also report a 0.6-point difference compared to receiving no AI assistance, but do not identify the specific advice condition in the supplied text.Source: AI’s Labor Impact · p. 276, Figure 4.4.24 · p. 276

Earnings Calls

• Quid used natural language processing to analyze 2023 Fortune 500 earnings calls for mentions of artificial intelligence, AI, machine learning, ML, and deep learning.Source: Earnings Calls · p. 277

Aggregate Trends

• 394 Fortune 500 earnings calls mentioned AI in 2023, up from 266 in 2022, and mentions had nearly doubled since 2018.The 2023 total represented nearly 80% of Fortune 500 companies.Source: Aggregate Trends · p. 277, Figure 4.4.25 · p. 277

Specific Themes

• Generative AI appeared in 19.7% of 2023 earnings calls, up from 0.31% in 2022; AI investments and growth initiatives appeared in 15.2%, and company/brand AIs in 7.6%.Source: Specific Themes · p. 278, Figure 4.4.26 · p. 278

Aggregate Trends

• 553,000 industrial robots were installed worldwide in 2022, up 5.1% from 2021; operational stock reached 3,904,000, up from 3,479,000.Installations had increased more than threefold since 2012.Source: Figure 4.5.1 · p. 283, Aggregate Trends · pp. 283–284, Figure 4.5.2 · p. 284

Industrial Robots: Traditional vs. Collaborative Robots

• Collaborative robots grew from 2.8% of new industrial robot installations in 2017 to 9.9% in 2022.They are designed to work alongside people, and the robotics community values their safety, flexibility, scalability, and iterative learning ability.Source: Figure 4.5.3 · p. 285, Industrial Robots: Traditional vs.… · p. 285

By Geographic Area

• China’s share of global industrial robot installations rose from 20.8% in 2013 to 52.4% in 2022, after it overtook Japan in 2013.In 2022, China installed 290,300 robots, compared with 50,400 in Japan and 39,500 in the United States.Source: By Geographic Area · pp. 286–287, Figure 4.5.4 · p. 286, Figure 4.5.5 · p. 287

Country-Level Data on Service Robotics

• Service robot installations increased in every application category from 2021 to 2022 except medical robotics; hospitality rose 2.3 times and transportation and logistics 1.4 times.Source: Figure 4.5.8 · p. 290, Country-Level Data on Service Robotics · p. 290
• The United States had approximately 2.06 times more professional service robot manufacturers than China in 2022; most surveyed countries’ manufacturers were established incumbents.Source: Figure 4.5.9 · p. 291, Country-Level Data on Service Robotics · p. 291

Sectors and Application Types

• In 2022, 266,000 industrial robots were installed for handling—3.1 times the 87,000 installed for welding and 4.4 times the 61,000 for assembly.Every application category except processing had more installations in 2022 than in 2020.Source: Figure 4.5.11 · p. 293, Sectors and Application Types · p. 293

China vs. United States

• China’s 2022 leading industrial robot installation sectors were electrical/electronics (100,000), automotive (73,000), and metal and machinery (31,000); U.S. automotive led with 14,500.In the United States, every other sector had fewer installations in 2022 than in 2021, except electronics.Source: Figure 4.5.12 · p. 294, China vs. United States · pp. 294–295, Figure 4.5.13 · p. 295

1. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance

• 2023 brought more significant science-related AI applications, including AlphaDev and GNoME.AlphaDev makes algorithmic sorting more efficient, while GNoME facilitates materials discovery.Source: §1 · p. 299
• In 2022, AI began to advance scientific discovery.Source: §1 · p. 299

2. AI helps medicine take significant strides forward. In 2023, several significant medical systems

• AI systems launched in 2023 included EVEscape for pandemic prediction and AlphaMissence for mutation classification; FDA-approved AI medical devices increased 12.1% from 2021 and more than 45-fold since 2012.Source: §2 · p. 299, §4 · p. 299

3. Highly knowledgeable medical AI has arrived. Over the past few years, AI systems have shown

• 90.2% accuracy on MedQA by GPT-4 Medprompt was 22.6 percentage points above 2022’s highest score, while performance on the benchmark nearly tripled since its 2019 introduction.The passage identifies MedQA as a test of AI clinical knowledge.Source: §3 · p. 299

AlphaDev

• AlphaDev found shorter-instruction algorithms for fundamental sorting tasks, and some were added to LLVM’s C++ sort library—the first update in over 10 years and its first reinforcement-learning-designed addition.Source: AlphaDev · p. 300, Figure 5.1.1 · p. 300

FlexiCubes

• FlexiCubes uses gradient-based optimization and adaptable parameters for localized mesh adjustments, producing extractions closer to ground truth than other leading differentiable-isosurfacing methods.Source: FlexiCubes · p. 301

Synbot

• Synbot’s closed-loop robotic system produced synthesis recipes with yields equal to or above established references; in one experiment, it reached the mid-80% reference range and completed synthesis faster.Source: Synbot · p. 303

GraphCast

• GraphCast makes accurate 10-day forecasts in under a minute; an AI flood model predicts some extreme events up to five days ahead, matches or surpasses state-of-the-art models, and is used in over 80 countries.Source: GraphCast · p. 304, Flood Forecasting · p. 306, Figure 5.1.11 · p. 306

GNoME

• GNoME identified 2.2 million new crystal structures and outperformed the Materials Project by finding a significantly larger number of stable crystals.Source: GNoME · p. 305

SynthSR

• SynthSR converts clinical brain scans into high-resolution T1-weighted images that correlate robustly with observed brain volumes, while a new neurodegenerative-disease diagnostic method combines AI-coupled infrared sensors with an immunoassay.Source: SynthSR · p. 307, Coupled Plasmonic Infrared Sensors · p. 309

EVEscape

• EVEscape predicted 50.0% of observed SARS-CoV-2 mutations, versus 46.2% and 32.3% for traditional lab studies and 24% for a previous model.It predicts viral escape independently of current-strain data, using historical sequences and biophysical and structural information; ImmunoSEIRA AI predictions closely matched actual fibril percentages.Source: Coupled Plasmonic Infrared Sensors · p. 309, EVEscape · p. 310, Figure 5.2.6 · p. 311

AlphaMissence

• AlphaMissense classified 89% of 71 million missense variants: 57% as likely benign and 32% as likely pathogenic, while human annotators had confirmed the nature of 0.1%.The remaining variants were categorized as uncertain.Source: AlphaMissence · p. 312, Figure 5.2.8 · p. 312

Human Pangenome Reference

• The AI-developed human pangenome map annotated median shares of 99.07% of protein-coding genes, 99.42% of protein-coding transcripts, 98.16% of noncoding genes, and 98.96% of noncoding transcripts.The report describes it as the most comprehensive and genetically diverse human-genome mapping to date.Source: Human Pangenome Reference · p. 313, Figure 5.2.10 · p. 313

MedQA

• GPT-4 Medprompt achieved 90.2% MedQA accuracy, 22.6 percentage points above the 2022 top score; AI performance on the benchmark has nearly tripled since its inception.MedQA is a dataset of more than 60,000 clinical questions derived from professional medical board exams.Source: MedQA · p. 314, Figure 5.2.11 · p. 314

CoDoC

• PANDA exceeded mean radiologists by 34.1% in sensitivity and 6.3% in specificity in validation, then reached 92.9% sensitivity and 99.9% specificity in a real-world test of approximately 20,000 patients.The model detects and classifies pancreatic lesions in X-rays; screening for pancreatic cancer is challenging because of low prevalence and false-positive risk.Source: CoDoC · p. 319

FDA-Approved AI-Related Medical Devices

• The FDA approved 139 AI-related medical devices in 2022, up 12.1% from 2021 and more than 45-fold since 2012; 87.1% were radiology devices and 7.2% cardiovascular.As of October 2023, no FDA-approved devices utilized generative AI or were powered by LLMs.Source: FDA-Approved AI-Related Medical Devices · pp. 321–322, Figure 5.2.20 · p. 321, Figure 5.2.21 · p. 322

ACCESS THE PUBLIC DATA

• American and Canadian CS bachelor’s graduates continued to rise, while new CS master’s graduates stayed relatively flat and PhD graduates grew modestly; since 2018, both graduate groups declined slightly.Source: ACCESS THE PUBLIC DATA · p. 328

2. The migration of AI PhDs to industry continues at an accelerating pace. In 2011, roughly

• By 2022, 70.7% of new AI PhDs entered industry and 20.0% academia, compared with 40.9% and 41.6% respectively in 2011; industry’s share rose 5.3 percentage points in the past year.Source: §2 · p. 328

6. AI-related degree programs are on the rise internationally. The number of English-language,

• AI-related postsecondary degree programs have tripled since 2017, while 201,000 AP CS exams were administered in 2022 after participation grew more than tenfold since 2007.Students at larger high schools and in suburban areas were more likely to have access to CS courses.Source: §6 · p. 328, §5 · p. 328

7. The United Kingdom and Germany lead in European informatics, CS, CE, and IT graduate

• The United Kingdom and Germany led Europe in graduate production across informatics, CS, CE, and IT; per capita, Finland led bachelor’s and PhD production, and Ireland led master’s production.Source: §7 · p. 328

CS Bachelor’s Graduates

• CS bachelor’s graduates in North America rose more than threefold over the decade, including a 7.9% year-over-year increase from 2021 to 2022.The international share fell from 16.3% in 2021 to 15.2% in 2022, despite a longer-term rise over the decade.Source: CS Bachelor’s Graduates · pp. 329–330, Figure 6.1.1 · p. 329

CS Master’s Graduates

• North American CS master’s graduates more than doubled over the decade, but their total fell 2.5% last year as international graduates declined 14.8 percentage points.The international and domestic shares of CS master’s graduates were roughly even in 2022.Source: CS Master’s Graduates · pp. 331–332, Figure 6.1.3 · p. 331, Figure 6.1.4 · p. 332

CS PhD Graduates

• North American CS PhD graduates reached 2,105 in 2022, the highest total since 2010, while their international share slipped from 68.6% to 65.9%.The international proportion had risen over the preceding decade.Source: CS PhD Graduates · pp. 333–334, Figure 6.1.5 · p. 333
• In 2022, 70.7% of new AI PhDs entered industry, compared with 20.0% entering academia; government roles remained around 0.7%.In 2011, the corresponding shares were 40.9% for industry and 41.6% for academia.Source: CS PhD Graduates · p. 335

CS, CE, and Information Faculty

• In 2022, North American departments hired a decade-high 915 new CS, CE, and information faculty, including 455 tenure-track hires.The U.S. had 7,084 CS faculty, 65.7% on the tenure track; its total rose 4.4% from 2021 and 45.0% from 2011.Source: CS, CE, and Information Faculty · pp. 337–338, Figure 6.1.10 · p. 337, Figure 6.1.11 · p. 338
• In 2022, 43% of new faculty appointments came from other academic positions; faculty transitions from industry fell to 7%, down from 11% in 2021 and 13% in 2019.The share of failed searches involving declined offers rose from 34% in 2011 to 55% in 2022.Source: CS, CE, and Information Faculty · pp. 339–340

Informatics, CS, CE, and IT Bachelor’s Graduates

• In 2022, the United Kingdom produced approximately 25,000 new informatics, CS, CE, and IT bachelor’s graduates, while Finland, Norway, and the Netherlands led per capita.The per-capita figures were 53.4, 42.6, and 38.6 graduates per 100,000 inhabitants, respectively.Source: Informatics, CS, CE, and IT Bachelor’s… · pp. 344–345, Figure 6.1.17 · p. 344, Figure 6.1.19 · p. 346

Informatics, CS, CE, and IT Master’s Graduates

• The United Kingdom led Europe in new informatics, CS, CE, and IT master’s graduates with approximately 20,000, while Germany, Turkey, and Spain recorded the largest decade-long growth.Growth was 259% in Germany, 197% in Turkey, and 194% in Spain.Source: Informatics, CS, CE, and IT Master’s… · p. 347, Figure 6.1.21 · p. 347
• Ireland led in master’s graduates per capita at 31.2, followed by the United Kingdom at 29.8 and Estonia at 27.4; Germany had the greatest decade-long per-capita growth at 243%.Source: Informatics, CS, CE, and IT Master’s… · p. 349, Figure 6.1.23 · p. 349, Figure 6.1.24 · p. 350

Informatics, CS, CE, and IT PhD Graduates

• The United Kingdom and Germany produced the most new informatics, CS, CE, and IT PhD graduates in 2022, while Finland led per capita.The counts were 1,060 in the United Kingdom and 910 in Germany; Finland had 2.1 graduates per 100,000 inhabitants, ahead of Estonia (1.9) and the United Kingdom (1.6).Source: Informatics, CS, CE, and IT PhD… · pp. 351, 353, Figure 6.1.25 · p. 351, Figure 6.1.27 · p. 353

AI-Related Study Programs

• The number of English-language AI university study programs tripled from 2017 to 2023, with programs concentrated at the master’s level and the United Kingdom leading by country.In 2023, 55.0% of programs were at master’s level, 39.8% at bachelor’s level, and 5.3% at PhD level; the United Kingdom had 744 programs, versus 667 in the United States and 89 in Canada. The data cover English-language programs only.Source: Total Courses · p. 355, Education Level · p. 356, Geographic Distribution · p. 357

Geographic Distribution

• Nearly every sampled country had more English-language AI university study programs in 2023 than in 2022, while Malta, the United Kingdom, and Cyprus led per capita.Source: Geographic Distribution · pp. 357–358, Figure 6.1.31 · p. 357, Figure 6.1.32 · p. 358

State-Level Trends

• In 2023, 30 U.S. states required all high schools to offer a foundational CS course, but the share of schools offering CS ranged from 28% in Minnesota to 99% in Maryland and Arkansas.Nevada was also among the top three at 96%; Montana (34%) and Louisiana (35%) were among the bottom three.Source: State-Level Trends · p. 359

Index Report 2024 6.2 K–12 CS and AI Education

• Approximately two-thirds of states now require high-school computer science, fund it, and have state education plans, compared with only a few states in 2017.Source: Index Report 2024 6.2 K–12 CS and AI… · p. 360

AP Computer Science

• 201,000 AP computer science exams were administered in 2022, an 11.1% increase from 2021 and more than tenfold the 2007 total.The chart covers both Computer Science A and Computer Science Principles exams.Source: Figure 6.2.4 · p. 361, AP Computer Science · p. 361
• Large schools are 15 percentage points more likely than medium-sized schools to offer computer science, and suburban students have better course access than urban or rural peers.The report says the gap widens further for small schools.Source: AP Computer Science · p. 363
• K–12 teacher ChatGPT use rose from 51% in March 2023 to 63% in July; users most often reported lesson planning and creative class ideas (30% each).Another 27% used it to enhance their background knowledge.Source: AP Computer Science · p. 364

1. The number of AI regulations in the United States sharply increases. The number of AI-related

• 25 AI-related regulations were in place in the United States in 2023, up from one in 2016, with a 56.3% increase in 2023 alone.Source: §1 · p. 369

2. The United States and the European Union advance landmark AI policy action. In 2023,

• The European Union reached a deal on the AI Act, enacted in 2024, while President Biden signed an AI Executive Order, the United States’ most notable AI policy initiative of 2023.Source: §2 · p. 369

3. AI captures U.S. policymaker attention. The year 2023 witnessed a remarkable increase in AI-related

• AI-related federal bills proposed rose to 181 in 2023 from 88 in 2022, while legislative proceedings mentioning AI nearly doubled to 2,175 from 1,247.AI was mentioned in proceedings in 49 countries, with at least one country from every continent discussing it.Source: §3 · p. 369, §4 · p. 369

5. More regulatory agencies turn their attention toward AI. The number of U.S. regulatory agencies

• The number of U.S. regulatory agencies issuing AI regulations rose from 17 in 2022 to 21 in 2023, including three agencies regulating AI for the first time.The new agencies included the Departments of Transportation and Energy and the Occupational Safety and Health Administration.Source: §5 · p. 369

Jul.21,

• Seven major AI firms pledged prelaunch security assessments, risk sharing, public issue reporting, and disclosure when content is AI-generated.The commitments were voluntary and aimed to promote safe, secure, and reliable AI.Source: Jul.21 · p. 372

Jul. 27,

• The CREATE AI Act proposed a national research resource offering compute, curated datasets, educational tools, and testbeds; a Senate bill proposed banning deceptive AI content about federal candidates.The research resource was intended to improve access for AI researchers and students and support system testing and evaluation.Source: Jul. 27 · p. 373, Sep. 12 · p. 373

Aug. 15,

• China shifted its generative-AI rules toward targeted oversight and encouraged development, while the U.K. proposed principles for competitive AI markets and consumer protection.The U.K. principles included accountability, access to essential inputs, business choice, and fair practices.Source: Aug. 15 · p. 373, Sep. 18 · p. 374

Oct. 30,

• The U.S. executive order set benchmarks for AI safety, security, privacy, equity, civil rights, competition, and innovation, and tasked Education and NIST with follow-up work.It also mandated a national security memorandum for AI use in military and intelligence operations.Source: Oct. 30 · p. 374

Nov. 01,

• The U.K. AI Safety Summit produced the Bletchley Declaration, endorsed by 28 countries, and the country established a Safety Institute for research and tools to govern advanced-AI risks.The Institute also aims to improve global understanding of advanced-AI dangers and position the U.K. as a center for safety research.Source: Nov. 01 · p. 375, Nov. 02 · p. 375

Dec. 09,

• The EU AI Act establishes risk-based rules, bans systems with unacceptable risks, applies transparency standards to generative AI, and imposes basic transparency duties on low-risk AI.High-risk systems are classified by product-based and critical sectors.Source: Dec. 09 · p. 375

Overview

• 32 countries enacted at least one AI-related bill, with 148 bills passed overall; 28 passed in 2023, down from 39 in 2022 but above 2016.The analysis classified bills by relevance: high-relevance bills focus fundamentally on AI policy, while medium-relevance bills contain significant AI policy elements within a broader focus.Source: Figure 7.2.1 · p. 376, Overview · p. 376, Figure 7.2.2 · p. 377

By Geographic Area

• Belgium passed five AI-related laws in 2023, while France, South Korea, and the U.K. passed three each; since 2016, the U.S. led with 23.Portugal passed 15 laws since 2016, followed by Belgium with 12.Source: Figure 7.2.3 · p. 378, Figure 7.2.4 · p. 378, By Geographic Area · p. 378

By Relevance

• Of 28 AI-related bills enacted globally in 2023, two were high-relevance and 18 were medium-relevance bills.The report classifies bills as expansive or restrictive, and finds a global shift toward restrictive legislation while commitment to enhancing AI capabilities remains.Source: By Relevance · p. 379, By Approach · p. 380, Figure 7.2.6 · p. 380

By Subject Matter

• In 2023, AI-related laws covered a broader range of primary subjects, with two bills passed in each of seven policy categories.The categories were armed forces and national security; civil rights and liberties, minority issues; commerce; education; labor and employment; and science, technology, and communication; economics and public finance had historically predominated.Source: By Subject Matter · p. 381

Federal Level

• Proposed U.S. federal AI-related bills more than doubled, rising from 88 in 2022 to 181 in 2023.The report suggests the increase likely reflects policymakers’ response to greater public awareness and AI capabilities, such as ChatGPT.Source: Federal Level · p. 382, Figure 7.2.8 · p. 382

State Level

• In 2023, U.S. states proposed 150 AI-related bills, up from 61 in 2022, and enacted a greater proportion of bills than the federal level.California enacted seven AI-related laws in 2023; across 2016–23, California led with 13 bills, followed by Maryland with 10 and Washington with 7.Source: State Level · pp. 383–384, Figure 7.2.9 · p. 383, Figure 7.2.11 · p. 384

Overview

• AI mentions in legislative proceedings nearly doubled, from 1,247 in 2022 to 2,175 in 2023; since 2016, mentions have risen almost tenfold.In 2023, the United Kingdom led with 405 mentions, followed by the United States with 240 and Australia with 227.Source: Overview · pp. 385–386, Figure 7.2.12 · p. 385, Figure 7.2.13 · p. 386

AI Mentions

• AI mentions appeared in proceedings in 48 of the 80 countries analyzed in 2023, reaching at least one country on every continent.The analysis covered proceedings containing the phrase “artificial intelligence” from 2016 to 2023; public records were not available for every country.Source: Overview · pp. 385–386, AI Mentions · p. 385

U.S. Committee Mentions

• AI mentions in U.S. committee reports decreased in the ongoing 118th session, but the session was only about halfway complete and was projected to surpass prior sessions.In the House, Appropriations and Science, Space, and Technology had the most mentions; in the Senate, Appropriations led with 9, followed by Homeland Security and Governmental Affairs with 3.Source: U.S. Committee Mentions · pp. 388–389, Figure 7.2.16 · p. 389, Figure 7.2.17 · p. 389

By Geographic Area

• Since Canada initiated the first national AI strategy in March 2017, 75 strategies have been unveiled; 24 appeared in 2019 and eight more in 2023.The new 2023 strategies came from countries in the Middle East, Africa, and the Caribbean.Source: §7.3 · p. 391, By Geographic Area · p. 391
• As of January 2024, new countries developing national AI strategies within the previous three years included Antigua and Barbuda, Barbados, Belarus, Costa Rica, Jamaica, Pakistan, and Senegal.The research team attempted to identify strategies for every nation, but some may have been missed.Source: By Geographic Area · p. 391

Overview

• 25 U.S. AI-related regulations were recorded in 2023, up from one in 2016; the total grew 56.3% in 2023 alone.Figure 7.4.1 covers U.S. regulations from 2016–23.Source: Figure 7.4.1 · p. 393, Overview · p. 393

By Relevance

• Four of 25 U.S. AI regulations in 2023 were highly relevant—the most since 2016—and issuing agencies rose to 21 from 17; the Executive Office of the President and Commerce each issued five.High- and medium-relevance regulations increased from 2022; HHS and the Industry and Security Bureau each issued four regulations in 2023.Source: By Relevance · pp. 394–395

By Approach

• In 2023, U.S. AI regulations included 10 restrictive versus three expansive actions, while foreign trade and international finance led subject matter with three.In 2020, four regulations were expansive and one restrictive; health, commerce, and science, technology, and communications each had two instances in 2023.Source: By Approach · p. 396, By Subject Matter · p. 397

Overview

• EU AI-related regulations rose from 22 in 2022 to 32 in 2023, after peaking at 46 in 2021.Figure 7.4.6 covers EU regulations from 2017–23.Source: Figure 7.4.6 · p. 398, Overview · p. 398

By Relevance

• Of 32 EU regulations in 2023, two were highly relevant to AI, 13 had medium relevance, and 17 had low relevance.The EU passed its first highly relevant AI regulations in 2021, establishing the Digital Europe Programme and Horizon Europe.Source: By Relevance · p. 399

By Agency

• The Council of the European Union (13) and European Parliament (9) were the leading originators of EU AI regulations in 2023.Source: By Agency · p. 400

By Approach

• In 2023, EU AI regulations were more expansive (12) than restrictive (8), and the leading subjects were science, technology, and communications (5) and government operations and politics (3).Source: By Approach · p. 401, By Subject Matter · p. 402

Federal Budget for AI R&D

• $1.8 billion was allocated to U.S. AI R&D in FY 2023, more than tripling since FY 2018; the FY 2024 request was $1.9 billion.For FY 2024, NSF had the largest NITRD AI R&D request at $531 million, followed by DARPA at $322.1 million and NIH at $284.5 million.Source: Federal Budget for AI R&D · p. 403, Figure 7.5.1 · p. 403, Figure 7.5.2 · p. 404

Index Report 2024 7.5 U.S. Public Investment in AI Chapter 7: Policy and Governance

• The DoD requested $1.8 billion for nonclassified AI RDT&E in FY 2024, up from $1.1 billion in FY 2023.Govini used supervised machine learning and natural language processing to categorize federal contracts, grants, and other transaction authority awards for its spending analysis.Source: Index Report 2024 7.5 U.S. Public… · pp. 405–406, Figure 7.5.3 · p. 405

AI Contract Spending

• $3.3 billion in U.S. government AI spending in 2023 was nearly 2.4 times the 2018 total, with machine learning at $1.5 billion and computer vision at $1.0 billion.Total spending rose marginally from $3.2 billion in 2022; Govini made minor classification-methodology adjustments in 2023, so totals may differ slightly from earlier editions.Source: AI Contract Spending · p. 406, Figure 7.5.4 · p. 406
• In FY 2023, prime contracts made up 50.6% of federal AI contracts and grants 47.6%; the contract share declined while the grant share rose over the prior year.Spending increased significantly for machine learning from FY 2022 to FY 2023, while computer vision and natural language processing spending also rose less prominently.Source: AI Contract Spending · pp. 407–408, Figure 7.5.5 · p. 407, Figure 7.5.6 · p. 408

Microelectronics and Semiconductor Spending

• $3.9 billion was spent by the U.S. government on microelectronics in 2023, up from $2.5 billion in 2022, with most spending allocated as contracts.The report notes that semiconductors such as GPUs have played a crucial role in recent AI technical improvements.Source: Microelectronics and Semiconductor… · p. 409, Figure 7.5.7 · p. 409, Figure 7.5.8 · p. 410

ACCESS THE PUBLIC DATA

• AI developers and training datasets often reflect Western perspectives, and the report warns that limited diversity can perpetuate or exacerbate societal inequalities and biases.The chapter draws on data about North American and European computer science departments, Women in Machine Learning workshop participation, and U.S. secondary computer science education; demographic data, particularly on sexual orientation, remains scarce.Source: ACCESS THE PUBLIC DATA · p. 413

1. U.S. and Canadian bachelor’s, master’s, and PhD CS students continue to grow more

• Since 2011, Asian CS bachelor’s graduates rose 19.8 percentage points and Hispanic CS bachelor’s graduates rose 5.2 percentage points, while white graduates remain most represented.Source: §1 · p. 414
• Every surveyed European country had more male than female informatics, CS, CE, and IT graduates at bachelor’s, master’s, and PhD levels; gaps narrowed slowly in most countries over the last decade.Source: §1 · p. 414
• Female students’ share of AP CS exams rose from 16.8% in 2007 to 30.5% in 2022, while Asian, Hispanic/Latino/Latina, and Black/African American participation increased year over year.Source: §1 · p. 414

CS Bachelor’s Graduates

• Female CS bachelor’s graduates reached 22.2% in 2022; over the decade, Hispanic graduates grew 4.7 times and African American graduates 2.5 times, while Asian representation doubled.Nonbinary/other-identifying graduates accounted for 0.1% in 2022. North American education data comes from the CRA Taulbee Survey; only 182 of 297 targeted departments responded, and the report cautions against per-capita comparisons with European data.Source: CS Bachelor’s Graduates · p. 415, Figure 8.1.1 · p. 415, Figure 8.1.2 · p. 416

CS Master’s Graduates

• Female CS master’s graduates rose from 24.6% in 2011 to 26.3% in 2022, while female CS PhD graduates dipped to 22.1% but retained their long-term trend.Ethnic diversity increased among both groups; in 2022, 41.1% of CS PhD graduates were Asian, Black, Hispanic, multiracial, American Indian, or Native Hawaiian.Source: CS Master’s Graduates · pp. 417–418, CS PhD Graduates · pp. 419–420

Disability Status of CS, CE, and Information Students

• Students reporting a need for disability accommodations remained a small, consistent share year over year: 4.1% of bachelor’s, 1.5% of master’s, and 1.1% of PhD students.The CRA asked departments to report students receiving disability accommodations over the preceding year.Source: Disability Status of CS, CE, and… · p. 421

CS, CE, and Information Faculty

• In 2022, CS, CE, and information faculty were 75.6% male, 24.3% female, and 0.1% nonbinary; women made up 28.0% of new hires, compared with 71.7% men.The share of women faculty has risen only slightly since 2011; the 28.0% share among new hires was above the proportion of new female PhDs.Source: CS, CE, and Information Faculty · pp. 422–423
• White faculty accounted for 57.3% and Asian faculty 30.1% in 2022; the gap between white faculty and the next-largest ethnic group narrowed from 46.1% in 2011 to 27.2% in 2021.These figures concern resident faculty in the United States and Canada.Source: CS, CE, and Information Faculty · p. 424

Informatics, CS, CE, and IT Bachelor’s Graduates

• Across most surveyed European countries, men remained the majority of informatics, CS, CE, and IT bachelor’s graduates; women’s shares were 14.8% in France, 17.8% in the UK, 21.5% in Germany, and 35.2% in Bulgaria.Bulgaria had the highest reported proportion of female graduates among the surveyed countries.Source: Informatics, CS, CE, and IT Bachelor’s… · p. 425, Figure 8.1.15 · p. 426

Informatics, CS, CE, and IT Master’s Graduates

• European informatics, CS, CE, and IT master’s and PhD graduates were predominantly male in most surveyed countries, though the PhD gender gap narrowed in the UK, Germany, and Switzerland and widened slightly in Finland and Spain.The greatest reported female master’s share was 40.4%; PhD proportions can shift sharply year to year in countries with few graduates.Source: Informatics, CS, CE, and IT Master’s… · p. 425

8.2 AI Conferences

• NeurIPS WiML workshop attendance reached 714 in 2023, nearly eight times its 2010 level of 89; 4.4% of conference attendees participated, and 56.4% of survey respondents lived in North America.Europe accounted for 21.8% of respondents, Asia 11.4%, and Africa 8.9%; participation had declined over the prior two years, while North America’s attendee share exceeded 2022’s.Source: Workshop Participants · p. 429, Figure 8.2.1 · p. 429, §8.2 · p. 430

8.2 AI Conferences

• 84.2% of participants at the 2022 WiML workshop identified as female, 10.0% as male, and 3.2% as nonbinary.Source: §8.2 · p. 431, Figure 8.2.4 · p. 431

AP Computer Science: Gender

• In 2022, male students accounted for 68.9% of AP CS exam-takers, female students 30.5%, and students identifying as neither male nor female 0.7%.Female participation has nearly doubled over the past decade.Source: AP Computer Science: Gender · p. 432, Figure 8.3.1 · p. 432

AP Computer Science: Ethnicity

• White students remained the largest group of AP CS exam-takers in 2022 at 38.2%, followed by Asian students at 27.8% and Hispanic/Latino/Latina students at 17.6%.Participation by Asian, Hispanic/Latino/Latina, and Black/African American students has grown over time.Source: AP Computer Science: Gender · p. 433

4. Demographic differences emerge regarding AI optimism. Significant demographic

• 59% of Gen Z respondents expect AI to improve entertainment options, compared with 40% of baby boomers.Higher-income and more-educated respondents are also more optimistic about AI’s positive effects on entertainment, health, and the economy than their lower-income and less-educated counterparts.Source: §4 · p. 438

5. ChatGPT is widely known and widely used. An international survey from the University of Toronto

• 63% of respondents are aware of ChatGPT, and around half of those aware report using it at least once weekly.Source: §5 · p. 438

AI Products and Services

• 66% expect AI to greatly change their lives in the near future, while 52% feel nervous about AI products and services, up 13 percentage points from 2022.Only 37% expect AI to improve their job, 34% think it will boost the economy, and 32% think it will enhance the job market; 54% say AI’s benefits outweigh its drawbacks, and about half trust AI companies to protect data.Source: §2 · p. 438, AI Products and Services · p. 439, Figure 9.1.1 · p. 439
• 78% of Indonesians, 74% of Thai respondents, and 73% of Mexicans view AI products and services as more beneficial than harmful, versus 37% of Americans.The United States and France were most skeptical among 31 countries; positivity rose in previously critical Western countries, including the Netherlands, where it increased from 33% in 2022 to 43% in 2023.Source: AI Products and Services · p. 440

AI and Jobs

• 57% think AI will change how they perform their current job within five years, and 36% fear it may replace their job.Gen Z and millennial respondents are more likely than Gen X and baby boomers to expect AI to change how they do their jobs.Source: AI and Jobs · pp. 444–445
• 66% of Gen Z respondents, compared with 46% of boomers, agreed that AI would likely affect their current jobs.People with higher incomes, more education, and decision-making roles were also more likely to foresee effects on their employment.Source: Figure 9.1.6 · p. 445, AI and Jobs · p. 445

AI and Livelihood

• Only 37% think AI will improve their job, and just 32% believe it will enhance the job market, compared with 39% who feel AI will benefit their health.Source: AI and Livelihood · p. 446

Attitudes on ChatGPT

• Figure 9.1.9 explored global public awareness of ChatGPT: 63% of global respondents claimed awareness, and India was among the countries with highest awareness, at 82%.Source: Attitudes on ChatGPT · p. 448

AI Concerns

• 49% of global respondents were concerned about AI misuse and job impacts, while 45% worried about privacy violations.Concern was comparatively lower about unequal access (26%), bias and discrimination (24%), and respondents’ own ability to use AI (22%).Source: Figure 9.1.11 · p. 451, AI Concerns · p. 451
• 52% of Americans in 2023 felt more concerned than excited about AI, up from 37% in 2021 and 38% in 2022.Although 49% said AI was more likely to help people find products or services online, 53% said it was more likely to harm personal-information privacy.Source: Figure 9.1.12 · p. 452, AI Concerns · p. 452, Figure 9.1.13 · p. 452
• Americans with higher education levels were more likely to believe AI could help than harm, including in doctors’ care delivery and online product discovery.The report describes this pattern across multiple use categories.Source: Figure 9.1.14 · p. 453, AI Concerns · p. 453

Dominant Models

• In Q4 2023, GPT-4 captured 45% of social-media attention about AI models, despite the release of numerous other models.Grok, Stable Diffusion, and Gemini also attracted significant attention.Source: Figure 9.2.2 · p. 455, Dominant Models · p. 455
• GPT-4 discussion after its March 2023 launch praised faster processing, improved accuracy, and productivity, while complaints cited crashes and debate over a proposed training pause.GraphCast and Claude 2.1 were among the 2023 models that received especially positive social-media sentiment.Source: Dominant Models · pp. 454, 456
• GPT-4 Turbo drew praise for longer conversations, improved contextual understanding, and image generation, but criticism cited its April 2023 knowledge cutoff and slower loading than GPT-4.Some feedback described the new capabilities as transforming programmers’ workflows.Source: Dominant Models · p. 456
• About 16 times more posts mentioned Stable Diffusion XL models than Stable Diffusion 2.0, while Stable Diffusion discussions declined toward the end of 2023.Positive sentiment reflected the tool’s growing popularity, creative potential, and technical improvements; negative sentiment raised legal and ethical concerns, potential harms, misinformation, and academic cheating.Source: Dominant Models · p. 457
• Gemini and Grok conversations increased in Q4 following their late-year launches; Gemini feedback praised accuracy, multilingual capabilities, and potential improvements to Google services, but also cited inaccurate results and launch skepticism.Negative opinions also reflected disappointment over Gemini’s delayed release and skepticism toward its AI demo.Source: Dominant Models · p. 457

AI Conference Attendance

• Conference attendance totals came from organizers when requested, or from totals already posted online.The AI Index contacted conference organizers in 2023, but used publicly reported totals without contacting organizers when available.Source: AI Conference Attendance · p. 460

Publications From CSET Merged Corpus of Scholarly Literature

• CSET deduplicated publications from six scholarly sources and analyzed English-language AI papers published since 2010 using an arXiv-derived classifier and MAG field labels.The merged corpus includes Web of Science, OpenAlex, The Lens, Semantic Scholar, arXiv, and Papers With Code.Source: Publications From CSET Merged Corpus… · pp. 460–461
• Over 70% of English-language AI papers published between 2010 and 2022 have citation data available.Publications without citation counts were excluded from the citation analysis.Source: Publications From CSET Merged Corpus… · p. 461

Patents From CSET’s AI and Robotics Patents Dataset

• AI patents were identified by CPC/IPC codes and keywords, counted by patent family and first publication year, and assigned countries by first filing, with same-day filings potentially attributed to multiple countries.Supranational filing organizations are categorized as “Rest of World.”Source: Patents From CSET’s AI and Robotics… · pp. 461–462

Ecosystems Graph Analysis

• For foundation models, the Ecosystems Graph analysis assigns authors to countries by credited affiliation, aggregates national contributions over time, and compares them for trends.Authors at international organizations are assigned to the headquarters country unless a more specific location is given.Source: Ecosystems Graph Analysis · p. 462
• The Epoch analysis uses a January 1, 2024 dataset snapshot of selected landmark-model papers, assigns authors to countries by affiliation, and aggregates contributions across time periods.The papers are selected using importance, relevance, and uniqueness criteria.Source: Ecosystems Graph Analysis · p. 463

Identifying AI Projects

• GitHub’s public AI repository dataset combines AI/ML and generative AI topic labels with repositories depending on PyTorch, TensorFlow, or OpenAI Python libraries.The identification methodology was developed with researchers from Harvard Business School, Microsoft Research, and Microsoft’s AI for Good Lab.Source: Identifying AI Projects · p. 463

Mapping AI Projects to Geographic Areas

• Public AI projects are mapped geographically using owners’ IP-based locations, with each project assigned its owners’ modal location for the year.If an owner changes location during the year, the mode is determined from the owner’s daily sampled locations.Source: Mapping AI Projects to Geographic Areas · p. 463
• The cost dataset included 48 models with sufficient information to estimate training costs, out of 138 qualifying large-scale ML systems.Costs were calculated by multiplying hardware hours by historical hourly cloud rental rates; the estimates represent using the developers’ training hardware, though actual costs may differ when developers purchased hardware.Source: Mapping AI Projects to Geographic Areas · pp. 463–464
• Training-cost estimates were limited by undisclosed training durations or hardware and unavailable cloud-compute prices.The report notes that an expanded dataset and additional hardware prices would appear in a forthcoming Epoch report.Source: Mapping AI Projects to Geographic Areas · p. 464

Environmental Impact

• The Index surveyed prominent foundation-model technical reports for carbon-emissions disclosures, reviewed carbon-footprint estimates, and searched model documentation for RLHF use.RLHF use was assigned to the model’s release year.Source: Environmental Impact · p. 466

Job Postings Data

• Lightcast mined job postings collected since 2010, aggregating postings from over 51,000 online sites, removing duplicates, and extracting job and qualification details.Extracted fields include job title, employer, industry, region, experience, education, and skills.Source: Job Postings Data · p. 478
• Over 99% of total labor demand is represented in Lightcast data, based on comparisons with JOLTS.Jobs not posted online are usually in small businesses or union hiring halls.Source: Job Postings Data · p. 478

Measuring Demand for AI

• A job posting counted as an AI job if its text mentioned any skill in Lightcast’s list of AI skills.The list drew on Lightcast’s taxonomy of over 30,000 skills; all skills shown in the report were counted as AI skills.Source: Measuring Demand for AI · p. 478
• The demand vocabulary spans AI inference and AIOps, autonomous systems, generative AI, and natural language processing.Listed terms include ChatGPT, autonomous vehicles, and AI copywriting.Source: Measuring Demand for AI · p. 479

Appendix Chapter 4: Economy

• The economy appendix lists natural-language processing terms alongside neural-network concepts and tools.Examples include natural language understanding, semantic search, Apache MXNet, and neural networks.Source: Appendix Chapter 4: Economy · p. 479
• The appendix also catalogs machine-learning methods and tools, plus robotics concepts and systems.Examples include AdaBoost, Apache Spark, reinforcement learning, motion planning, and robot operating systems.Source: Appendix Chapter 4: Economy · pp. 479–480
• Visual image recognition is represented by terms covering reconstruction, image analysis, tracking, and object recognition.The list includes 3D reconstruction, computer vision, face detection, and pose estimation.Source: Appendix Chapter 4: Economy · p. 480

Country Sample

• Countries are sampled when LinkedIn covers at least 40% of the labor force and they have at least 10 AI hires in any month; India is included despite missing the coverage threshold.India is included because of its increasing importance in the global economy.Source: Country Sample · p. 480

Skills (AI Engineering and AI Literacy skills)

• LinkedIn members self-report skills; its taxonomy identifies over 41,000 standardized skills in 249 groups, and the AI skills list now distinguishes AI Literacy from AI Engineering.The AI Literacy additions include ChatGPT, DALL-E, GPT-3, GPT-4, Generative Art, GitHub Copilot, Google Bard, Midjourney, Prompt Engineering, and Stable Diffusion.Source: Skills (AI Engineering and AI Literacy… · p. 480, AI Skills List Update Compared to Last… · p. 483

AI Skills Penetration

• AI skills penetration is the share of AI-group skills among an entity’s 50 TF-IDF-ranked characteristic skills, whose genome down-ranks ubiquitous skills.The skill genome is an ordered list of an entity’s 50 most characteristic skills; the penetration measure uses self-added skills and reweights their frequencies with TF-IDF.Source: Skills Genome · p. 480, AI Skills Penetration · p. 481

AI Jobs or Occupations

• Occupation representatives group roles and specialties regardless of seniority; AI jobs are representatives where AI-skill prevalence signals skills are required, while AI talent has explicit AI skills or an AI occupation.Country-level talent concentration compares AI-talent counts with LinkedIn membership and may be affected by LinkedIn coverage, so it should be used cautiously.Source: Jobs or Occupations · p. 481, AI Jobs or Occupations · p. 481, AI Talent · p. 481

Global Comparison

• Country comparisons use relative AI-skill penetration across overlapping occupations; industry results decompose this by sector, while gender comparisons use same-gender global averages and across-gender comparisons share one global average.The within-gender metric supports country rankings only within each gender, not comparisons between genders within a country.Source: Global Comparison · p. 482, Global Comparison: By Industry · p. 482, Global Comparison: By Gender · p. 482

Relative AI Talent Hiring Rate YoY Ratio

• 16.8%: India’s AI-talent hiring relative to overall hiring grew year over year in 2023, using a 12-month moving average of the ratio’s year-over-year change.The ratio is calculated by dividing AI Hiring Rate by Overall Hiring Rate within a country before calculating its year-over-year change.Source: Relative AI Talent Hiring Rate YoY Ratio · pp. 482–483

AI Talent Migration

• LinkedIn migration is inferred from members’ self-reported location changes; net flows are arrivals minus departures and are normalized by the country of interest’s year-end membership, then multiplied by 10,000.Migration data is available from 2019 onward, and the normalized metric represents relative talent migration into and out of the country from all countries.Source: AI Talent Migration · p. 483, Appendix Chapter 4: Economy · p. 483

AI Skills List Update Compared to Last Year

• The appendix adds skills including generative AI, large language models, image and text generation, and Responsible AI to the updated list.The passages also list autoencoders, chatbots, transformer models, and video generation among additions.Source: AI Skills List Update Compared to Last… · pp. 483–484

Search, Data Sources, and Scope

• Quid indexes over 8 million public and private company profiles, links documents by similar language into topic clusters, and identifies trends from keywords, people, companies, institutions, and metadata.Company descriptions can be searched and filtered using investment and firmographic information, including founding year and headquarters location.Source: Search, Data Sources, and Scope · p. 484

Data

• Quid combines Capital IQ data with Crunchbase additions to capture early-stage startups and funding events, and updates company information weekly.The report also analyzes Fortune 500 earnings-call transcripts for AI-related keyword mentions from January 2018 through December 2023.Source: Data · pp. 484–485

Search Parameters

• Quid searches company descriptions and websites with Boolean queries, filters results by company metadata, and visualizes companies by semantic similarity, selecting up to 7,000 relevant firms.The listed Boolean query uses “artificial intelligence,” “AI,” “machine learning,” or “deep learning.”Source: Search Parameters · p. 485

Companies

• Global AI and ML company searches covered investments from 2013 to 2023, including a separate sample requiring more than $1.5 million in the same 10-year period.The company search also included private investments, IPOs, and M&A.Source: Companies · p. 485

Target Event Definitions

• Private placements and minority investments involve stakes often or necessarily below 50%, while Quid defines M&A as acquisition of more than 50%.The Corporate Activity–Industry Adoption data came from a McKinsey survey of 1,684 participants, with results weighted by each respondent’s nation’s contribution to global GDP.Source: Target Event Definitions · pp. 485–486

Stack Overflow

• The 2023 Stack Overflow Developer Survey supplied AI-use data from 89,184 software developers in 185 countries.Source: Stack Overflow · p. 486

State-Level Data

• Code.org’s state-level education data is described in a policy primer that provides state-by-state detail on American K–12 education.The accompanying link contains Code.org’s data-collection methodology.Source: State-Level Data · p. 491

AP Computer Science Data

• AP Computer Science data came to Code.org under its agreement with the College Board and from the Board’s national and state summary reports.Source: AP Computer Science Data · p. 491

Access to Computer Science Education

• The CRA Taulbee Survey reaches doctoral departments and tracks overall CS production, but historically covers one-quarter to one-third of U.S. BS CS recipients and has a lower share of women graduates than overall.The Index reused CRA methodological notes from earlier editions.Source: Access to Computer Science Education · p. 491

Impact Research

• Two Impact Research surveys released in 2023 supplied data on ChatGPT use among teachers and students.The surveys were released in March and July.Source: Impact Research · p. 492
• Informatics Europe compiles higher-education statistics from official sources where a solid, reasonably complete national picture is available, using national definitions and concepts.The dataset omits aspects not reported by consulted agencies, and country-specific terminology for informatics creates integration challenges.Source: Impact Research · p. 492

Studyportals

• Studyportals classifies programs across 15 disciplines and 284 subdisciplines, with processors manually assigning one to three closest-fitting subdisciplines.Assignments follow scenarios including direct matches, interdisciplinary matches, specializations, combinations, and closest approximations.Source: Studyportals · p. 493

Global AI Mentions

• For global AI mentions in legislative proceedings, the AI Index searched for “artificial intelligence” on 80 countries’ parliamentary or congressional websites, generally in proceedings records.Searches used the respective languages; some databases allowed searches only by title, requiring use of site search functions.Source: Global AI Mentions · p. 495
• For AI-related laws, the Index searched bill full texts across 128 countries and counted state-level laws signed into law from 2016 to 2023, excluding repealed laws.Regulations and standards were outside this analysis; future reports hoped to include them.Source: Global AI Mentions · p. 495
• Two human coders classified legislation by AI relevance, regulatory approach, and subject matter, with a third coder resolving disagreements.AI relevance was categorized as low, medium, or high; regulatory approach as expansive or restrictive; and subject matter used the Congress policy typology.Source: Global AI Mentions · p. 496
• The EU regulation search used EUR-Lex and included only documents with binding regulatory authority, limited to legal acts, international agreements, and consolidated texts.Source: Global AI Mentions · p. 496

Govini

• Govini’s Ark platform and National Security Knowledge Graph fuse categorized data to depict federal spending and its supporting vendor ecosystem over time.The Index describes the resulting data as high-fidelity and comprehensive.Source: Govini · p. 497

Countries with AI Strategies in Place

• The Index identified countries with AI strategies in place, including Algeria, Australia, China, India, the United Kingdom, and the United States.An asterisk marks cases where the strategy itself was not found and a confirming news article was linked instead.Source: Govini · p. 497, Countries with AI Strategies in Place · p. 498

Countries with AI Strategies in Development

• Countries identified as developing AI strategies include Andorra, Armenia, Costa Rica, Kenya, Morocco, New Zealand, Nigeria, and Uzbekistan.Source: Countries with AI Strategies in… · p. 498

US AI Regulation

• The U.S. agency-regulation analysis covers 2016–2023 and used Federal Register keyword searches, followed by coding for AI relevance, regulatory approach, and subject matter.Two human coders performed the coding, with a third resolving disagreements.Source: US AI Regulation · p. 498

US Committee Mentions

• The U.S. committee-mentions search used the keyword “artificial intelligence” and the Committee Reports filter.Source: US Committee Mentions · p. 499

Index Report 2024 Chapter 9: Public Opinion

• The survey project collected 23,882 responses across 21 countries and 12 languages, using census-targeted samples of over 1,000 people in each country.It was conducted by the Schwartz Reisman Institute for Technology and Society and the Policy, Elections, and Representation Lab at the University of Toronto.Source: Index Report 2024 Chapter 9: Public… · p. 501

Pew Research

• The report uses data from the 2023 Pew Research Center survey but does not republish its methodology.Source: Pew Research · p. 501
• Quid analyzed 6.69 million social-media posts from 2023 to assess perceptions of AI model releases, with reactions addressing technical advances, business efficiency, and ethical considerations.Source: Pew Research · p. 501