From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolution

• AIR’s dramatic progress followed and largely resulted from ceding autonomy to external actors and adopting narrow evaluation, but this created a monoculture focused on scaling deep learning.The authors describe this history as a challenge to the idea that autonomous research is necessary for scientific progress.Source: §1 · p. 3
• Deep learning’s dominance displaced other research trajectories and brings epistemic, ethical, and environmental tradeoffs, despite its accomplishments challenging the need for autonomous research.The paper notes that scaling requires more computing, electricity, and data, while internet-scale data curation can make addressing social biases and privacy or intellectual-property concerns difficult.Source: §1 · p. 3

The Era of Symbolic AI (1955-1987)

• Competing theories and agendas, followed by tribalism and unmet expectations, turned the early fecundity of symbolic AI into the second AI Winter.Source: §1 · p. 4

The Benchmarking Era (1987-2011)

• Frustrated funders introduced benchmarking as a formal alternative to basic science, narrowing evaluation to predictive accuracy on applied tasks and favoring probability- and statistics-based machine learning.This focus set aside values prized in basic science, including explainability, theoretical consistency, parsimony, and efficiency.Source: §1 · p. 4

The Deep Learning Era (2012-2021)

• Deep learning became dominant not through major intellectual advancement but through access to large datasets and computing; tech firms’ investment and strategic orientation entrenched its monoculture.Source: §1 · p. 4

Discussion

• As other sciences adopt deep learning for epistemic purposes, they may also adopt AI research’s organizational and technological orientations, while model opacity increases reliance on benchmarking that canalizes scientific values.Source: §1 · p. 4

• Deep learning’s scale requirements will make academia increasingly reliant on industry’s capacities and interests to build foundation models for scientific research.Source: §1 · p. 4
• The spread of monoculture via deep learning may alter the 20th-century model of science.Source: §1 · p. 4
• The paper discusses generative AI’s organizational and epistemic implications.Source: §1 · p. 4

2 BACKGROUND: PROGRESS AND AUTONOMY IN SCIENCE

• Scientific evaluation spans intertwined conceptual, technological, and organizational dimensions through which claims move from research frontiers into a settled core.The section argues that understanding evaluation requires examining all three dimensions together.Source: §2 · p. 5

2.1 THE ORGANIZATIONS, EPISTEMOLOGIES, AND TECHNOLOGIES OF SCIENTIFIC EVALUATION

• Organic evaluation is decentralized, plural in values, and slow, while formal evaluation centralizes assessment around a narrow value set and standardizes procedures for clear feedback.Organic evaluation unfolds as researchers adopt or ignore claims over time; formal evaluation is often used when clear, timely, actionable information is wanted.Source: §2.1 · pp. 6–7
• Organic evaluation can leave progress and usefulness unclear over long timelines, whereas formal metrics offer clarity but embed values that constrain fields’ autonomy.The authors frame this as an inherent tension between evaluation regimes.Source: §2.1 · p. 7

2.2 EVALUATION AND AUTONOMY

• Although scientific communities often evaluate work for fellow researchers, funding, policy, and external audiences also shape evaluation, making that interface inherently fraught.Postwar support paired autonomy for developing systematic theory with expectations of technological, military, economic, and social benefits.Source: §2.2 · pp. 7–8
• Unlike speaking with colleagues in a field’s “core-set”, interactions with funders, industry, government, or the public risk undermining and swaying scientists’ “pure” motives through market, theater, and tribe.Source: §2.2 · p. 8
• In the first period, largely autonomous AI research pursued basic science but entered an “AI Winter” when unstructured evaluation failed to produce consensus.Source: §2.2 · p. 8
• DARPA shifted funding from open-ended “blue sky” grants toward narrowly specified tasks with clear metrics, and benchmarks enabled researchers to chart progress and compare approaches.The paper presents this as the second period in its history of AI research.Source: §2.2 · p. 8
• Greater data access and compute favored deep learning, whose benchmark success crowded out competing approaches and helped make machine learning an epistemic monoculture.The paper characterizes that monoculture by a handful of organizations, a constrained research trajectory, and a single epistemic value.Source: §2.2 · pp. 8–9
• The authors argue that task-driven science is a distinct mode of pursuing progress, not merely a derivative of basic research, with both advantages and risks.Source: §2.2 · p. 9

3.1 INTERVIEWS

• The study’s interviews draw on participants connected to early AI, the transition to machine learning, and deep learning research across government, industry, and academia.The authors report 45 in-depth, open-ended interviews and supplement early-era interviews with archival sources.Source: §3.1 · p. 9

3.2 ARCHIVAL RESEARCH

• Archival research uses early-AI documents and talks alongside publications and conference proceedings spanning later decades.Sources include the Edward Feigenbaum Archives, AI Magazine, recordings and transcripts, and publications from major AI and ML venues.Source: §3.2 · pp. 9–10

3.3 COMPUTATIONAL ANALYSIS

• The computational analysis tracks topic publication rates in 90,000 papers from 14 AI venues published between 1993 and 2018.Keywords and labeled topics identify five AI approaches; LiteRate estimates publication rates and credible intervals and flags statistically significant shifts.Source: §3.3 · p. 10

4 1956 TO THE MID-80s: THE ERA OF SYMBOLIC AI

• AI stagnated as a proto-science, with parallel projects unable to build shared theoretical language or empirical methods to adjudicate debates.The field did not develop rigorous evaluation mechanisms or a shared agenda.Source: §4 · p. 10

4.3 ORGANIZATIONAL AUTONOMY AND DIVISION

• Exceptional funder autonomy encouraged fractured, personality-driven networks, while the field lacked a shared agenda and rigorous evaluation and experienced recurring AI Winters.Without theoretical consensus or empirical progress, fields can fail to maintain funder interest.Source: §4 · p. 10, §4.3 · p. 13

4.1 THE EPISTEMIC TRIBALISM OF EARLY AI

• From Dartmouth onward, researchers disagreed on what “artificial intelligence” meant and which epistemic values made a theory good, producing distinct schools rather than a coherent paradigm.The mind and brain were too poorly understood to settle competing appeals to realism, parsimony, or mathematical formalism.Source: §4.1 · pp. 11–12

4.2 THE TECHNOLOGICAL LIMITATIONS OF EARLY AI

• Limited computing confined ambitious AI theories to toy problems, and symbolic approaches faced combinatorial explosion as tasks grew more complex.The assumption that toy-task success would scale to complex problems was rarely supported.Source: §4.2 · p. 12

4.3 ORGANIZATIONAL AUTONOMY AND DIVISION

• DARPA granted elite AI researchers exceptional autonomy, and a small insider group helped set the agency’s agenda and organize the field.Feigenbaum characterized ARPA as politically divided between insiders and outsiders, and recalled senior researchers planning projects together.Source: §4.3 · pp. 13–14
• AI evaded accountability: program managers lacked technical expertise to assess funded work, and the field relied on hype rather than measured progress.The section argues that external accountability and evaluation systems help drive consensus formation and provide science with epistemic and social legitimacy.Source: §4.3 · pp. 14–15

4.4 THE AI WINTER OF THE LATE 1980s

• Expert systems could not naturally generalize because humans had to hand-encode knowledge relationships, making them brittle and contributing to the field’s collapse (Schank, 1987, 1991).The collapse also reflected shifting market forces and ambitious expected advances that never materialized.Source: §4.4 · p. 15
• The late-1980s AI winter shuttered startups and dried up funding; no-strings grants and plentiful academic positions disappeared as symbolic AI’s challenges had no clear path to resolution.The collapse followed limitations of expert systems, shifting market forces, and expected advances that never materialized.Source: §4.4 · pp. 15–16

5 1980s TO 2012: THE BENCHMARKING ERA

• Benchmarking replaced organic, multi-value evaluation with predictive accuracy, helping AI move beyond GOFAI’s theoretical and material impasses through machine learning.The paper identifies this mono-value optimization system, alongside the displacement of organic evaluation, as a primary seed of AI monoculture.Source: §5 · p. 16

5.1 THE ORGANIZATION OF THE FIELD THROUGH BENCHMARKING

• The Common Task Framework measured performance on unseen datasets, defining AI progress as predictive accuracy on specific commercial and military tasks rather than theoretical disputes or holistic general AI.It recast AI scientists as task-focused engineers, while requiring the field to cede autonomy to external actors who set problems and tasks.Source: §5.1 · p. 17
• Benchmarking clarified progress and enabled buy-in when funding was scarce, but it gave DARPA authority to choose problems, tasks, participants, and monetary rewards.This arrangement displaced scientists’ autonomy to define the field’s norms and priorities.Source: §5.1 · p. 17

5.2 TECHNOLOGICAL ADVANCES IN COMPUTE AND DATA ACCESS

• Benchmarking showed decisively that statistical algorithms, which could probabilistically handle ambiguities in real data, were more effective than rule-based systems; an HMM also trounced a rule-based system decisively.Source: §5.2 · p. 18
• Leaderboards charted state-of-the-art accuracy over time, distinguishing steady progress from major innovations.Large benchmark-score leaps signaled major innovations.Source: §5.2 · p. 18
• Benchmarking spread from speech and language research to vision and other AI communities, while corporate sponsors began rivaling or supplanting DARPA/NIST.State-of-the-art benchmark results became a necessary, though not sufficient, criterion for acceptance at major conferences.Source: §5.2 · pp. 18–19

5.3 CONSTRAINED EPISTEMIC PLURALISM

• The section opens by noting that scientific contributions are value-laden and that epistemic virtues are valued for promoting epistemic goals (Dotan (2021)).The supplied passage introduces constrained epistemic pluralism but does not provide its substantive argument.Source: §5.3 · p. 19
• Predictive accuracy crowned machine learning’s evaluative values and let the field grow without consensus on the best AI approach, while peer review and theory kept accuracy from being the only value.An algorithm achieving state-of-the-art empirical performance was considered worthy of attention.Source: §5.3 · p. 19
• Learning curves paired training-example count (data efficiency) with accuracy (generalization), and expectations for these plots incentivized data-efficient algorithms.The figure contrasts learning curves with accuracy-only benchmarking tables.Source: §5.3 · p. 19, Figure 2 · p. 21
• PAC learning emphasized computational and data efficiency alongside accuracy, while statistical learning theory paired generalization with parsimony.PAC efficiency included polynomial-time computation and limited training data; SLT treated simplicity as a tradeoff with generalization.Source: §5.3 · p. 21
• Fashions shifted from explainable “white box” decision trees and Bayesian networks in the 1980s and early 1990s to SVMs in the mid-1990s, combining empirical performance with attractive statistical properties and guarantees, then non-parametric Bayesian graphical models in the 2000s.Source: §5.3 · p. 22
• The diverse theories and methods that emerged made machine learning productive, and its tools remain useful when deep-network training is impractical or other epistemic values matter.These tools are used by data scientists and academics around the world.Source: §5.3 · p. 22

6 2012 TO 2021: THE ERA OF DEEP LEARNING MONOCULTURE

• After 2013, deep learning research exploded while other methodological paradigms faded, as benchmarking rewarded safe iterations toward state-of-the-art accuracy.The authors attribute the field’s narrowness partly to incentives to achieve state-of-the-art results with minimal resource investment.Source: Figure 3 · p. 23, §6 · p. 24
• Scaling traded accuracy gains against explainability, parsimony, and theoretical consistency, while reducing reliance on slower peer review and formal theory.The authors argue this helped the field grow and advance faster than before.Source: §6 · p. 24

6.1 THE TECHNICAL RAMIFICATIONS OF (REALLY) BIG DATA AND COMPUTE

• The ImageNet 2012 competition was influential for revealing material affordances: by 2007 neural-network training could be vastly accelerated using NVIDIA graphical processing, alongside the rise of large computer-vision datasets.At the time, ImageNet consisted of 1.3 million images.Source: §6.1 · pp. 24–25
• Avoiding laborious, expensive data labeling unlocked far larger datasets; Transformers leveraged text, image, and graph structure to greatly improve ANN performance, though their material needs for input text scaled quadratically.Source: §6.1 · pp. 25–26

6.2 ORGANIZATIONAL POWER SHIFTS FROM ACADEMIA AND GOVERNMENT TO INDUSTRY

• Scaling’s greater data and compute demands shifted power toward large IT firms, while Yelp and Netflix funded benchmarking competitions and Google and Facebook were positioned to dominate.The passage identifies Google and Facebook (now Meta) as uniquely positioned because data underpins their businesses.Source: §6.2 · p. 26
• Large IT firms’ data, computing, talent, and investment shifted agenda-setting power from government and academia to industry, whose influence extends to benchmark-defined tasks.The passage links the concentration of research resources at large firms to their power to shape the field’s agenda.Source: §6.2 · pp. 26–27
• Industry did not make academia irrelevant: shared papers and open-source resources kept academics involved in company-relevant research and PhD training, while proprietary data distinguished firms.The passage says models rarely changed during the scaling years, making large proprietary training datasets a source of industry value.Source: §6.2 · p. 27
• Corporate incentives favored predictable scaling over high-risk basic research, despite diminishing returns, because extra data and computation offered a more guaranteed path to benchmark gains.An Apple scientist characterized intellectual breakthroughs as unpredictable compared with adding data and computation; Chollet described years of diminishing returns from scaling.Source: §6.2 · pp. 27–28

6.3 THEORY AND PEER REVIEW IN A BENCHMARKING WORLD

• Scaling’s accuracy gains displaced other epistemic values: it relied on vast compute and datasets, while networks with billions of neurons remained opaque and limited in explanatory power.The passage contrasts scaling with statistical learning theory’s emphasis on parsimony and notes that large networks are difficult to interpret mechanically.Source: §6.3 · p. 28
• Large ANN training lacks convergence guarantees and relies on tacit knowledge, intuition, trial and error, and potentially thousands of false starts.The passage says this makes theoretical discussions of convergence somewhat irrelevant in practice.Source: §6.3 · p. 28
• With benchmark progress as the singular value, theory and peer review became optional and declined as slow, laborious processes.The passage describes theory and peer review as organic institutions that bridge multiple values, and notes that formal theory-building and peer review were on the decline.Source: §6.3 · p. 29
• Benchmarking’s speed, immediate verifiability, and easy intelligibility were key factors in the field’s unprecedented acceleration over the past decade.Source: §6.3 · p. 29

7 DISCUSSION

• Benchmarking favored statistical learning over symbolic AI, and data and compute later enabled deep learning to dominate benchmarks through scaling.This era also shifted research gravity toward tech firms with the resources to pursue scaling, while diminishing emphasis on organic evaluation and theory-building.Source: §7 · pp. 29–30
• Ceding autonomy and adopting benchmarks aligned AIR’s organization, epistemic orientation, and resources into an efficient but narrow monoculture.External actors could streamline research agendas, while benchmarks made progress on specific tasks simple and verifiable.Source: §7 · p. 30
• Benchmarking accelerates research by narrowing the problem space and making progress verifiable, but SOTA accuracy rewards incremental advances over innovation.The authors argue that external actors, especially industry, favor low-risk work with immediate applications, even when it may have less long-term potential.Source: §7 · p. 30
• Control of data, compute, and money shifted power toward industry, leaving academia dependent on firms for grants, data, pretrained models, and compute.The paper also describes academia as becoming a training pipeline for industry.Source: §7 · p. 31

7.1 THE SOCIAL IMPLICATIONS OF AIR MONOCULTURE

• The success of AIR’s monoculture challenges the belief that basic research is necessary for scientific progress, though whether monocultures are normatively better remains unresolved.Source: §7.1 · p. 31
• Large-scale deep learning is less parsimonious, explainable, and data- or compute-efficient than earlier approaches, and its opacity can perpetuate social inequalities.The discussion links its resource demands to electricity use and large-scale data collection to privacy and intellectual-property disputes.Source: §7.1 · pp. 31–32
• Despite these costs, deep learning has produced expert-beating classifiers and now handles open-ended, high-skill tasks, advancing toward general AI through commercial applications.The authors argue that this progress arose from a market for artificial customer-service agents, not autonomous exploration or a theory of mind.Source: §7.1 · p. 32

7.2 THE DIFFUSION OF THE DEEP LEARNING MONOCULTURE ACROSS THE SCIENCES

• The authors contend that deep learning and generative AI’s greater black-box and scale-related compromises may spread AIR’s epistemological and organizational features across sciences.Black-box models have already prompted debate about prediction versus explanation, and machine learning has been applied to problems considered taskifiable.Source: §7.2 · p. 32
• AlphaFold effectively solved protein-folding prediction, yet researchers remain no closer to understanding how folding occurs, sharpening the tension between predictive and explanatory goals.The authors present this as a crossroads for structural biology and other sciences: build on the opaque solution or continue treating protein folding as a theoretical problem.Source: §7.2 · p. 33
• Benchmarking might help sociology address unresolved social explanations, but the Fragile Families Competition could not overcome dataset sparsity and benchmark-driven publication could reduce autonomy.The authors suggest outcomes may change as models evolve, while noting that a journal ranking work by highest R2 on an inequality task could help sociology elide an intractable project.Source: §7.2 · p. 33
• Scaling’s data and compute needs are centralizing repositories and increasing reliance on technology companies, potentially forcing universities to cede compute autonomy or fall behind.The discussion describes cloud compute, paid access to large models, and early public-funded supercomputer initiatives as part of this organizational shift.Source: §7.2 · pp. 33–34
• Finance and medicine face epistemic and organizational dilemmas similar to science as they rely more on black boxes and the technology industry.Source: §7.2 · p. 34

7.3 CONCLUSION: THE TRADEWINDS OF GENERATIVE AI

• Generative AI challenges basic science by showing task-driven monoculture can approach general AI, while dynamic crowd-sourced benchmarks may be organic yet opaque.The authors say it is too early to tell whether AI research’s monoculture is in flux; crowd-sourced model comparisons may weigh multiple values, but users’ criteria and expertise remain unclear.Source: §7.3 · p. 34
• Up to 15% of peer reviews in top AI conferences contain LLM-generated content, while OpenAI’s commercialization curtailed academic access to full papers and code.The authors suggest this may require reconsidering peer review and say academia’s role in state-of-the-art research became increasingly murky.Source: §7.3 · p. 34
• Expensive scaling has prompted efficiency innovations and open models that give academic research oxygen, but they have not restored autonomy or erased the resource gap.Training Llama 2’s 70 billion parameters from scratch is estimated to cost 1-2 million dollars, and open models may lag enterprise-scale state-of-the-art systems while scaling remains important.Source: §7.3 · pp. 34–35