Source-linked AI summary
The Hardware Lottery
Sara Hooker
TL;DR
This essay examines how available hardware and software can disproportionately determine which AI research ideas succeed, rather than their superiority. It introduces the hardware lottery concept and uses historical examples to argue that specialized hardware may widen differences between research directions while making alternatives costlier to explore.
Problem
AI research tooling has disproportionately influenced which ideas succeed or fail, even though breakthroughs may require different algorithm, hardware, and software combinations.
Method
The essay defines the hardware lottery and illustrates it through historical examples linking research outcomes to hardware and software compatibility.
Results
Historical hardware lotteries delayed recognition of deep neural networks, while GPUs later enabled efficient training and deeper networks.
Takeaways & Limitations
As hardware becomes more heterogeneous and specialized, research progress may become more uneven, increasing the value of quantifying the opportunity cost of existing hardware and software.
Takeaways & Limitations
Exploring broader hardware-software-algorithm combinations is constrained by chip development costs of $30-80 million and development times of 2-3 years.
Abstract
from arXiv · showhide
Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas. This essay posits that the gains from progress in computing are likely to become even more uneven, with certain research directions moving into the fast-lane while progress on others is further obstructed.
1 Introduction
The essay argues that artificial intelligence research has been disproportionately shaped by available software and hardware. It introduces the hardware lottery to explain why compatible ideas can succeed over superior alternatives, with increasingly specialized hardware potentially widening uneven progress.
- Tooling has introduced noise and inertia into scientific progress by influencing which artificial intelligence research directions succeed or fail.
- The hardware lottery describes research ideas winning because they fit available software and hardware rather than because they are superior alternatives.
- Closer collaboration among hardware, software, and machine learning communities is driven by changing hardware economics, larger deep learning architectures, and edge-device deployment demands.
- Domain-specific hardware creates efficiency gains for mainstream deep-neural-network research but can make deviating from established research paths more costly.
- An increasingly heterogeneous hardware landscape may make computing progress more uneven, while future AI breakthroughs may require a different algorithm, hardware, and software combination.
2 Separate Tribes
The essay traces how computing moved from necessary specialization to general-purpose hardware, encouraging hardware, software, and algorithm research to evolve separately. High development costs then led machine learning researchers to treat hardware as fixed, even though hardware-software choices continued influencing which ideas succeeded.
- Early computers were specialized because electronics were expensive and cross-purpose software was unavailable, not because customized hardware was considered intrinsically better.
- The general-purpose computer era encouraged hardware, software, and machine learning communities to evolve in isolation and made their choices appear independent.
- Moore’s law and Dennard scaling enabled a threefold increase in microprocessor performance between 1980 and 2010, favoring risk-averse general-purpose hardware design.
- Universal processors displaced experimentation with specialized research hardware because predictable general-purpose improvements reduced the appeal of uncertain specialized rewards.
- Developing a next-generation chip typically costs $30-80 million and takes 2-3 years, creating substantial barriers to hardware exploration.
- Machine learning researchers consequently treated hardware as a sunk cost to work around, although historical hardware lotteries still shaped which ideas succeeded or failed.
3 The Hardware Lottery
The essay argues that hardware and software compatibility can determine which research ideas succeed, delaying recognition of alternatives that lack suitable tools. Historical examples show how specialized hardware, computing constraints, and programming languages shaped the trajectory of AI research.
- Babbage’s analytical engine was not built partly because the precision needed to fabricate its parts was unavailable.The machine’s theoretical foundations were only supported by electromagnetic technology that emerged during World War II.
- Ideas associated with Babbage’s analytical engine were rediscovered roughly a century later, when available tools enabled empirical demonstration.The rediscovered concepts included stored programs, modifiable code, memory, and conditional branching.
- 3.1 The Lost Decades: Deep neural networks were delayed as a promising research direction because general-purpose CPUs were poorly suited to their matrix-heavy, highly parallel workloads.CPU training exhausted memory bandwidth, while the von Neumann bottleneck restricted data movement between CPUs and memory.
- 3.1 The Lost Decades: Specialized neural-network hardware efforts in the 1980s and 1990s remained fragmented, costly to iterate, or oriented toward languages and paradigms poorly suited to connectionist networks.Several projects were short-lived because custom silicon was expensive and lacked a sufficiently large consumer market.
- 3.1 The Lost Decades: Repurposed GPUs eventually made massive parallelism useful for deep neural-network training, and deeper networks produced large ImageNet performance gains.One comparison used 16,000 CPU cores for classification, while a later system used two CPU cores and four GPUs for the same task.
- 3.2 Software Lotteries: Symbolic AI benefited from fitting the prevailing software and hardware frameworks, including LISP and Prolog’s support for logic expressions.Connectionist researchers lacked a clearly suitable programming language until Matlab emerged in 1992.
4 The Persistence of the Hardware Lottery
Domain-specialized hardware improves efficiency for mainstream deep neural networks while making it costlier to pursue research directions that fall outside established hardware and software support. Limited data makes the counterfactual difficult to assess, but capsule networks illustrate this boundary.
- The end of Moore’s law is shifting computing from task-agnostic CPUs toward hardware tailored to costly deep-neural-network operations such as matrix multiplies.Examples include TPUs, edge-TPUs, Arm Cortex-M55, and Facebook’s Big Sur.
- Current hardware and compilation kernels do not support successful compression techniques such as unstructured pruning and weight-specific quantization.Hardware architects are developing designs and kernels intended to address these incompatibilities, with early sparsity-supporting designs already on the market.
- Hardware co-design largely targets older models with durable commercial use cases because hardware is economically viable only when those use cases last more than three years.Matrix multiplication is treated as a safe optimization target because it is established and commercially relevant.
- Evidence that specialized hardware constrains new research directions remains difficult to quantify because data are limited and counterfactual success under different hardware is hard to model.Capsule networks required novel operations such as squashing and routing by agreement, while hardware design prioritized commercial use cases over flexibility for new research ideas.
- Specialized hardware makes it more costly to train architectures such as capsule networks that depart from accepted convolutional-network building blocks.The authors frame this as a question about whether researchers implicitly overfit to ideas that operationalize well on available hardware.
5 The Likelyhood of Future Hardware Lotteries
The paper argues that future progress may depend on research directions unlike deep neural networks, while specialized hardware and declining returns to scale make commitment to current approaches risky. Biological intelligence provides examples of alternative computational organization, but the essay leaves the next breakthrough open.
- Betting heavily on specialized hardware is most defensible if future breakthroughs depend on pairing deep neural networks with increasing data and computation.The paper identifies this as an ongoing debate rather than a settled conclusion.
- For many subfields, returns from adding parameters are decreasing: Inception V3 to V4 nearly doubled parameters from 21.8 to 41.1 million while ImageNet accuracy rose from 78.8 to 80%.The estimated cost of training GPT-3 alone exceeded $12 million.
- Human brains combine over 85 billion neurons with energy use equivalent to an electric shaver, raising concerns about the cost of scaling deep neural networks toward comparable intelligence.The paper describes this gap as a reason to regard deep neural networks as a risky sole path forward.
- Biological intelligence differs from deep neural networks through decentralized local updates, efficient learning from fewer labelled examples, and task-specific pathways.Humans can walk and talk simultaneously more easily than they can read and talk, suggesting specialized pathways for different stimuli.
- Typical deep-learning models activate the entire model for every example, producing a quadratic blow-up in training cost, whereas the brain appears to infill expected inputs from memory.The passage contrasts full forward and backward passes with a process based on matching expected and incoming sensory data.
- Catastrophic forgetting occurs when new information interferes with previously learned behavior, causing performance on the original task to deteriorate.This is presented as another difference between current deep neural networks and biological intelligence.
- The paper presents alternative models of intelligence as evidence that the next breakthrough may require a different combination of hardware, software, and algorithm.It does not claim that deep neural networks are not the way forward, but identifies a possible present-day hardware lottery.
6 The Way Forward
Avoiding hardware lotteries requires cheaper, faster exploration of hardware–software–algorithm combinations, but hardware development remains costly and specialized. Software abstractions, profiling, and sustained public-private investment can improve portability and broaden experimentation, though flexibility introduces trade-offs.
- The Way Forward: Cheaper, less time-consuming exploration of hardware–software–algorithm combinations is necessary to avoid hardware lotteries.Scientific progress depends on alignment among hardware, software, and algorithms.
- The Way Forward: Reconfigurable hardware avoids lock-in to a single use case, but flexibility requires higher FLOPS and tailored software development.Even simple FPGA algorithms remain painful and time-consuming to code.
- The Way Forward: Funding remains the bottleneck for hardware serving use cases that are not immediately commercially viable, including neuromorphic, optical, analog, biological, and quantum approaches.The essay argues that sustained private and public funding is needed; new-material hardware may require 10-20 years of lead time.
- The Way Forward: Hardware fragmentation makes efficient code increasingly dependent on niche skills, producing uneven progress across hardware ecosystems and programming languages.Some ecosystems benefit from healthy software support, while others are stymied by too few critical end users.
- The Way Forward: Domain-specific languages and auto-tuning can improve portability and performance across heterogeneous hardware by abstracting implementation details.The trade-off is reduced expressive power for domain-specific languages and less direct human visibility into implementation.
- A Software Revolution: Software has substantial room for efficiency gains because predictable compute growth previously compensated for inefficiencies in the software stack.The paper describes these opportunities as low-hanging fruit for more efficient code.
7 Conclusion
The essay argues that algorithmic success is inseparable from the hardware and software on which algorithms run. As hardware becomes more heterogeneous, the hardware lottery persists, making it important to quantify the opportunity cost of relying on existing systems.
- Conclusion: Algorithm performance is fundamentally intertwined with the hardware and software on which it runs.The essay uses hardware lottery to describe how these downstream choices determine whether research ideas succeed or fail.
- Conclusion: The hardware lottery persists in an increasingly heterogeneous hardware landscape, and the gap between winning and losing research directions is expected to grow.The essay connects this widening gap to the need for better measurement of opportunity costs.