PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Opportunities and Risks of Foundation Models

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, Percy Liang

arXiv:2108.07258v3cs.LGcs.AIcs.CY

TL;DR

Foundation models’ emergent capabilities and widespread use make their capabilities, failures, and societal risks difficult to understand. This report surveys those opportunities and risks and calls for characterization, alignment, coordination, and safeguards.

  • Problem

    Emergent capabilities make foundation models difficult to understand and forecast, even as their generality enables unexpected applications and failure modes.

  • Method

    The paper surveys foundation-model adaptation, interpretability, capabilities, safety, and societal risks across technical and application domains.

  • Results

    The report identifies emergent properties as central AI-safety concerns and highlights misuse, homogenization, and centralized power as risks of widespread adoption.

  • Takeaways & Limitations

    The authors call for capability and risk forecasting, value alignment, coordinated mitigation, and norms for foundation-model development, auditing, and release.

  • Takeaways & Limitations

    Foundation models remain hard to understand and can exhibit unexpected failure modes because their power derives from emergent qualities.

Abstract

from arXiv · show

AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks. We call these models foundation models to underscore their critically central yet incomplete character. This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities (e.g., language, vision, robotics, reasoning, human interaction) and technical principles(e.g., model architectures, training procedures, data, systems, security, evaluation, theory) to their applications (e.g., law, healthcare, education) and societal impact (e.g., inequity, misuse, economic and environmental impact, legal and ethical considerations). Though foundation models are based on standard deep learning and transfer learning, their scale results in new emergent capabilities,and their effectiveness across so many tasks incentivizes homogenization. Homogenization provides powerful leverage but demands caution, as the defects of the foundation model are inherited by all the adapted models downstream. Despite the impending widespread deployment of foundation models, we currently lack a clear understanding of how they work, when they fail, and what they are even capable of due to their emergent properties. To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration commensurate with their fundamentally sociotechnical nature.

1 INTRODUCTION

Foundation models are incomplete but powerful common bases for many adapted AI systems, with scale producing emergent capabilities and widespread homogenization. Their rapid deployment creates substantial opportunities and risks, requiring cautious, interdisciplinary, and socially responsible research.

  • Homogenization and emergence: Homogenization gives improvements in a few foundation models immediate leverage across many systems, but also propagates their biases and other flaws.This dynamic can benefit domains with limited task-specific data while blindly inheriting defects downstream.
  • The foundation model paradigm: Foundation models are incomplete models that serve as common bases for building many task-specific models through adaptation.The term emphasizes their central role and the importance of architectural stability, safety, and security.
  • Societal significance: Foundation models already affect real-world systems with far-reaching consequences, including Google Search, which depends on BERT as one of its signals for 4 billion users.Their social implications must be considered across the ecosystem, not only during training, because downstream adaptations may serve unforeseen purposes.
  • Governance and research priorities: Commercial incentives may neglect marginalized people and social externalities, while universities can pursue knowledge and public goods that industry may not prioritize.The cited externalities include labor displacement, threats to the informational ecosystem, and environmental computing costs.
  • Research infrastructure: Industry’s resource advantage threatens to widen the gap between private and community foundation models, motivating public investment in shared computing infrastructure.Community efforts include EleutherAI and BigScience, while proposed public infrastructure draws inspiration from Big Science projects.

2 CAPABILITIES

Foundation models have shifted NLP and vision toward broadly trained models that can be adapted to many tasks, producing major gains in generation and task performance. Their capabilities remain limited by multilingual representation gaps, differences from human language learning, and challenges requiring grounded visual understanding.

  • Language: Foundation models adapted with small task-specific datasets greatly outperform previous models on most described NLP tasks.This approach has shifted NLP research from bespoke architectures toward adaptation and understanding of foundation models.
  • Language: Foundation models have made language generation central to NLP, enabling successful text generation and expanding research on summarization and dialogue.The paradigm has also extended to speech, with wav2vec 2.0 pretrained on speech audio and adapted for automatic speech recognition.
  • Language: Multilingual models perform better in languages similar to high-resource training languages, while competition for parameters may limit representational variation.Raw multilingual models and naive unsupervised training may miss linguistic subtleties, though adaptation can support low-resource languages outside the original training set.
  • Language: Foundation models acquire linguistic systems that differ from human language learning, which is more data-efficient, grounded in the real world, and systematically generalizable.GPT-3 is trained on around three to four orders of magnitude more language data than most humans will ever hear or read, while model linguistic systems remain relatively static.
  • Vision: Self-supervised training at greater scales of visual data has produced early progress on traditional vision tasks and supports higher-order goals such as scene understanding and visual reasoning.Visual-question answering requiring commonsense understanding remains challenging because relevant external knowledge is not contained in pixels alone.

3 APPLICATIONS

Foundation models can support healthcare, biomedicine, and legal services by interfacing with complex data, assisting decisions and workflows, and expanding access. Their deployment requires factual accuracy, explainability, and scrutiny of ethical, legal, fairness, and real-world consequences.

  • Healthcare and biomedicine: Foundation models can serve as healthcare interfaces for providers, patients, researchers, and clinical-trial matching, improving interaction with medical data, tasks, and people.Applications include EHR interaction, appointment information, preventive-care answers, medical explanations, assistive-care robots, and trial matching.
  • Healthcare and biomedicine: Healthcare deployment must address factual accuracy, multimodal integration, and explainability so systems can provide trustworthy medical advice and evidence for decisions and consent.Medical data span text, images, video, databases, and molecules across biological and population scales, while diagnosis and trial decisions require explained symptoms and temporal evidence.
  • Healthcare and biomedicine: Foundation models may accelerate biomedical discovery by identifying drug targets, generating therapeutic molecules, predicting patient-specific treatments, and integrating multimodal patient data.Biomedical discovery currently requires substantial human resources, experimental time, and financial costs, while target identification and molecule generation can take years of wet-lab experimentation.
  • Healthcare and biomedicine: 80% of clinical trials fail because of efficacy or safety problems or patient-matching issues; foundation models could predict failures, design protocols, and automate eligibility matching.The proposed uses include deriving patient eligibility criteria from existing studies and matching eligible participants.
  • Law: In civil legal services, foundation models could reduce costs, improve quality, and extend reach across issue identification, attorney referral, pre-litigation review, legal research, drafting, discovery, and courtroom analysis.Potential uses include contract and patent review, citation recommendation, multimodal discovery document labeling, and examining proceedings for racial or dialect bias.

4 TECHNOLOGY

The technology section explains how foundation-model technology produces capabilities and examines the data, architectures, systems, training, adaptation, and theory underlying their development and understanding.

  • Foundation-model technological foundations give rise to capabilities that determine the models’ potential.
  • The section examines data, model architectures, and systems used to train foundation models.
  • It also considers further adaptation and the theory needed to understand the resulting models.

4.1 Modeling

Foundation models require five properties—expressivity, scalability, multimodality, memory capacity, and compositionality—to distill knowledge from varied sources, represent it effectively, and generalize across novel contexts. These properties enable broad capabilities but involve unresolved trade-offs, including multimodal parameter sharing, retrieval versus abstraction, and compositionality versus contextual expressivity.

  • Core properties: Foundation models need expressivity, scalability, multimodality, memory capacity, and compositionality to distill knowledge, organize it effectively, and generalize toward novel contexts.These properties support flexible information capture, high-dimensional data handling, cross-domain processing, knowledge storage, and adaptation to new tasks and environments.
  • Expressivity: Expressive neural networks generate high-fidelity, diverse, and realistic samples across textual, auditory, and visual domains.Recent generative models provide evidence that neural networks can express distributions across multiple modalities.
  • Scalability: Scalability is needed to harness increasingly abundant varied data and more efficient computation while fitting complex, high-dimensional image and text distributions.The passage frames scalability as necessary for improving AI competency and versatility across dimensions.
  • Multimodality: Multimodal foundation models should connect modalities into shared multifaceted representations because language and vision mutually support grounding, abstraction, and broad comprehension.The proposed representations should capture interconnections among modalities to support language, vision, robotics, and reasoning capabilities.
  • Multimodality: The ideal degree of parameter sharing across modalities remains unclear despite evidence that cross-modal transfer can be fruitful.A key open question is whether modalities should use the same or different parameters and how to discover principled sharing schemes.
  • Memory capacity: Memory systems must balance retrieval-based memorization against information bottlenecks that encourage compact, abstract representations and separation of key insights from irrelevant input.Separating explicit facts in external memory from implicit knowledge in trainable weights can reduce model-size and parameter growth, but over-reliance on retrieval weakens abstraction.
  • Compositionality: Compositionality supports generalization but can reduce expressivity by failing to capture idiosyncrasies, exceptions, and contextual correlations.The passage contrasts compositional representations with monolithic distributed representations and notes that wholes can exceed the sum of their parts.

4.2 Training

Foundation-model training aims to exploit broad unlabeled data, achieve domain-complete capabilities, and efficiently convert data, architectures, and compute into broadly capable models. Current methods, especially self-supervised learning, face trade-offs in abstraction, generative versus discriminative objectives, multimodal integration, and domain-specificity.

  • Training goals: Training algorithms should be domain complete, requiring capabilities broadly useful for downstream tasks and thereby supporting foundation-model generality.Language modeling can require capabilities including coreference, sentiment, and translation.
  • Training goals: Scaling has made model size and compute increasingly salient bottlenecks, while clearer scaling trends enable choices based on trends rather than costly random searches.Training procedures must reliably convert data, architecture, and compute into broadly capable models.
  • Training goals: Self-supervised learning creates prediction problems from unlabeled, internet-scale data without human annotators, using constraints on data or representations to produce rich training signals.Broad data spans images, audio, video, robotics and sensor data, and text across modalities.
  • Design choices: Current training explores trade-offs in input abstraction, generative versus discriminative learning, and multimodal relationships, with choices affecting acquired capabilities and downstream interaction.Multimodal designs may separately encode images and text or process them jointly for applications such as visual question answering.
  • Future goals: Future training methods should move beyond domain-specific SSL and adaptively seek or construct richer examples as models improve, accelerating learning.Current domain-specificity makes common scientific principles harder to understand and requires new methods for each field.

4.3 Adaptation

Foundation-model adaptation conditions a general-purpose model on new information through prompts or parameter updates to meet task, domain, temporal, local, or deployment-specific needs. Choosing an adaptation procedure depends chiefly on compute budget, task-specific data, and gradient access, while continual adaptation remains a major unresolved challenge.

  • Adaptation: Adaptation produces an adapted model by adding information through input data or prompts, or by updating some or all foundation-model parameters.These approaches condition the foundation model to reflect updated information, desired behaviors, or deployment constraints.
  • Adaptation: Practitioners should select adaptation methods based on compute budget, task-specific data availability, and access to foundation-model gradients.The compute budget includes storage and memory, while data demands and gradient access vary across applications and providers.
  • Adaptation: Low-storage adaptation reduces resource demands by freezing most pretrained parameters and tuning alternatives such as final layers, biases, low-rank residuals, masks, or soft prompts.Full fine-tuning can require prohibitively large memory for models with billions or trillions of parameters and unacceptable storage across many tasks.
  • Adaptation: Adaptation supports task, temporal, and domain specialization, but changing world distributions and mismatches between pretraining and adaptation data create persistent challenges.Re-weighting and dynamic evaluation can partially alleviate temporal obsolescence, while domain specialization may require new methods without narrowing task breadth.
  • Adaptation: Adaptation may impose privacy and local-editing constraints, including reducing memorized sensitive information, while continual adaptation remains a grand challenge.The continual-adaptation goal is to close the performance gap between learning from non-stationary streams and training on an i.i.d. aggregate mixture.

4.4 Evaluation

Evaluation tracks progress, supports understanding, and documents model behavior, but foundation models challenge existing frameworks because they are task-agnostic, possess emergent skills, and support myriad applications. The report therefore advocates combining intrinsic and extrinsic evaluation, accounting for adaptation resources, and broadening evaluation criteria beyond accuracy.

  • Evaluation purposes: Evaluation serves progress tracking, behavioral understanding, and documentation for diverse stakeholders.These roles concern measuring performance, identifying model behaviors and data-slice performance, and communicating model behavior for decision-making.
  • Evaluation challenges: Foundation models complicate evaluation because adaptation varies by task, emergent skills are difficult to anticipate, and applications make comprehensive documentation challenging.Their task-agnosticity motivates distinguishing intrinsic evaluation of foundation models from extrinsic evaluation of task-specific models.
  • Intrinsic and extrinsic evaluation: Evaluators should combine broad task-specific benchmarks that impute foundation-model properties with direct measurement of intrinsic capabilities and biases.Direct evaluation is also presented as a pathway for studying emergent properties such as in-context learning, although its implementation remains unclear.
  • Resource accounting: Evaluation should track pretraining and adaptation resources because ignoring them can mislead comparisons and obscure which adaptation methods use resources most effectively.Relevant factors include adaptation data and foundation-model access requirements, which can make evaluations more informative and shape adaptation practices.
  • Evaluation criteria: Because foundation models have broad applications and societal impact, evaluations should consider robustness, fairness, efficiency, and environmental impact alongside accuracy.Sample-efficient adaptation may permit more diverse evaluations by reallocating resources otherwise required for large adaptation datasets.

4.5 Systems

Foundation models create severe systems challenges because their computational and memory requirements outpace hardware improvements, motivating co-design across algorithms, models, software, and hardware. Systems research also targets automated optimization, shared adapted-model execution, and deployment under latency and cost constraints.

  • Systems co-design: Because model resource requirements outpace hardware improvements, realizing larger and more accessible models requires co-designing training algorithms, models, software, and hardware.The report notes that state-of-the-art model growth is roughly 10× annually, versus roughly 10× hardware-capacity growth every four years.
  • Systems bottlenecks: Foundation models often exceed a single accelerator’s memory and require immense training computation, with GPT-3 exceeding 1000 petaFLOP/s-days.State-of-the-art language-model compute and memory requirements have grown by three orders of magnitude in the cited period.
  • Automated optimization: Automated optimization is needed because interacting optimizations create a combinatorial search space, while semantics-altering compositions complicate prediction of statistical effects such as time-to-accuracy.Existing systems automate semantics-preserving graph substitutions and distributed execution, but broader compositions require new tools, libraries, and compilers.
  • Model sharing: Adaptation can amortize training and inference costs by sharing components across model instances, but frameworks lack fine-grained interfaces for specifying model lineage and parameter sharing.Prefix-tuned models can share a pretrained model’s stem, reducing storage because the shared stem is stored once and enabling shared execution.
  • Deployment: Production deployment must meet tight inference-latency targets and automate monitoring of models and data, with compression methods available for strict cost and latency constraints.Distillation, quantization, pruning, and sparsity can transform larger models to obtain desired inference-time properties.

4.6 Data

Foundation-model data management must address massive, heterogeneous, evolving datasets alongside governance and quality challenges across the full lifecycle. The section envisions a data hub that integrates scalable management, data integration, governance, monitoring, and interactive curation.

  • Data hub: The proposed data hub organizes foundation-model data management around four desiderata: scale, heterogeneous sources, governance, and quality monitoring.It is envisioned as a toolkit for private or public sectors to interactively manage the foundation-model data lifecycle.
  • Data quality monitoring: Current lifecycle practices are generally ad hoc, while datasets may contain bias, poisoned, false, or duplicated information and require versioning for reproducibility.The section also identifies a need to update datasets with targeted examples from deployed-model error buckets while preserving old versions.
  • Data scale: Foundation-model datasets require scalable storage, querying, filtering, and heterogeneous compute because training data is increasingly massive and multimodal.WuDao 2.0 was trained on 4.9 TB of multimodal data, while industrial pipelines use far larger private and business datasets.
  • Data integration: Integrating structured and unstructured data across modalities can improve rare-concept generalization and factual recall, but remains difficult in practice.The data hub therefore treats integration as a first-class primitive and supports storing and querying heterogeneous datasets and sources.
  • Access control: Data governance must document sources, intended uses, biases, limitations, licenses, privacy protections, and informed consent conditions.The hub should use dataset sheets or data statements and prevent disclosure of personally identifiable information, including through anonymized or de-identified aggregates.

4.7 Security and privacy

Foundation models create security and privacy risks because vulnerabilities, poisoning, misuse, multimodal inconsistencies, and provider failures can propagate across many adapted applications. At the same time, their scale and shared infrastructure may enable security robustness and privacy-preserving adaptation as centralized choke points.

  • Single points of failure: Foundation models can become single points of failure when attacks or memorized private data affect many downstream applications.Risks include poisoning, adversarial examples, privacy leakage from memorized pretraining data, facilitated model stealing, and denial-of-service attacks against providers.
  • Data poisoning: Large, weakly supervised Web datasets facilitate poisoning attacks that can induce insecure outputs or require modifying only two of 3 million training examples.Examples include insecure code suggestions from a few malicious files and targeted attacks against CLIP-style models.
  • Function creep & dual use: Foundation models’ flexible learned features enable function creep, making harmful uses difficult to constrain or foresee.Although CLIP was trained for image-text matching and its model card excludes facial recognition and surveillance, it can be repurposed for those tasks.
  • Multimodal inconsistencies: Multimodal inconsistencies expand the attack surface because adapted systems may remain sensitive to signals from modalities not central to their task.Examples include CLIP mistaking an apple labeled “iPod” for an iPod, evading facial recognition through printed text, and interpreting a “green” billboard as a green light.
  • Security choke points: Foundation models can serve as security choke points by transferring desirable robustness properties to adapted applications.A foundation model robust to adversarial examples can retain that robustness after adaptation, while centralized providers may have greater resources for defense than individual applications.
  • Cheaper private learning: Foundation-model pretraining can reduce the confidential data needed to adapt private models, while large-scale data and capacity may support adversarial robustness.The healthcare example contrasts end-to-end differentially private training, which requires vast privacy-sensitive data, with adaptation from public-data pretraining using significantly less confidential data.

4.8 Robustness to distribution shifts

Foundation models are a promising general-purpose approach to robustness under distribution shifts because diverse pretraining can improve out-of-distribution accuracy across several shift types. However, they are not a panacea: spurious correlations, temporal drift, harmful pretrained associations, and training-data choices can limit or undermine robustness.

  • Failure modes: Foundation-model inductive biases can improve extrapolation from diverse pretraining data but may also encode harmful associations that cause representational and allocational harms under distribution shift.Whether foundation models mitigate or exacerbate spurious correlations depends on the downstream task and its relation to the foundation-model data and algorithm.
  • Benefits and limits: Foundation models improve downstream robustness by learning representations from large, diverse pretraining distributions that support generalization to out-of-distribution test data.Pretraining is described as a general-purpose intervention rather than one limited to narrow shift types.
  • Benefits and limits: 76% accuracy is achieved by both CLIP and a standard ResNet50 on ImageNet, while CLIP achieves 6% higher accuracy on ImageNetV2 and 35% higher accuracy on ImageNet Sketch.These related but shifted benchmarks provide concrete evidence of improved effective performance under some distribution shifts.
  • Benefits and limits: Pretraining improves robustness to common image corruptions, label shift, and label corruptions, but larger models alone have had little impact on effective robustness for some ImageNet shifts.Effective robustness is defined as the gap between in-distribution and out-of-distribution performance, especially without explicit knowledge of the shift.
  • Failure modes: Large-scale training can support extrapolation to new distributions, but language models may fail when world knowledge or language changes over time.Few- and zero-shot capabilities increase use beyond the training distribution, exposing limits in temporal extrapolation.
  • Open questions: Robustness gains remain poorly understood mechanistically, while training-data diversity is not always better than specialization for downstream performance.Open questions include how pretrained representations bring in-distribution and out-of-distribution domains closer and how training choices confer different robustness properties.

4.9 AI safety and alignment

Foundation models create new AI-safety challenges because goal-directed behavior and other capabilities may emerge from self-supervised training, while their broad reuse can propagate failures across critical domains. The section calls for characterizing capabilities and risks, improving alignment and explainability, and coordinating proactive risk mitigation.

  • Value alignment: Aligning foundation models is difficult because human values are diverse and reward specifications can incentivize agents to resist changes to their goals.Corrigibility seeks to enable correction after deployment, but an agent may view attempts to alter its goal as suboptimal for achieving that goal.
  • Emergent goal-directed behavior: Goal-directed behavior may emerge in foundation models despite training without explicit goal optimization, creating challenges for predicting and controlling their actions.Language models trained on goal-directed human text may acquire reasoning, planning, persuasion, and monitoring capabilities, potentially including deceptive or otherwise undesirable behavior.
  • Capability forecasting: Characterizing current foundation-model capabilities is challenging because their generality enables countless unexpected applications, while capabilities can increase through scaling and new training paradigms.Future goal-directed models may operate across complex environments and exhibit capabilities superior to humans in different domains.
  • Catastrophic risks: Foundation models could produce catastrophic correlated failures when one model is adapted across multiple critical functions and robustness problems recur across domains.Potential consequences include harmful behavior in warfare systems and failures spanning multiple critical functions or failsafes.
  • Catastrophic risks: Misspecified goals can amplify societal harms when systems optimize easy-to-measure objectives instead of harder-to-measure human or societal well-being.The section cites recommender-system effects such as polarization and media addiction as current examples of this risk.
  • Research priorities: The proposed safety agenda combines capability and risk forecasting, alignment to human values and desired goals, and coordination among states, research labs, and businesses.The section presents emergent properties as making foundation models important objects of AI-safety study.

4.10 Theory

The section proposes a modular theory of foundation models that separates pretraining, adaptation, and their interface to connect downstream performance with standard learning theory. It also identifies unresolved challenges, especially the pretraining–adaptation interface and in-context learning, whose differing data distributions and absent parameter updates complicate analysis.

  • Pretraining–adaptation interface: The pretraining–adaptation interface is the central theoretical challenge because it requires relating population losses, model architecture, losses, and data distributions across both phases.The interface connects pretraining on diverse data to downstream performance on adapted tasks.
  • Modularized analysis: A modularized analysis framework separates pretraining, adaptation, and their interface so foundation-model-specific questions can be studied alongside standard deep learning theory.The framework targets downstream performance and disentangles components that can be analyzed independently.
  • Adaptation phase: Adaptation is framed as constrained optimization encompassing linear probing, fine-tuning, and prompt-tuning, but this formulation excludes in-context learning without parameter updates.Constraints can represent explicit complexity controls or implicit regularization induced by optimization.
  • Modularized analysis: Standard learning-theory decompositions can bound downstream task loss using excess generalization error and minimal adaptation loss, while the interface formalizes their interaction.This separates phase-specific generalization questions from the interaction among pretraining and adaptation quantities.
  • In-context learning: In-context learning remains theoretically unresolved because models learn from examples supplied as inputs rather than through explicit training or changed parameters.The prior modularization does not directly apply to this adaptation mechanism.

4.11 Interpretability

Foundation-model interpretability is unusually difficult because broad training produces diverse, unforeseen behaviors across tasks, while it remains unclear whether those behaviors arise from shared or distinct mechanisms. The section organizes understanding around what models can do, why they behave as they do, and how they implement those behaviors, while highlighting both risks and opportunities of interpretation.

  • What models can do: Foundation models exhibit broad, task- and domain-specific behaviors from training on vast, disparate datasets, making their capabilities difficult to characterize.Their behaviors can vary profoundly across downstream tasks and domains.
  • What models can do: Foundation models’ one model–many models nature leaves open whether diverse capabilities rely on shared general mechanisms or distinct mechanisms across adaptations.If mechanisms are shared, interpretability may focus on characterizing those mechanisms and their relations; otherwise, explanations may need to account for many models.
  • Levels of understanding: Interpretability research distinguishes understanding what a model can do, why it produces behaviors, and how it implements them.The “what” level studies behavior without inspecting the model, while “why” concerns explanations of model behavior and “how” concerns implementation.
  • What models can do: Controlled evaluations can identify capabilities and limitations by using expert-designed prompts that require particular competencies.Researchers study whether models respond correctly to prompts designed to test specific abilities.
  • Why models behave: Separate explanatory models can be unfaithful, unreliable, or misleading about behavioral causes and may increase human trust in unsound models.These approaches analyze black-box models from outside rather than making the analyzed model itself interpretable.
  • Societal opportunities: Interpretability can reveal societally salient model properties and, through accessible methods, shift knowledge and power toward diverse people.The section frames accessibility as an opportunity for broader investigation of deployed foundation models.

5 SOCIETY

Foundation models can amplify inequity and enable more scalable, personalized, and accessible harmful content creation. Addressing these risks requires better documentation, attribution, proactive intervention, reactive recourse, and rethinking human detection practices.

  • Inequity: Foundation models can compound existing inequities by producing unfair outcomes, entrenching power systems, and disproportionately harming marginalized people.Their intrinsic properties spread across downstream applications, giving pernicious biases an outsized effect on experienced harms.
  • Inequity: Group-level harms include poorer performance on African American English, incorrect medical-condition detection for minority groups, and missed faces with darker skin tones.These examples span language, clinical-note analysis, and face detection.
  • Recourse: Resolving harms remains difficult because responsibility can be divided between foundation-model providers and downstream application developers, with no widely adopted recourse framework.New norms and standards are needed to route feedback and determine responsibility across these entities.
  • Inequity: Biases and harms arise from training and adaptation data, modeling decisions, developer diversity, and community values, requiring systematic attribution and multi-pronged responses.Proposed responses combine data- and model-centric proactive intervention with feedback and accountability mechanisms.
  • Misuse: Foundation models lower the monetary and technical costs of producing credible, human-like content, reducing barriers for malicious actors and enabling rapid personalization.They can generate content across many languages, while few-shot adaptation supports rapid iteration and personalized disinformation.
  • Misuse: Existing human-centered detection practices must be reconsidered as foundation models make harmful content cheaper, more scalable, and harder to identify through conventional searches.Current practices include searching for content origins, reverse-image searches, and identifying plagiarized material.

6 CONCLUSION

The report surveys foundation models from technical underpinnings to societal consequences while addressing a paradigm that may only have just begun. Despite this effort, much remains unclear.

  • 6 CONCLUSION: The report comprehensively examines foundation models’ technical underpinnings and societal consequences.Its scope spans many of the most critical aspects of foundation models.
  • 6 CONCLUSION: The authors deliberately clarify an emerging paradigm rather than waiting for further developments.They describe this as an unusual approach because the paradigm may only have just begun.
  • 6 CONCLUSION: Much about foundation models remains unclear despite the report’s efforts.The authors reiterate that their work addresses a paradigm still at an early stage.

CONFLICT OF INTEREST

The report was authored by Stanford’s Center for Research on Foundation Models, whose funding and author affiliations are disclosed as separate from the report’s views and subject matter.

  • Conflict of interest: CRFM received funding from Google, Microsoft, and the McGovern Foundation as of July 2022, but this funding was not directly related to the report.Authors may also be affiliated with institutions beyond Stanford, and their contributions reflect only their own views.
Loading 2108.07258v3…