EleutherAI: Going Beyond "Open Science" to "Science in the Open"

1 Introduction

• EleutherAI says it is, to its knowledge, the only major research lab doing its work primarily in public.The paper situates this model amid the growth of independent AI research groups and non-traditional challengers.Source: §1 · p. 1
• The authors report that public-facing research has improved discussion, accelerated collaboration, and engaged participants from non-traditional backgrounds.They also credit this approach with fostering an active, motivated research community.Source: §1 · p. 1
• The paper presents EleutherAI’s experience, benefits, and pitfalls as a reason for other groups to consider greater openness and transparency.Source: §1 · p. 1

2.1 A Brief History Lesson2

• Founded in July 2020 to create a public GPT-3 replication, EleutherAI began with many members from outside ML and NLP academia.The founding participants included software engineers, ML hobbyists, and researchers from other fields.Source: §2.1 · p. 1
• EleutherAI released GPT-Neo 1.3B and 2.7B, GPT-J-6B, and GPT-NeoX-20B, each the largest publicly available decoder-only English language model at its release.GPT-NeoX-20B was also the largest publicly available English language model of any type at its release; The Pile [7] and LM-Eval-Harness became widely used resources.Source: §2.1 · p. 2
• The organization’s work later expanded to interpretability, multimodal models [6], large-model training infrastructure, and reinforcement learning from human feedback [10] [2].Source: §2.1 · p. 2

2.2 Organization and Collaboration

• Discord’s open discussion channels can be chaotic, but the authors say they also foster cross-pollination and awareness of other members’ work.Participation is voluntary, with no formal membership; most channels are open and accessible to participants.Source: §2.2 · p. 2

2.3 The Project Development Pipeline

• Projects begin as grassroots efforts, then may recruit volunteers, draw on shared compute, and use dedicated channels for technical discussion and project updates.Completed projects with significant EleutherAI participation are announced to the whole server.Source: §2.3 · p. 2

3 Public-Facing Research as a Mission

• EleutherAI’s public-facing mission makes discussion public by default and keeps barriers to spectating or contributing low.The paper focuses on openness in collaboration, rather than the open-access end products of projects.Source: §3 · p. 3

3.1 Public-Facing Research Enables Conversation

• Public access to planning and implementation details helps cultivate active technical discussion and attracts researchers who share suggestions and experience.The authors say this exchange proliferates technical discussion around similar challenges.Source: §3.1 · p. 3
• A lighthearted Discord culture makes researchers more comfortable sharing implementation details, tentative conjectures, and negative results, which the authors say are often lacking in traditional settings.Source: §3.1 · p. 3
• Because research coordination happens mostly on searchable Discord channels, people across the globe can participate asynchronously and newcomers can review the community’s history before contributing.Source: §3.1 · p. 3
• An online organization with participation from hobbyists and academic researchers around the world serves as a round-the-clock discussion and collaboration hub.Source: §3.1 · p. 3

3.2 Public-Facing Research Fosters Collaboration

• Open research collaboration can let contributors build on shared data, code, and results and provide feedback on ideas, code, and papers.The paper frames public discussion as a way to identify problems and potential solutions.Source: §3.2 · p. 3
• Low barriers help newcomers learn from and join ongoing projects before proposing their own, shortening the path from discussion to execution.The authors report that this speed encourages more serious discussion and that shared implementations can be iteratively improved; the VQGAN-CLIP [6] Colab drew community variants and tweaks.Source: §3.2 · pp. 3–4

3.3 Public-Facing Research Creates Community

• Public discussion supports a welcoming community where members can vary their involvement while still sharing knowledge and suggestions.The authors particularly emphasize aspiring researchers from non-traditional or non-academic backgrounds, who may lack a clear path into ML research.Source: §3.3 · p. 4

• Volunteer commitments can fade or arrive late, slowing projects; sustaining and rewarding high-quality contributions is crucial for continued success.Participation is unpaid, and initially enthusiastic members may reduce involvement or fail to deliver assigned work.Source: §4 · p. 4
• Public discussion creates opportunities for others to publish EleutherAI ideas or findings without attribution, sometimes despite steps to correct the record.The authors say citing tweets and documenting findings on their blog is not always sufficient, but accept this risk for the broader benefits.Source: §4 · p. 4
• Major research artifacts receive careful release consideration, but public-facing work prevents similar scrutiny of every discussion, leaving responsible sharing to community norms amid potential societal risks.Source: §4 · pp. 4–5

• The authors connect their commitment to open research in both product and process with a tradition of freely accessible scholarship, including the JMLR founding effort [6, 7].Forty researchers left the paid-access Machine Learning journal to support JMLR, citing the need for freely available research.Source: §5 · p. 5
• They contrast that tradition with an era in which cutting-edge research increasingly relies on proprietary models and datasets developed opaquely by industry labs.Source: §5 · p. 5
• EleutherAI attributes its success partly to public research, which supported collaboration across backgrounds and a vibrant researcher community; the authors say the approach may not fit every organization.They nevertheless urge the community to conduct more research and discussion openly.Source: §5 · p. 5

• EleutherAI says it is the only major research lab, to its knowledge, that does its work primarily in the public eye.Source: B Comparison with Other Open Research… · p. 7
• BigScience uses invite-only Slack, while LAION has private threads and a separate private Discord where substantial research occurs before public release.Source: B Comparison with Other Open Research… · p. 7
• The authors say EleutherAI places greater emphasis on technical AI alignment and existential risk than the compared groups, alongside alignment discussions, blog posts, and a reading group.Source: B Comparison with Other Open Research… · p. 7