PaperlayerView exact revision on arXiv

Source-linked AI summary

Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators

Andreas Liesenfeld, Alianda Lopez, Mark Dingemanse

arXiv:2307.05532v1cs.CL

TL;DR

Proprietary instruction-tuned generators raise concerns about reliability, scrutiny, and accountability, while the extent of openness in alternatives is unclear. The paper surveys these alternatives across openness dimensions and finds uneven sharing, especially for instruction-tuning data and scientific documentation.

  • Problem

    Proprietary instruction-tuned generators raise concerns about reliability, reproducibility, and scrutiny, while users cannot assess their constraints and risks from interfaces alone.

    Source: §1 · p. 2, §1.1 · p. 2

  • Method

    The paper surveys open-source instruction-tuned generators, assessing projects across 13 features in availability, documentation, and access methods.

    Source: §1.1 · p. 2, §2 · p. 3, §2.2 · p. 3

  • Results

    Most surveyed projects offer source code and some pre-training data, but reinforcement-learning data is supplied by at best half, and only a handful have scientific write-ups; none had undergone peer review.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Openness can support reproducible workflows and accountability, but does not resolve the harms of thoughtless deployment or questionable copyright implications.

    Source: §5 · p. 4

    Openness alone will not mitigate harms from thoughtless LLM deployment or questionable copyright implications of scraping publicly available data.

    Source: §5 · p. 4

Abstract

from arXiv · show

Large language models that exhibit instruction-following behaviour represent one of the biggest recent upheavals in conversational interfaces, a trend in large part fuelled by the release of OpenAI's ChatGPT, a proprietary large language model for text generation fine-tuned through reinforcement learning from human feedback (LLM+RLHF). We review the risks of relying on proprietary software and survey the first crop of open-source projects of comparable architecture and functionality. The main contribution of this paper is to show that openness is differentiated, and to offer scientific documentation of degrees of openness in this fast-moving field. We evaluate projects in terms of openness of code, training data, model weights, RLHF data, licensing, scientific documentation, and access methods. We find that while there is a fast-growing list of projects billing themselves as 'open source', many inherit undocumented data of dubious legality, few share the all-important instruction-tuning (a key site where human annotation labour is involved), and careful scientific documentation is exceedingly rare. Degrees of openness are relevant to fairness and accountability at all points, from data collection and curation to model architecture, and from training and fine-tuning to release and deployment.

1 INTRODUCTION

The introduction argues that proprietary instruction-tuned text generators pose risks to reproducibility, scrutiny, and accountability, while openness supports cumulative research. The paper surveys ChatGPT alternatives and prior work to track degrees of openness and transparency.

1 INTRODUCTION

  • Open research underpins cumulative scientific progress, as illustrated by AlexNet’s human-annotated images and AlphaFold’s use of openly deposited protein structures.The introduction also notes that decades of prior protein-folding work had reached 17%.Source: §1 · p. 2
  • ChatGPT’s proprietary convenience carries lock-in and reliability risks: ending Codex API support with three days’ notice undercut the reproducibility of at least 100 research papers.Source: §1 · p. 2
  • Proprietary systems can obscure ethical oversight, labour practices, privacy, and copyright, while concentrated LLM-training compute contributes to AI de-democratisation [9].Source: §1 · p. 2

1.1 Why openness matters

  • Openness supports transparency, reproducibility, and quality control, and enables critical scrutiny of datasets and computational literacy [7].Source: §1.1 · p. 2
  • Because instruction-following systems use human-feedback-based reinforcement learning and users cannot assess their constraints and risks through interfaces alone, the paper argues for open alternatives [12] [43] [59] [55].Source: §1.1 · p. 2
  • The survey maps openness dimensions, reviews the state of open-source instruction-tuned generators, and aims to support tracking transparency and accountability.Source: §1.1 · p. 2

1.2 Previous work

  • Prior work includes informal resource lists and rare structured model surveys, while data statements and model cards document curation and help users judge assumptions and risks [38].A prior survey of 400 papers found that about a third shared test datasets, 8% shared code, and only one shared all three datasets with results.Source: §1.2 · p. 2, §1.1 · p. 2
  • Open alternatives include BLOOM, The Pile, and LLaMA, but the initiatives had not undergone rigorous peer review or data auditing, and openness does not settle legal concerns [56] [28] [39] [16] [46] [52].Source: §1.2 · p. 3

2 METHOD

The method surveys open-source instruction-tuned text generators, focusing on enduring transparency and accountability dimensions. It selects eligible LLM+RLHF projects and rates 13 features across availability, documentation, and access methods.

2 METHOD

  • The authors focus on enduring transparency and accountability dimensions in a fast-changing field and assess scientific documentation and access methods as well as openness.Source: §2 · p. 3

2.1 Requirements

  • Eligible projects use an LLM with reinforcement learning from human feedback and aim for openness; proprietary, undocumented systems and API wrappers of ChatGPT are excluded.Source: §2.1 · p. 3
  • The survey includes small or early-stage projects when they are open, sufficiently documented, and released under an open-source license, and identified at least 15 projects launched in six months.Source: §2.1 · p. 3

2.2 Survey elements

  • The survey assesses projects on 13 features across availability, documentation, and access methods, rating each feature from maximum to partial to no openness.Table 1 overviews the 13 features.Source: §2.2 · p. 3, Table 1 · p. 3

3 RESULTS

The reviewed projects range from small code-focused tools to better-documented alternatives, but openness is often limited by inherited data, unavailable RLHF datasets, and sparse peer review. Synthetic instruction data and unclear licensing add further complications.

3 RESULTS

  • Larger initiatives include xmtf, based on BLOOMZ and mT0, and OpenAssistant, which offers open crowd-sourced RLHF data and text and graphical interfaces; some projects are wrappers of closed models.The smaller projects often offer source code and can serve as entry points for learning LLM+RLHF tools, despite lacking performance.Source: §3 · p. 3
  • Many tools inherit undocumented, often web-scraped datasets from their base models, sometimes with dubious legality.Some small projects cannot disclose architecture, training data, and documentation because they build on closed-source data.Source: §3 · p. 3
  • RLHF training data is often not shared, limiting smaller teams’ performance and hampering reproducible research on instruction-tuned conversational interfaces.Building these datasets requires labour-intensive work by human annotators.Source: §3 · p. 3
  • None of the reviewed systems is documented in a peer-reviewed paper, leaving them without that quality-assurance mechanism and risking weaker scientific and ethical scrutiny.Most follow a corporate “release by blog post” model.Source: §3 · pp. 3–4
  • Synthetic instruction data, including Self-Instruct and Baize, makes reinforcement learning partly synthetic rather than directly based on human feedback; its consequences at scale remain unknown.Baize is generated through ChatGPT interactions seeded by human-generated questions scraped from online knowledge bases.Source: §3 · p. 4
  • Baize’s research-only license conflicts with commercial-use claims by Falcon models that incorporate Baize models and data, illustrating legal dependencies obscured by claims of openness.The paper presents this as one example of complex dependencies and legal quagmires.Source: §3 · p. 4

4 DISCUSSION

The survey finds varying degrees of implementation and openness, with RLHF data and scientific documentation especially scarce. It argues that openness can support reproducible workflows and reduce reliance on corporate control, while enabling scrutiny of technology’s consequences.

4 DISCUSSION

  • Most surveyed projects provide source code and some pre-training data, but RLHF data is available from at best half; only a handful have scientific write-ups, and none has undergone peer review.The RLHF step is described as crucial to simulating instruction-following in these interfaces.Source: §4 · p. 4
  • All surveyed projects are significantly more open than ChatGPT, though the paper describes openness as having many shades.ChatGPT lacks technical documentation, while the GPT-4 report is characterized as providing minimal technical detail.Source: §4 · p. 4
  • The authors argue that data provenance and harm documentation deserve scholarly and societal credit, and point to interactive model cards as a promising documentation approach.They connect this proposal to incentive structures that favor moving fast over careful scientific work.Source: §4 · p. 4
  • The paper links openness to ethical scholarship and transparent, reproducible research, while warning that corporate capture and user lock-in can give companies control over research infrastructure.It notes that conversational formats can repackage extracted human labour.Source: §4 · p. 4
  • The authors frame counterfoil research as a way to clarify how people relate to their tools and consider consequences including exploited labeling workers, energy demands, and plausible-looking text without regard for truth.They present publicly funded science as able to make room for reflection rather than automatically following technological trends.Source: §4 · p. 4
  • The surveyed systems may support reproducible workflows, resource-cost accounting, and less reliance on corporate whims; openness may also prompt calls for responsibly limited technology.The paper presents this as a possible outcome, not a guaranteed effect.Source: §4 · p. 4

5 CONCLUSION

Openness is not a complete solution to the scientific and ethical challenges of conversational text generators, but it makes original research and accountability checks possible.

  • Openness does not mitigate harmful deployment consequences or questionable copyright implications of scraping publicly available data.Source: §5 · p. 4
  • Openness makes original research possible, including reproducible workflows and work to understand LLM + RLHF architectures.Source: §5 · p. 4
  • Openness enables checks and balances that foster accountability for data and curation, models, and deployment.Source: §5 · p. 4
Loading 2307.05532v1…