PaperlayerView exact revision on arXiv

Source-linked AI summary

AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation

Vijayaraghavan Murali, Chandra Maddila, Imad Ahmad, Michael Bolin, Daniel Cheng, Negar Ghorbani, Renuka Fernandez, Nachiappan Nagappan, Peter C. Rigby

arXiv:2305.12050v2cs.SEcs.AI

TL;DR

At Meta, developers must keep up with internal knowledge spread across a large, changing codebase. CodeCompose adapts InCoder for internal, multilingual code authoring and is evaluated offline and in production. It reproduced masked lines 40%–58% of the time, improved 1.4× and 4.1× over the Public model, and supplied 8% of typed code.

  • Problem

    At Meta, developers must navigate knowledge embedded in a large, evolving codebase, including internal libraries and practices that may be known by only a small group.

    Source: §1 · p. 2

  • Method

    The authors built CodeCompose on InCoder, fine-tuning it for Meta’s internal code and supporting multilingual, bidirectional suggestions across coding surfaces.

    Source: §1 · p. 2, §2.1 · p. 5

  • Results

    LCM CodeCompose exactly recreated masked lines 40%–58% of the time, an improvement of 1.4× and 4.1× over the Public model; 8% of typed code came from suggestions.

    Source: §1 · p. 3, §8 · p. 18

  • Takeaways & Limitations

    Developer feedback highlighted API discovery and documentation assistance, and CodeCompose was rolled out to all developers.

    Source: §8 · p. 18

    Because CodeCompose was deployed only at Meta, its findings and the value of fine-tuning on internal code may not generalize to other organizations.

    Source: §5.1 · p. 14

Abstract

from arXiv · show

Generative LLMs have been shown to effectively power AI-based code authoring tools that can suggest entire statements or blocks of code during code authoring. In this paper we present CodeCompose, an AI-assisted code authoring tool developed and deployed at Meta internally. CodeCompose is based on the InCoder LLM that merges generative capabilities with bi-directionality. We have scaled up CodeCompose to serve tens of thousands of developers at Meta, across 9 programming languages and several coding surfaces. We present our experience in making design decisions about the model and system architecture for CodeCompose that addresses these challenges. To release a LLM model at this scale, we needed to first ensure that it is sufficiently accurate. In a random sample of 20K source code files, depending on the language, we are able to reproduce hidden lines between 40% and 58% of the time, an improvement of 1.4x and 4.1x over a model trained only on public data. We gradually rolled CodeCompose out to developers. At the time of this writing, 16K developers have used it with 8% of their code coming directly from CodeCompose. To triangulate our numerical findings, we conduct a thematic analysis on the feedback from 70 developers. We find that 91.5% of the feedback is positive, with the most common themes being discovering APIs, dealing with boilerplate code, and accelerating coding. Meta continues to integrate this feedback into CodeCompose.

1 INTRODUCTION

CodeCompose applies LLM-based code authoring to Meta’s large, changing internal codebase, where developers must keep up with evolving libraries and internal knowledge. The paper describes the tool, its development and deployment, and evidence from offline evaluation, usage, and developer feedback.

1 INTRODUCTION

  • The paper frames the challenge as helping developers navigate knowledge embedded in an evolving internal codebase and concentrated among a small set of developers.Meta’s codebase spans billions of lines and 9+ programming languages.Source: §1 · p. 2
  • CodeCompose combines an InCoder-based model fine-tuned on Meta’s internal code with multilingual support, natural-language comment understanding, and suggestions using code beyond the cursor.It was integrated with Meta’s VSCode and other editors.Source: §1 · p. 2
  • LCM CodeCompose exactly recreated masked lines 40%–58% of the time, an improvement of 1.4× and 4.1× over the Public model, depending on language.The evaluation question asks how well the model generates one hidden line from existing code snippets.Source: §1 · pp. 2–3
  • CodeCompose made over 4.5 million suggestions to 16K engineers, with 8% of changed code written by CodeCompose and a 22% acceptance rate.The system was rolled out to 100% of Meta engineers.Source: §1 · p. 3
  • Only 8.5% of feedback comments were negative, and the leading themes were API discovery and boilerplate (23%) and coding acceleration (20%).The thematic analysis drew on feedback from 70 engineers and informed iterative improvement.Source: §1 · p. 4

2 MODEL DEVELOPMENT AND EVALUATION METHODOLOGY

The paper develops an InCoder-based model and evaluation approach for code suggestions that use both preceding and following code. Its methodology adapts the training objective to production editing, fine-tunes on internal data, and evaluates against a held-out code sample.

2.1 Model Architecture and Training Objective

  • Language Causal Masking (LCM) adapts InCoder’s Causal Masking for editing by predicting a masked span with code context on both sides.LCM masks at production trigger characters, adds metadata, uses one mask, and allocates input length between code before and after the cursor at a 70–30 split.Source: §2.1 · p. 5

2.2 Training data

  • The authors fine-tuned InCoder-1.3B on first-party Meta code, using recent diffs and filters for freshness and production status.The resulting training set comprised tens of millions of files and a few billion lines across 10+ languages.Source: §2.2 · pp. 5–6

2.3 Model Evaluation Method and Measures

  • Evaluation used an unseen, filepath-hash-separated sample of 20K files across Hack, C++, Python, and Flow, masking code spans with before-and-after context.The authors compared public InCoder with fine-tuned CM and LCM models using Exact Match and BLEU.Source: §2.3 · pp. 6–7

3 SYSTEM DESIGN

CodeCompose uses a client-server architecture with a shared language server to connect editors to model inference. The section also describes latency-conscious serving, telemetry, and a phased mixed-methods production evaluation.

3 SYSTEM DESIGN

  • The system separates editor clients from a GPU inference tier, with a shared LSP mediating requests and returning inline suggestions.Requests include code around the cursor and file metadata; the server prioritizes latency and chooses beam search or greedy generation based on input length.Source: §3 · p. 7, §3.1 · pp. 7–8

3.2 Language Server Protocol

  • The Rust LSP orchestrates inference, inline completion display, telemetry, debouncing, and caching to reduce duplicate requests and response time.It supports the inline-completions request type for CodeCompose.Source: §3.2 · p. 8

3.3 Clients

  • Clients connect editors to the shared LSP; owning the stack makes it straightforward to integrate CodeCompose into internal tools such as web-based Jupyter notebooks.Editors without native LSP support use an adapter.Source: §3.3 · p. 8

3.4 Telemetry

  • The LSP records most client-side metrics from document-change events and uses a client notification to measure full user-perceived round-trip time.This limits the integration demands placed on clients.Source: §3.4 · p. 8

3.5 Evaluation Methodology for CodeCompose in Production

  • Production evaluation combined usage telemetry with thematic analysis, rolling out by language and in 25% increments before reaching all developers.The deployment tracked suggestions and acceptance; 70 engineers’ feedback was manually coded and grouped into themes.Source: §3.5 · pp. 8–9

4 RESULTS

Fine-tuned LCM reproduces hidden lines in 40%–58% of cases and improves over the Public model by 1.4×–4.1×, depending on language. At deployment, CodeCompose made over 4.5 million suggestions to 16,000 engineers, with a 22% acceptance rate and 8% of changed code written by the tool.

4.1 RQ1. Model Evaluation Results

  • The evaluation masks a random line and supplies surrounding code; Public exact matches range from 14% for Hack to 23% for C++, and BLEU from 28% for Hack to 40% for C++.Source: §4.1 · p. 9
  • LCM always outperforms CM; versus Public, Hack improves 4.1 times on Exact Match and 2.6 times on BLEU, while C++ improves 1.7× and 1.4×, respectively.Source: §4.1 · p. 9
  • LCM exactly reproduces hidden lines 40%–58% of the time and achieves BLEU scores of 56%–73%, improving over the Public model by 1.4×–4.1×, depending on language.The comparison is reported across programming languages in Table 1.Source: §4.1 · p. 10, Table 1 · p. 10

4.2 RQ2. Adoption Results

  • Across nine programming languages, CodeCompose made 4.5 million suggestions to 16,000 developers and had a 22% acceptance rate for suggestions displayed at least 750 milliseconds.Source: §4.2 · p. 11
  • CodeCompose accounted for 8% of changed code and was rolled out to 100% of Meta engineers.The 8% figure was calculated from accepted suggestion lengths and characters typed by developers.Source: §4.2 · p. 11

How do developers perceive CodeCompose in their daily work?

Feedback was mostly favorable, with users citing API discovery, boilerplate automation, and faster coding. Negative feedback centered on unusual APIs and interaction with traditional autocomplete.

4.3.1 Evidence of Positive Experiences.

  • 23% of users valued API discovery and boilerplate automation, while 20% said CodeCompose accelerated coding.Developers also reported producing more in-code and API documentation; the strongest fit was typical patterns, standard libraries, and heavily templatized work.Source: How do developers perceive CodeCompose… · p. 11, §4.3.1 · p. 12

How do developers perceive CodeCompose in their daily work?

4.3.2 Evidence of Negative Experiences.

  • Negative feedback flagged hallucinations on uncommon APIs and URLs and disruptive competition with traditional autocomplete, including overloaded Tab shortcuts.CodeCompose also struggled with specialized APIs and atypical patterns; the authors identify coexistence UX and grounding as ongoing concerns.Source: §4.3.2 · pp. 13–14

5 THREATS TO VALIDITY

The threats concern whether findings transfer beyond Meta, how well the chosen measures capture usefulness, and possible biases in deployment metrics and feedback.

5.1 Generalizability

  • Because CodeCompose was deployed only at Meta, its results—and the value of internal-code fine-tuning—may not generalize to other organizations.The authors caution that external AI coding assistants may perform differently, while noting evidence for internal-data fine-tuning at Google.Source: §5.1 · p. 14

5.2 Construct Validity

  • The study selected BLEU and exact match, while triangulating adoption and acceptance measures with thematic analysis of feedback from 70 developers.The authors characterize exact match as conservative because it succeeds only when code is perfectly generated.Source: §5.2 · p. 14

5.3 Internal Validity

  • Internal validity may be affected by external events, small language cohorts, reluctance to share negative feedback, and subjective manual categorization.Two authors reconciled thematic labels, but a different team or more systematic method might have categorized feedback differently.Source: §5.3 · pp. 14–15

6 RELATED WORK

The related work highlights limited industrial-scale evidence on code-generation accuracy and developer experience. CodeCompose adds results from Meta’s production setting across nine languages and compares its adoption and feedback with prior tools.

6 RELATED WORK

  • CodeCompose’s evaluation used 20,000 data points from real Meta codebases across nine programming languages, unlike common public benchmarks focused on Python tasks.MBPP contains 974 tasks and HumanEval 164 problems.Source: §6 · p. 15
  • Limited public research evaluates AI-assisted code generation in industrial environments with thousands of daily users and many programming languages.The paper also identifies a shortage of large-scale industrial accuracy evaluations and mixed-method studies of developers.Source: §6 · p. 15
  • 22% of 4.5 million single-line completions were accepted, and CodeCompose suggestions accounted for 8% of Meta engineers’ code versus Google’s reported 3%.Google reported a 6% reduction in coding iteration time and a 25% acceptance rate for single-line completions.Source: §6 · p. 15
  • Copilot studies report varied outcomes: developers switched between acceleration and discovery use cases [10], while Java suggestions scored 57% correctness and JavaScript 27%.The paper notes that LeetCode problems differ substantially from normal engineering work.Source: §6 · p. 15
  • CodeCompose achieved 52% correctness and 73% BLEU for Hack, versus 40% correctness and 56.2% BLEU for C++ on Meta’s production-code evaluation.The paper contrasts these results with prior industrial and open-source evaluations, including Exact Match 20.10% and BLEU 32.79% on CodeSearchNet [27].Source: §6 · p. 16
  • 91.5% of feedback from 70 Meta engineers was favorable, with API discovery and coding acceleration among common themes also reported in Copilot research [10].Other reasons included boilerplate, documentation, suggestion accuracy, and predicting what developers intended to type.Source: §6 · p. 16

7 DISCUSSION AND LESSONS LEARNED

The discussion frames trust, interaction design, and evaluation as practical challenges for AI coding assistants. CodeCompose’s lessons emphasize curated training data, restrained suggestions, latency control, and metrics that go beyond acceptance alone.

7.1 Trust

  • Code-generation LLMs may suggest nonexistent APIs or code that does not compile, making developer trust a central challenge.Training-data validity, bias, security, vulnerabilities, and data drift can also affect trust.Source: §7.1 · p. 16
  • Trust involves balancing recall against precision: noisy incorrect suggestions can erode confidence, while sparse suggestions can reduce engagement and weaken the investment case.Developers may accept reworking suggestions as useful starting points, but extensive rework can also reduce trust [1].Source: §7.1 · p. 16
  • CodeCompose filtered obsolete and non-production patterns and trained only on code modified within the previous two years.Developer feedback also led the team to incorporate code after the cursor, the file being edited, or the notebook kernel into training objectives.Source: §7.1 · p. 17

7.2 User Experience

  • Offering one line at a time struck a balance between adequate code and avoiding a disruptive block that forces developers into on-the-fly review.CodeCompose generally withholds suggestions when code remains to the cursor’s right, except for certain closing tokens.Source: §7.2 · p. 17
  • Developers accepted 300–500 ms suggestion latency, but not beyond 1 s; acceptance fell as end-to-end latency increased.Caching and debouncing helped reduce latency, while repeated Tab presses let developers request and edit successive lines.Source: §7.2 · p. 17

7.3 Metrics

  • Acceptance rate measures usage, not necessarily productivity, and strict metrics may miss benefits when developers rework suggestions.Evaluations should also surface inaccurate suggestions and user-experience pain points, although privacy and compliance can limit telemetry.Source: §7.3 · pp. 17–18
  • CodeCompose tracked active users, engagement, retention, suggestions offered, acceptance rate, and the share of developer-typed code originating from suggestions.These deployment measures complement assessment of product impact.Source: §7.3 · p. 18

8 CONCLUSION

The conclusion presents CodeCompose as an internally deployed code-authoring system, describing its architecture, model, and evaluation. It reports substantial usage and favorable developer feedback, while outlining possible future extensions.

8 CONCLUSION

  • CodeCompose combines an inference service, Language Server Protocol, and client to generate tokens and coordinate how suggestions appear in editors.The service performs inference; the LSP and client orchestrate requests and display suggestions.Source: §8 · p. 18
  • The InCoder-based model uses Language Causal Masking, and fine-tuning on Meta’s internal code improved offline results by 1.4× to 4.1×.The authors also report 4.5 million suggestions shown and 8% of user-typed code coming from suggestions.Source: §8 · p. 18
  • 91.5% of developer feedback was favorable; reported uses included discovering unfamiliar APIs and writing documentation.CodeCompose had been rolled out to all developers, with less than 1% disabling it.Source: §8 · p. 18
  • Planned directions include richer cross-file context, inference-time feedback, conversational IDE interaction, code explanations, and uses beyond code authoring.The authors also describe exploring semantic processing to reduce hallucinations.Source: §8 · p. 18
Loading 2305.12050v2…