Automatic Code Documentation Generation Using GPT-3

• Manual documentation is costly and time-consuming to create and maintain, and can become obsolete as code changes.Source: §1 · p. 1
• Codex is evaluated for code documentation to address the lack of systematic evaluation of GPT-3 for this task.Source: §1 · pp. 1–2
• In a preliminary evaluation of code documentation on CodeSearchNet, one-shot Codex achieved state-of-the-art overall performance in documentation generation, outperforming all previous models with an average BLEU score of 20.63.By language, it outperformed all models in four of six languages (Python, Ruby, JavaScript, and GO) and ranked second in Java and PHP.Source: §1 · p. 2
• The study tests Codex on CodeSearchNet across six programming languages using zero-shot and one-shot learning.Source: §1 · p. 2

• The paper identifies template-based, information-retrieval, and learning-based approaches as three types of automatic code documentation work.Source: §2 · p. 2
• Template-based work uses predefined rules and layouts, including templates that summarize method statements or merge contextual information.Source: §2 · p. 2
• Information-retrieval approaches include LSI and VSM for class and method documentation, topic models, and code-clone-based retrieval.Source: §2 · p. 2
• Learning-based work spans neural summarizers and transformers, including CodeBERT, CoTexT, and PLBART.Other examples combine lexical and structural code information or retrieve relevant code and summaries to improve generated documentation.Source: Figure 1 · p. 2
• GPT-3 had not been systematically evaluated for code documentation, motivating this study’s evaluation of Codex against existing approaches.Source: §2 · p. 2

3.1 Dataset Collection and Preprocessing

• The study used CodeSearchNet pairs from six languages and removed comments and examples failing parsing, length, token, or English-language filters.Source: §3.1 · pp. 2–3

3.2 GPT-3 Model and Parameter Setup

• Codex, a GPT-3 descendant, was trained on natural language and billions of lines of public GitHub code.Source: §3.2 · p. 3
• One-shot prompting provided one randomly selected training code-documentation pair before asking Codex to document another code sample; zero-shot provided no example.Source: §3.2 · p. 3
• Temperature was set to 0.2, top-p to 1.0, repetition penalties to 0.0, and output capped at 256 tokens.Source: §3.2 · p. 3

3.3 Evaluation of Generated Documentation

• Evaluation sampled 1,000 test examples per language and used smoothed BLEU because short outputs may lack higher-order n-gram overlap.Source: §3.3 · p. 3

3.3.1 Performance Analysis.

• 20.63 average BLEU beats CoTexT's 18.55 by 11.21%; Java (22.81 vs. REDCODER's 22.95) and PHP (25.13 vs. CodeBERT's 25.16) are exceptions.Table 2 compares BLEU scores for documentation generation.Source: §3.3.1 · p. 3, Table 2 · p. 4

3.3.2 Qualitative Analysis.

• Generated documentation averaged 8 words and Flesch-Kincaid 5.97, versus actual documentation's 12 words and 6.77; average TF-IDF was 1.94 versus 2.28.The PHP example uses clearer “base-10 number” wording; the Ruby example adds a correct noop condition absent from the reference.Source: §3.3.2 · p. 4

4 CONCLUSION AND FUTURE WORK

• Codex showed state-of-the-art performance with basic settings, while previous approaches require task- or language-specific retraining or fine-tuning.Source: §4 · p. 4
• Evaluation covered 1,000 samples per language and zero- or one-shot learning only; exemplar choice and systematic parameter effects remain untested.Source: §4 · p. 4
• Future work targets few-shot learning, parameter tuning, fine-tuning when available, and fixing documentation issues such as doc smells.Source: §4 · p. 4