PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Mitigating Library-Related Issues in LLM-Generated Code

Yacine Majdoub, Rinad Hamid, Eya Ben Charrada, Ahmad Abdellatif, Haifa Touati

arXiv:2610.00622v1cs.SE

TL;DR

Rapidly evolving libraries can make LLM-generated code rely on stale or incorrect imports and APIs. The paper proposes a documentation-grounded agentic workflow and reports fewer library-related errors and higher correctness across evaluated models.

  • Problem

    Because libraries evolve while LLMs learn from historical code, their generated usage can become outdated and fail in fast-evolving frameworks.

    Source: §I · p. 1

  • Method

    The approach combines dependency analysis, documentation grounding, code generation, and automated validation in a library-aware generation workflow.

    Source: §X · p. 13

  • Results

    38.1%–54.6% reduction in library-related errors, with correctness rates increasing by up to 16% across evaluated models.

    Source: §I · p. 2, §X · p. 13

  • Takeaways & Limitations

    Relative improvements were consistent across five models, although differences in model capabilities may affect absolute performance.

    Source: §I · p. 2, B. External Validity · p. 12

    Different models’ capabilities, context windows, and instruction-following accuracy may affect absolute performance; evaluation covered five models.

    Source: B. External Validity · p. 12, §VIII · p. 12

Abstract

from arXiv · show

Software practitioners increasingly rely on Large Language Models (LLMs) to generate code that integrates external libraries. However, LLMs often produce incorrect library usage, such as invalid imports, outdated API calls, and hallucinated dependencies, leading to compilation or runtime failures that reduce the reliability of AI-assisted software development. In this paper, we propose an agentic approach to mitigate libraryrelated errors in LLM-generated code. More specifically, we first conduct an exploratory study to characterize the library-related issues produced by LLMs. Our analysis of 100 LLM-generated code files reveals that 84% of generated files contain at least one library-related error, with recurring patterns including incorrect import paths, missing imports, hallucinated libraries, deprecated library usage, and unused imports. Based on these findings, we design an agentic approach that integrates task analysis, documentation grounding, code generation, and automated validation to improve library usage during code synthesis. We evaluate our approach on 300 code generation tasks derived from realworld implementations of rapidly evolving Python frameworks, including LangChain and AutoGen, across five LLMs: GPT-5, DeepSeek-V3, Qwen3, Mistral, and Llama 3. The results show that our approach consistently improves code generation quality across all evaluated models, reducing library-related errors by 38.1% - 54.6% and increasing code correctness by up to 16%.

I. INTRODUCTION

LLMs can produce plausible but incorrect library usage as fast-evolving APIs outpace historical model knowledge. The paper characterizes these issues and proposes documentation-grounded generation with automated validation.

I. INTRODUCTION

  • Fast-evolving libraries can outpace LLMs’ historical knowledge, yielding incorrect imports, outdated APIs, or nonexistent modules.The paper names LangChain and AutoGen as frameworks where frequent API changes can make generated code fail.Source: §I · p. 1
  • Prior work found 4.6% of libraries in generated code unresolved automatically; a single incorrect dependency can prevent execution and require manual compatibility debugging.The paper links these failures to reduced productivity gains from LLM-based development.Source: §I · p. 1
  • In an analysis of 100 generated files, 84% contained library-related errors, including incorrect paths, missing or hallucinated imports, deprecated usage, and unused imports.Faulty files averaged 2.7 errors.Source: §I · p. 1
  • The proposed agentic pipeline analyzes dependencies, retrieves relevant documentation, generates documentation-grounded code, and validates library usage against current APIs.A Validator agent detects and corrects inconsistencies using feedback from compilation and dependency management.Source: §I · pp. 1–2
  • Across 300 code-generation tasks and five LLMs, versus direct prompting, the system consistently improved code correctness, reducing library-related errors by up to 54.6% and increasing correctness rates by up to 16%.Source: §I · p. 2

II. MOTIVATING EXAMPLE

  • In the LangChain RAG example, three generated imports were incompatible with the current library version, although the imported symbols themselves were valid abstractions.The errors arose from changed module paths, such as PyPDFLoader’s move to langchain_community.document_loaders, and prevented execution.Source: §II · p. 2

III. EXPLORATORY STUDY

The exploratory study characterizes library-related errors in generated code and uses realistic repository-derived tasks to examine their prevalence. Its dataset combines manually reviewed prompts with generated solutions and reference implementations.

III. EXPLORATORY STUDY

  • The study systematically identifies and measures recurring library-related import issues, especially those associated with fast-evolving frameworks.Its objective is to characterize failure patterns and their prevalence in generated programs.Source: §III · pp. 2–3

A. Dataset Creation

  • The dataset was built from real-world framework use: about 400 repositories were filtered, then 100 repositories with concrete target-framework API usage were selected manually.Repositories were public Python projects; forks and projects failing predefined criteria were excluded.Source: A. Dataset Creation · p. 3
  • Repository functionality was converted into natural-language prompts, then two authors reviewed and refined each prompt for clarity, completeness, and consistency with its reference code.ChatGPT was used to draft prompts from the reference implementations.Source: A. Dataset Creation · p. 3
  • LLMs generated solutions for the 100 prompts, with original repository implementations serving as references across tasks such as ingestion, multi-agent coordination, RAG, and API integration.The curated dataset consists of 100 prompt–code pairs.Source: A. Dataset Creation · p. 3

B. Annotation Process

The annotation process developed a shared error taxonomy, applied it through independent checks against prompts and documentation, and resolved disagreements through review meetings.

B. Annotation Process

  • The annotation process identifies, categorizes, and quantifies issues in import statements and library usage.It was designed to support systematic analysis of library-related errors.Source: B. Annotation Process · p. 3

1) Preliminary Thematic Analysis:

  • Before formal annotation, two annotators jointly reviewed 25 samples, refined error categories and guidelines, and reached consensus on ambiguous cases such as valid imports using deprecated APIs.The preliminary thematic analysis established shared criteria for subsequent annotation.Source: 1) Preliminary Thematic Analysis · p. 3

2) Independent Review of Generated Code:

  • Two annotators independently reviewed the remaining files against prompts and official documentation, checking package existence, import paths, classes, and usage.They also compared imports with reference code and examined missing or unused imports.Source: 2) Independent Review of Generated Code · p. 3

3) Taxonomy Construction and Classification:

  • Annotators iteratively grouped and refined observed failure mechanisms into a standardized taxonomy for consistent classification and quantitative analysis.The taxonomy distinguishes high-level categories such as structural path errors and semantic hallucinations.Source: 3) Taxonomy Construction and… · pp. 3–4

4) Inter-Annotator Agreement Assessment:

  • After each batch, annotators met to resolve disagreements by consensus and assessed file-level agreement as full, partial, or none.These levels distinguished identical findings, limited differences, and substantially different findings.Source: 4) Inter-Annotator Agreement Assessment · p. 4

C. Study Results

The exploratory study finds frequent, clustered import-related errors in generated code, with five recurring categories. The proposed approach then structures library-aware code generation around task analysis, documentation retrieval, generation, and validation.

C. Study Results

  • 84 of 100 files contained import-related errors, with 228 instances across those files—about 2.7 errors per failing file.The analysis excluded seven prompt–code pairs where no code was generated.Source: C. Study Results · p. 4
  • Incorrect paths (40.8%) and missing imports (34.6%) were most common, followed by hallucinated (12.3%), deprecated (7.5%), and unused imports (4.8%).The categories are not mutually exclusive, so one file may contain multiple error types.Source: C. Study Results · pp. 4–5
  • The taxonomy captures structural and semantic errors that the study associates with rapidly evolving libraries and version-inconsistent code generation.Annotators reached Cohen’s κ = 0.86; initial disagreements were reconciled jointly.Source: C. Study Results · pp. 4–5

IV. APPROACH FOR SOLVING LIBRARY-RELATED ISSUES

  • The proposed approach organizes library-aware code generation as task analysis and documentation retrieval followed by code generation, automated validation, and correction.Figure 2 presents an overview; the workflow uses Analyzer, Code Generator, and Validator agents with shared documentation-retrieval and compilation modules.Source: §IV · p. 5, Figure 2 · p. 5

A. Analyzer

Analyzer turns task intent into structured dependencies and filtered, relevant API documentation; Code Generator is constrained to use that guidance before forwarding its implementation for validation.

1) Dependency Analysis.:

  • Analyzer identifies required third-party libraries, frameworks, and submodules while excluding Python standard-library modules, then records their purpose in structured JSON.The dependency specification is designed for processing by later components.Source: 1) Dependency Analysis · p. 5

2) Documentation Processing.:

  • Analyzer filters documentation to retain verified imports, API descriptions, and relevant usage examples while removing duplicate or unrelated content.The filtered material is aligned to the identified dependencies and intended task.Source: 2) Documentation Processing · p. 6

B. Documentation Retrieval

  • Documentation Retrieval uses trusted web sources to collect official API pages, extracting verified imports and concise executable snippets rather than passing raw pages downstream.The module parses documentation code blocks and usage sections, then returns imports and snippets to Analyzer.Source: B. Documentation Retrieval · p. 6
  • Analyzer applies task-specific filtering to the retrieved bundle, removing outdated, redundant, or incompatible API references before passing it to Code Generator.This final refinement aligns retained documentation with inferred task intent.Source: B. Documentation Retrieval · p. 6

C. Code Generator

  • Code Generator follows the documentation-derived imports, module paths, and usage patterns to synthesize a complete implementation, which is then checked for syntax, executability, API conformance, and task alignment.The generated implementation remains provisional until validation.Source: C. Code Generator · p. 6

D. Compilation and Dependency Management

Compilation and dependency checks surface environment and syntax failures, while Validator compares generated code with documentation and task requirements and returns targeted corrections. The evaluation setup compares the full system with direct prompting and tests component ablations.

D. Compilation and Dependency Management

  • The module compiles candidate code without executing its logic; on failure, it sends error logs to Validator to guide correction.After successful compilation, it checks imports and package availability, installs missing dependencies when available on PyPI, and compiles again.Source: D. Compilation and Dependency Management · p. 6

E. Validator

  • Validator checks dependency resolution, imports, API calls, and semantic alignment with the task, then sends structured, actionable feedback for regenerating affected code.The feedback identifies the issue category and code location, describes the mismatch, and gives correction instructions.Source: E. Validator · p. 7
  • Validation and correction repeat until criteria are met or the configured iteration limit is reached, allowing a trade-off between refinement and computational cost.Validator rechecks updated code against the same criteria at each iteration.Source: E. Validator · p. 7

V. EXPERIMENTAL SETUP

  • RQ1 compares the system with direct prompting on 300 tasks using correctness, library-error counts, and error-reduction rates across five models; RQ2 removes four components in separate ablations.The ablations assess Analyzer, Documentation Retrieval, Compilation and Dependency Management, and Validator under the same conditions.Source: §V · p. 7

A. Evaluation Dataset

The evaluation uses tasks derived from recent implementations in rapidly evolving Python frameworks, with prompts that omit implementation-specific details. It compares direct prompting and the proposed approach across five models using correctness and library-error measures.

A. Evaluation Dataset

  • The dataset contains 300 tasks derived from implementations in LangChain, AutoGen, CrewAI, LlamaIndex, and Agno, selected for adoption and frequent evolution.Source repositories were created or actively maintained after June 2025 and represented realistic LLM application scenarios.Source: A. Evaluation Dataset · p. 7
  • Tasks were written from implementation functionality and manually reviewed by two annotators, while omitting framework-specific names, import paths, and low-level details.The source implementations were references for task construction, not inputs given to evaluated models.Source: A. Evaluation Dataset · p. 7

B. Evaluation Metrics

  • Code correctness rate measures the percentage of generated files that compile and satisfy library-usage requirements.The metric uses the number of correct files and total evaluated files.Source: B. Evaluation Metrics · p. 8
  • Error reduction rate measures the relative decrease in library-related errors versus direct prompting, and the evaluation also records median refinement iterations and analyzes error categories.The baseline and system error counts are the quantities compared for the reduction measure.Source: B. Evaluation Metrics · p. 8

C. Evaluation Settings

  • The system is evaluated on GPT-5, DeepSeek-V3, Qwen3, Mistral, and Llama 3 against direct prompting using the same tasks, prompts, and model configurations.The selected models include proprietary and open-weight families.Source: C. Evaluation Settings · p. 8

VI. RESULTS

Across five models, the system improves correctness and reduces library-related errors compared with direct prompting. Error reductions span categories, and the iterative feedback loop reaches final solutions within a limited number of correction cycles.

1) Impact on library-related errors:

  • Correctness rose 4%, 12%, 15%, 14%, and 16% for GPT-5, DeepSeek-V3, Qwen3, Mistral, and Llama 3, respectively, versus direct prompting.Table II compares the system and direct prompting across five LLMs on 300 tasks.Source: 1) Impact on library-related errors · p. 8, Table II · p. 9
  • Library-related errors fell 38.1% for GPT-5, 41.8% for DeepSeek-V3, 42.5% for Qwen3, 54.6% for Mistral, and 48.1% for Llama 3.The reported reductions vary with models’ direct-prompting performance; GPT-5 also benefits despite strong baseline results.Source: 1) Impact on library-related errors · p. 8

2) Iterative Refinement:

  • Median refinement ranged from 1.68 iterations for GPT-5 to 2.53 for Llama 3, and stronger initial code generation was associated with fewer correction cycles.The authors characterize the required iteration counts as limited and the refinement overhead as not excessive.Source: 2) Iterative Refinement · p. 8

3) Effects across library-related error categories:

  • The system consistently reduces all library-error categories, with large reductions in hallucinated libraries and deprecated usage; hallucinated-library errors fell from 31 to 13 for GPT-5 and 48 to 21 for DeepSeek-V3.The reported reductions for those examples are 60.7% and 58.1%, respectively.Source: 3) Effects across library-related… · p. 9
  • Deprecated-usage errors fell from 102 to 40 for Mistral (60.8%) and from 91 to 48 for Llama 3 (47.3%), while incorrect, unused, and missing imports also declined across models.The figure’s heatmap covers reduction rates across error categories and evaluated models, with strong reductions highlighted for hallucinated libraries and deprecated usage.Source: 3) Effects across library-related… · p. 9, Figure 3 · p. 9
  • The results indicate broad improvements across models and error types, addressing failures beyond isolated imports through documentation retrieval and post-generation validation.The conclusion includes incorrect imports, hallucinated libraries, and deprecated APIs among the targeted failure modes.Source: 3) Effects across library-related… · p. 9

B. Contribution of System Components

The ablation study uses DeepSeek-V3 to compare the complete system with variants that each remove one optional component. The results show that the components contribute complementarily, with documentation retrieval having the largest impact on library usage correctness.

B. Contribution of System Components

  • 83% correctness, 114 library-related errors, and 51 affected files are the complete system’s best ablation results.Each variant removes one component while keeping the others enabled.Source: B. Contribution of System Components · p. 10, Table III · p. 10
  • Removing Documentation Retrieval lowers correctness to 75% and raises library-related errors from 114 to 177, with 76 affected files.This is the largest reported component impact on library usage correctness.Source: B. Contribution of System Components · p. 10
  • Removing the Validator lowers correctness from 83% to 78%, while errors rise to 149 and affected files to 65.The authors attribute its contribution to post-generation verification and correction.Source: B. Contribution of System Components · p. 10
  • Without the Analyzer, correctness falls to 76%, with 164 library-related errors and 71 affected files.The Analyzer identifies relevant library usage patterns before generation.Source: B. Contribution of System Components · p. 10
  • Disabling Compilation yields 81% correctness, 132 errors, and 56 affected files; compilation feedback can still detect environment-specific issues.The correctness decrease is 2%, and compilation can catch missing dependencies or unresolved imports.Source: B. Contribution of System Components · p. 10

VII. DISCUSSION

The discussion interprets the approach as complementing pretrained model knowledge with current library information and verification. It also considers remaining failures, component contributions, and consistency across evaluated models.

A. Mitigation of Library-Related Errors

  • Documentation retrieval and validation reduce library-related errors across evaluated models, indicating that the errors are partly linked to missing connections to current library information.The approach supplements pretrained knowledge with retrieved documentation and checks against library constraints, but does not eliminate failures when documentation is insufficient, libraries interact complexly, or reasoning exceeds available documentation.Source: A. Mitigation of Library-Related Errors · p. 11

B. Contribution of Individual Components

  • Removing individual components increases errors, while the ablation results attribute distinct roles to dependency identification, current API information, and verification.The components work together; no individual component completely solves library-related failures.Source: B. Contribution of Individual Components · p. 11
  • The component findings extend prior work on retrieval-augmented and agent-based systems to library-dependent code, where correctness also depends on compatibility with evolving APIs [28].The authors describe the components as complementary rather than relying on a single module.Source: B. Contribution of Individual Components · p. 11

C. Generalization Across Language Models

  • Improvements are reported across GPT-5, DeepSeek-V3, Qwen3, Mistral, and Llama 3, although their magnitudes vary with models’ pretrained knowledge and library-reasoning abilities.The discussion attributes consistent gains to documentation grounding and validation complementing pretrained knowledge.Source: C. Generalization Across Language Models · p. 11

D. Implications

The implications discuss documentation-aware retrieval and automated verification for development workflows, dependency identification for research, and threats to validity in the evaluation and exploratory study.

1) Implications for Developers:

  • Direct prompting produced common library-related errors, especially incorrect import paths; the authors recommend documentation-aware retrieval in AI-assisted development workflows [28].They also argue that retrieval should be combined with automated verification and refinement, including compilation and dependency validation.Source: 1) Implications for Developers · pp. 11–12

2) Implications for Researchers:

  • The Analyzer’s structured dependency specifications support relevant documentation retrieval while avoiding unnecessary context, motivating intelligent context selection over simply expanding context windows [32].The authors identify automatic selection of task-relevant libraries, APIs, and documentation snippets as a research direction.Source: 2) Implications for Researchers · p. 12
  • Some programs still failed after the maximum refinement iterations despite error reductions of up to 54.6%, so iterative validation alone did not recover every dependency-related error.Proposed future directions include richer compiler feedback, adaptive retrieval, multi-agent collaboration, and fine-tuning with debugging and repair traces.Source: 2) Implications for Researchers · p. 12

A. Internal Validity

  • The evaluation prompts were LLM-generated from GitHub implementations, creating possible formulation or difficulty bias despite author review and refinement.The authors say the review aimed to preserve developer intent and targeted library challenges while keeping benchmark instances consistent.Source: A. Internal Validity · p. 12
  • Manual error categorization may involve judgment bias in borderline cases, though two annotators resolved disagreements and achieved Cohen’s κ = 0.86.They independently examined the programs using predefined guidelines.Source: A. Internal Validity · p. 12

B. External Validity

The paper situates its approach among research on LLM code failures and library-aware generation, then summarizes results and their scope. Its external-validity discussion cautions that the findings may not transfer directly beyond the selected fast-evolving Python frameworks.

B. External Validity

  • The evaluation used fast-evolving Python frameworks; the findings may not directly generalize to stable libraries or ecosystems in Java, JavaScript, or C++.The authors plan broader evaluation across frameworks and languages.Source: B. External Validity · p. 12

A. Errors in LLM-generated code

  • Prior work reports hallucinated APIs, deprecated methods, and incorrect imports, while many code evaluations emphasize functional correctness and overlook library- and import-related failures [10], [24],, [38].The cited studies also describe nonexistent packages and misuse through incorrect aliases, missing submodules, or incompatible API combinations [37].Source: A. Errors in LLM-generated code · p. 13

B. Library-aware and API-grounded code generation

X. CONCLUSION

  • Across five LLMs and 300 tasks from rapidly evolving Python frameworks, the approach reduced library-related errors by 38.1%–54.6% and increased correctness by up to 16%.The exploratory study found at least one library-related error in 84% of 100 programs.Source: §X · p. 13
  • Future work includes extending library-aware reasoning to API migration, dependency management, and repository-level development, and adding software knowledge sources and execution feedback.Source: §X · p. 13

Related papers

Loading 2610.00622v1…