PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepCodeSeek: Real-Time API Retrieval for Context-Aware Code Generation

Esakkivel Esakkiraja, Denis Akhiyarov, Aditya Shanmugham, Chitra Ganapathy

arXiv:2509.25716v1cs.SEcs.AIcs.IR

TL;DR

ServiceNow code completion needs relevant Script Includes retrieved from partial code, where intent may be implicit and generic search and models lack domain-specific knowledge. DeepCodeSeek combines metadata-based filtering, enriched indexing, code expansion, and reranking, with post-training for a compact reranker. The authors report 87.86% top-40 retrieval accuracy and a 0.6B reranker that outperforms an 8B model on top-5 accuracy while maintaining reduced latency.

  • Problem

    Retrieving relevant ServiceNow Script Includes from partial code is difficult because explicit queries are absent, intent is nuanced, and general-purpose search and LLMs lack platform-specific knowledge.

    Source: §1 · p. 1

  • Method

    DeepCodeSeek combines platform-metadata filtering, enriched indexing, LLM code expansion, reranking, and compact-reranker post-training with synthetic data, supervised fine-tuning, and reinforcement learning.

    Source: §1 · p. 1, §3 · pp. 2–3

  • Results

    87.86% top-40 accuracy, more than doubling BM25, while the optimized 0.6B reranker reaches 68.58% top-5 accuracy versus 66.10% for 8B with 2.5x reduced latency.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The pipeline supplies API context for code-completion scenarios while addressing the challenge of inferring developer intent from partial code.

    Source: §8 · p. 8

    The evaluation focuses on Script Includes, synthetic data may not capture real-world complexity, and the 204-sample synthetic dataset constrains training experiments.

    Source: §8.1 · p. 8

Abstract

from arXiv · show

Current search techniques are limited to standard RAG query-document applications. In this paper, we propose a novel technique to expand the code and index for predicting the required APIs, directly enabling high-quality, end-to-end code generation for auto-completion and agentic AI applications. We address the problem of API leaks in current code-to-code benchmark datasets by introducing a new dataset built from real-world ServiceNow Script Includes that capture the challenge of unclear API usage intent in the code. Our evaluation metrics show that this method achieves 87.86% top-40 retrieval accuracy, allowing the critical context with APIs needed for successful downstream code generation. To enable real-time predictions, we develop a comprehensive post-training pipeline that optimizes a compact 0.6B reranker through synthetic dataset generation, supervised fine-tuning, and reinforcement learning. This approach enables our compact reranker to outperform a much larger 8B model while maintaining 2.5x reduced latency, effectively addressing the nuances of enterprise-specific code without the computational overhead of larger models.

1. Introduction

The paper addresses retrieval of relevant ServiceNow Script Includes from partial code for LLM-powered completion and Build Agent tasks. It proposes a multi-stage retrieval and reranker-training approach, then reports its contributions and outlines the paper’s later sections.

1. Introduction

  • ServiceNow code completion requires retrieving relevant Script Includes from partial developer code, where explicit queries are absent and developer intent can be nuanced.Traditional keyword and basic vector search miss nuanced intent and hierarchical platform relationships; general-purpose LLMs also lack domain-specific knowledge.Source: §1 · p. 1
  • DeepCodeSeek combines platform metadata and advanced IR techniques with post-training of compact rerankers using synthetic data, supervised fine-tuning, and reinforcement learning.The authors report that their optimized 0.6B reranker surpasses 8B models with reduced latency.Source: §1 · p. 1
  • The paper next details its retrieval method and dataset and indexing, followed by experimental results, ablations, and reranker post-training.Source: §1 · pp. 1–2

2. Related Work

The paper situates its approach in neural code retrieval, RAG, search refinement, and reranking, adapting these techniques to an enterprise setting. Its related work motivates domain-aware retrieval and metadata-based filtering for ServiceNow Script Includes.

2. Related Work

  • The paper adapts advances in neural code retrieval, RAG, structural code analysis, and search refinement to a large-scale enterprise environment.Source: §2 · p. 2

2.1. Neural Code Retrieval

  • General-purpose dense retrievers can degrade out of domain, motivating BM25 as a domain-agnostic baseline and a ServiceNow-specific retrieval pipeline.The discussion traces code search from BM25 to CodeBERT and cites CoIR [4] on domain-shift degradation.Source: §2.1 · p. 2

2.2. RAG with Filtering and Query Enhancement

  • RAG supplies context to coding assistants, while code structure and hypothetical-code query enhancement inspired by [8] can improve context relevance.Unlike source-code knowledge graphs [10], this pipeline builds its graph from ServiceNow metadata for scope-level filtering.Source: §2.2 · p. 2

2.3. Reranking

  • Cross-encoder or long-context LLM rerankers [11] reorder candidates, and this paper applies RL fine-tuning to enterprise-specific Script Include pairs.The discussion also describes list-wise reasoning in REARANK [12] and code-task RL in SWE-RL [13].Source: §2.3 · p. 2

3. Proposed Method

The proposed method is a multi-stage retrieval pipeline for supplying relevant Script Includes to code completion. It combines metadata-informed search, enriched indexing, code expansion, and reranking.

3. Proposed Method

  • The multi-stage retrieval pipeline is designed to provide highly relevant Script Includes for code completion.Figure 1 presents an overview of the pipeline.Source: §3 · p. 2, Figure 1 · p. 3
  • Platform-metadata Knowledge Graph filtering narrows the candidate search space before the main retrieval stage.Source: §3 · p. 2
  • The pipeline enriches its index with grouped Script Include methods, metadata, and JSDoc; LLM code expansion clarifies partial-code queries, and reranking prioritizes relevant candidates.Post-training uses synthetic data, supervised fine-tuning, and reinforcement learning to optimize compact rerankers.Source: §3 · p. 3
  • The combined metadata-informed search, enriched indexing, query generation, and reranking pipeline is reported to outperform vanilla retrieval methods for code generation tasks.Source: §3 · p. 3

4. Dataset and Index Construction

The authors build a real-world ServiceNow completion dataset and index, and use platform metadata to construct a Knowledge Graph for search-space pruning. They find JSDoc summaries outperform raw code for retrieval.

4.1. Dataset Construction

  • The dataset contains 850 real-world ServiceNow completion scenarios, with code_before and code_after providing context to retrieve the Script Include needed for code_middle.The prefix omits the target Script Include, while the suffix adds lines without exposing the API; code_after availability determines FIM or non-FIM input.Source: §4.1 · pp. 3–4, Figure 2 · p. 4
  • An LLM judge classified 705 of 850 samples (83%) as clear-intent cases for primary evaluation; the remaining 145 ambiguous cases were excluded from main results and analyzed separately.Source: §4.1 · p. 4

4.2. Index Construction

  • JSDoc summaries provide superior retrieval performance to raw code, so the final optimized index uses JSDoc summaries.The index covers 277 namespaces and 3,337 APIs; scripts range from 66 to 10,407 tokens, while JSDoc summaries range from 157 to 5,368 tokens.Source: §4.2 · p. 4

4.3. Knowledge Graph Construction

  • The hierarchical Knowledge Graph captures relationships among packages, scopes, and Script Includes to filter candidates by contextual relevance before expensive retrieval.Source: §4.3 · p. 4

5. Experimental Setup

The evaluation uses Top-K accuracy and MRR to assess retrieval, compares against BM25, and checks how often models produce the correct Script Include without retrieval context.

5.1. Evaluation Metrics

  • Top-K accuracy measures whether the correct Script Include appears within the first K results, while MRR summarizes its reciprocal rank across queries.Reported K values are 5, 10, 20, and 40.Source: §5.1 · p. 4

5.2. Baselines and Implementation

  • BM25 is the primary baseline at 53.02% top-40 accuracy; the implementation also uses Linq-Embed-Mistral, Qwen-8B and Qwen-0.6B rerankers, and Gemini 2.5 Flash.Gemini 2.5 Flash evaluates intent clarity.Source: §5.2 · p. 5

5.3. Pretraining Knowledge Check

  • Without retrieval context, models produced the correct Script Include namespace in only 5% of evaluation cases.The test used autocomplete without FIM, a knowledge graph, or an index.Source: §5.3 · p. 5

6. Experiments and Results

The experiments compare retrieval approaches on a clear-intent subset and report retrieval quality alongside latency. Hypothetical Code Generation leads the retrieval methods, while the optimized compact reranker is reported to match or exceed the larger model at lower latency.

6.1. Main Results

  • 87.86% top-40 accuracy for Hypothetical Code Generation on 705 clear-intent samples more than doubles the BM25 baseline.The paper reports superior MRR ranking quality for this approach as well.Source: Table 1 · p. 5, §6.1 · p. 5
  • The MRR results confirm that Hypothetical Code Generation has superior ranking quality to the other evaluated retrieval approaches.The methods compared are Prefix Code Embed, LLM Description, and Hypothetical Code Generation.Source: Table 2 · p. 5, §6.1 · p. 5

6.2. Latency Analysis

  • The latency analysis reports that the optimized 0.6B reranker matches or exceeds the 8B model while maintaining significantly reduced latency.Table 3 reports latency measurements for different pipeline configurations.Source: Table 3 · p. 5, §6.2 · p. 6

7. Post Training and Optimization

The paper develops a post-training pipeline to improve a compact reranker for real-time predictions, using separate open-source and synthetic datasets for training. The final SFT + RL model outperforms the 8B baseline on reported top-5 accuracy while retaining a latency gain.

7. Post Training and Optimization

  • The post-training pipeline targets ranking quality comparable to or better than the 8B reranker at lower latency and cost for real-time prediction.The compact model is intended to fit tighter memory budgets, support higher concurrency, and reduce tail latency.Source: §7 · p. 6

7.1. Training Dataset

  • To avoid evaluation contamination, the training data uses previously unused Script Include namespaces; SFT draws on JavaScript and TypeScript samples from CodeR-Pile.The CodeR-Pile subsets were selected to provide diversity and scale for supervised fine-tuning.Source: §7.1 · p. 6, §7.1.1 · p. 6

7.1.2. Synthetic Dataset for RL

  • The RL dataset began with 892 held-out namespaces and synthetic code triplets, then filtering reduced the pool to 204 examples with suitable hard negatives.Fuzzy matching removed 894 close variants after earlier leakage and proximity filtering.Source: §7.1.2 · p. 6

7.2.1. Supervised Fine-Tuning (SFT)

  • LoRA adapters performed best among the tested SFT approaches, while CodeR-Pile alone generalized better than synthetic-only or mixed training, which overfit or failed to improve results.Training randomly selected positive or negative samples and used negative log-likelihood loss.Source: §7.2.1 · pp. 6–7

7.2.2. SFT + RL

  • RL on the small synthetic dataset alone caused catastrophic forgetting; the reported approach applies RL to a CodeR-Pile SFT checkpoint before fine-tuning with synthetic data.The pipeline uses GRPO with a completion-aware binary reward and samples eight completions per prompt.Source: §7.2.2 · p. 7

7.2.3. Extended SFT

  • Extended SFT on the synthetic dataset after CodeR-Pile SFT produced decent results but no noticeable improvement over the earlier checkpoint.Source: §7.2.3 · p. 7

7.3. Training Results and Validation

  • The SFT + RL optimized 0.6B model achieved 68.58% top-5 accuracy versus 66.10% for the 8B model, a 2.48-percentage-point lead, while retaining a 2.5x latency gain.Source: §7 · p. 6, §7.3 · p. 7

8. Conclusion

DeepCodeSeek’s multi-stage retrieval pipeline achieves 87.86% top-40 accuracy, more than doubling BM25 performance. Its optimized 0.6B reranker also outperforms an 8B model with 2.5x reduced latency; the evaluation is limited by its dataset and synthetic training data.

8. Conclusion

  • 87.86% top-40 accuracy, more than doubling BM25 baseline performance, is achieved by the multi-stage retrieval pipeline.The conclusion frames this as addressing developer-intent inference from partial code.Source: §8 · p. 8
  • 68.58% vs 66.10% top-5 accuracy: the optimized 0.6B reranker outperforms the 8B model while maintaining 2.5x reduced latency.The conclusion presents this as supporting real-time applications.Source: §8 · p. 8
  • Knowledge graph filtering reduces the search space by 59%, enhanced indexing improves accuracy by 31 percentage points, and LLM reranking adds 7 percentage points.The authors describe these components as enabling real-time code completion in production environments.Source: §8 · p. 8

8.1. Limitations and Future Work

  • The evaluation is limited by its focus on Script Includes, synthetic data that may miss real-world complexity, and a 204-sample synthetic dataset.Future work includes broader data coverage, refining the knowledge graph, and task-specific reranker specialization.Source: §8.1 · p. 8

Declaration on Generative AI

The authors state that they used no generative AI tools in preparing the work. The appendix reports ablation studies, including graph-based filtering and enhanced indexing results.

Declaration on Generative AI

A. Ablation Studies

  • The appendix conducts comprehensive ablation studies to assess the contribution of each pipeline component.Source: A. Ablation Studies · p. 10

A.1. Knowledge Graph Filtering

  • 59%: Prioritizing globally scoped Script Includes reduces candidate sets; the graph covers 17,701 includes, and 84% of new usages follow existing patterns.Source: §A.1 · p. 10

A.2. Indexing Strategy Impact

  • Grouping methods under parent namespaces and enriching them with JSDoc documentation improves accuracy across all reported metrics.Table 5 quantifies the impact of the enhanced indexing strategy, but the supplied passages do not give its cell values.Source: §A.2 · p. 10, Table 5 · p. 10

A.3. Reranking Analysis

The reranking analysis reports higher top-5 accuracy for Gemini 2.5 Flash than for the Qwen 8B reranker. The following ablations examine context length, prefix versus suffix code, and the proximity of code to the API invocation.

A.3. Reranking Analysis

  • 72.60% top-5 accuracy for Gemini 2.5 Flash exceeds 65.84% for the Qwen Reranker (8B).Table 6 is a reranker comparison for Non-FIM, Dense 20.Source: §A.3 · p. 10, Table 6 · p. 10

A.4. Code Trimming and Context Length

  • 8–10 lines before the cursor yield the best Prefix Code Embed (Non-FIM) Search performance, a 1.82% relative increase over using a larger context.The passage cautions that downstream code generation may require more context.Source: §A.4 · p. 10, Figure 4 · p. 11

A.5. Code Before vs. After Analysis

  • Prefix code consistently outperforms suffix code for Script Include retrieval, though suffix code still provides an additional boost.Source: §A.5 · p. 11, Figure 5 · p. 11

A.6. Code Proximity and Relevance

  • The last 1–2 lines before the API invocation contain maximum retrieval information; removing them degrades Top-K accuracy and MRR across context lengths.The comparison is illustrated in Figure 6; broader context provides some benefit, but those immediately preceding lines are disproportionately valuable.Source: §A.6 · p. 12, Figure 6 · p. 12

B. Knowledge Graph Analysis

The analysis shows that package and scope metadata can sharply narrow Script Include search. Later experiments examine model selection, indexing and query strategies, reranker training, and out-of-distribution generalization.

B. Knowledge Graph Analysis

  • Approximately 59% of the search space is removed by focusing on non-global packages and scopes, and about 97% of non-global Script Includes map one-to-one to a package-scope pair.Many such pairs therefore map to a single Script Include, often avoiding deeper search within those contexts.Source: B. Knowledge Graph Analysis · p. 12

C. Ablation Study on Model Selection

D. Code Summarization Strategy

  • JSDoc-based indexing consistently outperforms raw-code indexing across retrieval methods, with particularly significant improvements in top-5 and top-10 accuracy.The paper attributes the improvement to JSDoc's structured, focused API descriptions, which reduce implementation-detail noise.Source: D. Code Summarization Strategy · p. 13, Figure 8 · p. 13

E.1. Supervised Fine-Tuning (SFT) Loss

  • During supervised fine-tuning, loss decreases steadily across epochs, while the reinforcement-learning training-loss figure tracks optimization convergence.The paper describes the supervised fine-tuning progression as stable convergence.Source: §E.1 · p. 14, Figure 9 · p. 14, §E.3 · p. 15

E.2. Reinforcement Learning Reward Progression

F. Out-of-Distribution Generalization

  • The trained Qwen 0.6B models neither significantly outperform nor degrade from the base model on an unseen dataset, indicating retained generalization without catastrophic forgetting.The evaluation uses an out-of-distribution dataset not seen during training.Source: F. Out-of-Distribution Generalization · p. 16, Table 7 · p. 16

H. Analysis of Query Enhancement Techniques

  • Natural-language summaries of user code can miss important details or keywords, while code-based query methods provide more specific information and perform better in some situations.The paper compares LLM descriptions with Prefix Code Embed and Hypothetical Code Generation.Source: H. Analysis of Query Enhancement… · p. 17

I. Qualitative Examples of Retrieval Techniques

  • On the shared array-intersection example, prefix-code and hypothetical-code retrieval rank ArrayUtil first, while an LLM-generated description ranks it second.The example contrasts retrieval outcomes for the same partial code.Source: §I.2 · p. 18, §I.3 · p. 18, §I.4 · p. 19

Related papers

Loading 2509.25716v1…