Source-linked AI summary
A Tool-Augmented, GPT-4 Chatbot for Real-Time Repository Data Analysis
Muhammad Jawad Chowdhury, Md. Sakib Khan
TL;DR
Repository information is difficult for non-technical users and developers to query despite its value for understanding software projects. The paper introduces a GPT-4 chatbot that parses queries, selects repository tools, and generates responses from retrieved data. On 66 in-scope questions, it achieved 98.48% accuracy, while correctly abstaining on all 14 out-of-scope questions.
Problem
Repository data is difficult for non-technical stakeholders and developers to access because querying interconnected repository information requires specialized expertise.
Method
The chatbot parses natural-language queries, extracts filters, selects tools, and uses GPT-4 with GitHub repository data to generate responses.
Results
98.48% accuracy was achieved on 66 in-scope questions, while all 14 out-of-scope questions were correctly identified and answered with abstention.
Takeaways & Limitations
The framework makes repository insights more accessible to technical and non-technical users through targeted prompt engineering and strategic tool use.
Takeaways & Limitations
The system’s query coverage and versatility remain constrained by the available toolset, especially for specialized or complex repository-analysis tasks.
Abstract
from arXiv · showhide
Software repositories contain vast amounts of data on code contributions, bug reports, and project activities, yet this information remains challenging for non-technical stakeholders and developers to access due to limited expertise in querying repositories. To address this, we introduce a novel chatbot architecture leveraging OpenAI's GPT-4 model for automated extraction and analysis of repository data. In contrast, our architecture takes a structured path first by parsing the user's query to extract relevant parameters, then selecting the correct tool to employ based on that analysis, and finally invoking the GPT-4 model to create a highly detailed response. In contrast to previous work based on multi-component systems with embedding models and document retrievers, our architecture inverts the process by relying on prompt engineering and tool selection to fit with the query intent. To validate our approach, we conducted experiments on various question types, including Issues, Pull Requests, Commits, Compound Questions, and General Repository Information, evaluating our target prompts' ability to improve the accuracy of responses from the model. Beyond demonstrating the utility of this architecture to a diverse set of users, our findings suggest that this architecture can make repository data more accessible to technical and non-technical audiences through the production of actionable insights.
I. INTRODUCTION
Repository data can support software-quality insights, but its interconnected elements and querying demands make analysis difficult, especially for non-technical stakeholders. The paper proposes a tool-augmented GPT-4 chatbot to extract and analyze repository information through structured query processing and tool selection.
- Motivation: Repository data from GitHub includes commits, pull requests, issues, and contributor activities that can inform software quality and development-process understanding.The interconnected nature of these elements makes useful extraction time-consuming and challenging.
- Motivation: Non-technical stakeholders can benefit from repository analysis through better collaboration, clearer communication, and better decision-making.On-demand summary reports can help them monitor progress, identify risks, and make data-driven decisions without relying solely on developers.
- Background: Large language models are being applied across software-engineering tasks, including code generation, program repair, and code summarization.Prior work also includes LLM-based software-engineering chatbots and prompt-engineering approaches for improving code summarization.
- Contributions: The proposed architecture parses natural-language repository queries, extracts parameters, selects tools, and retrieves real-time data through the GitHub REST API.The evaluation dataset contains 80 repository-related questions spanning pull requests, commits, issues, compound queries, general information, and out-of-scope questions.
B. Software Engineering Chatbots
The paper situates its chatbot among systems that answer repository, Stack Overflow, and code-related questions. Its interaction design combines system instructions, summarized query history, and the latest user query to generate context-aware responses.
- Software Engineering Chatbots: Previous software-engineering chatbots retrieved repository data, summarized multi-answer Stack Overflow posts, or answered code-related questions.The cited systems were designed to provide concise answers or improve the speed and accuracy of developer tasks.
- User Interaction: The user submits a GitHub repository URL and query, which the system combines with system messages and summarized conversation history.This combined input is processed to generate the chatbot response.
- Context Construction: System messages define the chatbot’s responsibilities and response format, while summarized prior queries preserve multi-turn context without exceeding the token limit.The history excludes previous responses and is appended to the latest query.
B. System Prompt:
The system prompt establishes the chatbot’s GitHub-analysis role, identifies the repository under analysis, and specifies the tools available for answering queries.
- System Prompt: The system prompt is a predefined instruction that describes the chatbot’s role, capabilities, and operating context.It is provided at the beginning of the interaction.
- Agent Identification: The prompt identifies the chatbot as a GitHub Repository Analysis Agent named Github-chatbot.This defines the agent’s identity and purpose for repository analysis.
- Repository Specification: A repository URL placeholder dynamically specifies which GitHub repository the chatbot should analyze.The placeholder is populated with the URL of the target repository.
- Tool Access: Tool-access information lists the operations available for repository analysis, including repository reports and reports for commits, issues, and pull requests.These tools retrieve repository structure, metadata, contributions, and detailed activity information.
C. Query Processing:
The query-processing stage classifies a user’s repository question and applies requested filters so the system can direct it toward targeted GitHub data.
- Query Classification: The system parses and classifies each query to identify key components and select an action using keywords and structural rules.This determines how the query should be handled.
- Data Classification: A keyword-matching function f : Q → C directs queries into repository-data categories such as issues, pull requests, and commits.The categories represent the primary data streams used by the system.
- Filter Application: User-specified filters F = {f_1, f_2, . . . , f_n} refine queries using criteria such as state, author, date range, or result limit.The resulting subset Q_F ⊂ Q is intended to match the user’s requirements and produce targeted results.
D. Tool Selection and API Interaction:
The system analyzes a query to determine whether specialized repository tools are needed, selects an appropriate tool, and uses GitHub’s REST API to retrieve requested data.
- The system determines whether specialized tools are required to retrieve or transform the requested repository data.
- The Repository Report Tool generates comprehensive reports on repository metadata and overall structure.
- The Commits/Issues/PR Report Tool handles data related to commits, issues, and pull requests.
- After tool selection, the system formulates an API request using the repository URL, endpoint, scope, and filters.The request is handled through GitHub’s REST API, which fetches the required data.
E. Response Generation
Response generation adapts to tool availability: the system uses internal knowledge without a selected tool and integrates retrieved data chunks when a tool is selected.
- The response-generation process adapts according to whether a tool is selected.
- Without a selected tool, the system generates a response using internal knowledge.
- With a selected tool, the system generates a response using tool-based data and processes each retrieved data chunk.
- The conditional framework supports future tools and algorithms through modular expansion of response-generation capabilities.
IV. DATASET
The evaluation dataset contains 80 repository-related questions generated from exemplar queries, organized across distinct repository-analysis categories and including unanswerable cases.
- The custom dataset contains 80 questions generated from 15 exemplar repository-analysis queries.The variations preserve the original intents while diversifying entities and phrasings.
- The questions were divided into five distinct categories for repository-related analysis.
- General Information questions test high-level repository details and general metadata.
- The evaluation includes unanswerable questions to assess whether the model recognizes its limitations.
- Figure 2 shows a sunburst distribution of the dataset’s first two words.
V. RESULTS AND ANALYSIS OF QUERY HANDLING AND RESPONSE GENERATION
The evaluation examined query-handling iterations and out-of-scope behavior, finding that simple questions used one iteration while most questions used two.
- A. Scope and Limitations Handling: For out-of-scope questions, GPT-4 recognized that the requests exceeded the available tools and refrained from fabricating answers.The authors report that this behavior helped manage user expectations about tool constraints.
- B. Iteration Count Analysis: The evaluation analyzed iteration counts across question categories using a frequency distribution.
- 1) Simple Queries: Simple questions such as repository-name queries required only one iteration.The model responded quickly without complex processing.
- Most questions required two iterations: one for tool selection and one for final-answer generation.This two-step process was effective for queries aligned with the available tools.
2) Two-Iteration Queries:
Compound repository questions typically require multiple processing iterations, while the chatbot’s limited toolset constrains the breadth of analyses it can perform. Expanding available tools could improve accuracy and versatility for specialized repository questions.
- Compound Questions: Compound questions typically required three iterations to break down, classify, and answer each component cohesively.The process was intended to support accurate and complete responses to multi-part queries.
- Iteration Patterns: Some questions required a second iteration for missing data, while out-of-scope queries sometimes required up to five iterations before limitation recognition.Higher iteration counts reflected attempts to reclassify and process difficult or unsupported queries.
- Available Tools: The chatbot uses two primary tools: one for comprehensive repository reports and one for entity-specific issues, commits, and pull requests.Tool selection is therefore divided between broad repository reporting and targeted entity questions.
- Tool Availability: Additional tools for issue sentiment and contribution trends could make the system more accurate, versatile, and comprehensive.The authors identify tool availability as an important factor in chatbot effectiveness.
D. Evaluation of Model Accuracy
The evaluation compared responses with GitHub-derived ground truth across in-scope and out-of-scope questions. The model achieved high in-scope accuracy, abstained on unsupported questions, and was presented as a tool-augmented GPT-4 architecture for accessible repository analysis.
- Evaluation Criteria: Responses were marked correct only when they returned expected values and applied all requested filters.Out-of-scope responses were correct when they explicitly acknowledged unavailable tool capabilities.
- Scope Recognition: All 14 out-of-scope questions were correctly identified and answered with abstentions.Out-of-scope correctness required explicitly indicating that available tools could not answer the query.
- Accuracy Results: 65 of 66 in-scope questions were correct, achieving 98.48% accuracy.The single error involved selecting the wrong date parameter and using the current timestamp instead.
- Architecture and Accessibility: The proposed GPT-4 chatbot architecture streamlines repository data extraction and analysis for technical and non-technical users.The broader evaluation used precise prompts and strategic tool use across repository question types.