Source-linked AI summary
Code4MeV2: a Research-oriented Code-completion Platform
Roham Koohestani, Parham Bateni, Aydin Ebrahimi, Behdad Etezadi, Kiarash Karimi, Maliheh Izadi
TL;DR
Commercial coding assistants restrict researchers’ access to interaction data and control over study conditions. Code4Me V2 is an open-source, modular JetBrains research platform; its measured end-to-end completion latency was 186.31 ms (± 139.50), and evaluations elicited positive feedback on extensibility alongside improvement areas.
Problem
Method
Code4Me V2 is an open-source JetBrains plugin with modular client–server architecture and configurable data collection for research on AI-assisted software development.
Source: §1 · p. 1, §3.1 · p. 2, §3.3 · p. 2Results
186.31 ms (± 139.50) was the measured end-to-end code-completion latency; expert researchers praised modularity and extensibility, and daily users reported fewer timing issues than experts.
Source: §4 · p. 4Takeaways & Limitations
Code4Me V2 is presented as shared research infrastructure that can let researchers focus on experimental design and data analysis rather than building collection systems.
Source: §6 · p. 4The interface for experiment configuration can have a steep learning curve for non-experts, and completion speed, model quality, and context retrieval remain short of commercial counterparts.
Source: §5 · p. 4
Abstract
from arXiv · showhide
The adoption of AI-powered code completion tools in software development has increased substantially, yet the user interaction data produced by these systems remain proprietary within large corporations. This creates a barrier for the academic community, as researchers must often develop dedicated platforms to conduct studies on human--AI interaction, making reproducible research and large-scale data analysis impractical. In this work, we introduce Code4MeV2, a research-oriented, open-source code completion plugin for JetBrains IDEs, as a solution to this limitation. Code4MeV2 is designed using a client--server architecture and features inline code completion and a context-aware chat assistant. Its core contribution is a modular and transparent data collection framework that gives researchers fine-grained control over telemetry and context gathering. Code4MeV2 achieves industry-comparable performance in terms of code completion, with an average latency of 200~ms. We assess our tool through a combination of an expert evaluation and a user study with eight participants. Feedback from both researchers and daily users highlights its informativeness and usefulness. We invite the community to adopt and contribute to this tool. More information about the tool can be found at https://app.code4me.me.
1 Introduction
Research on AI-assisted programming remains constrained by closed tools and bespoke study infrastructure. Code4Me V2 is an open-source, modular JetBrains plugin intended to support experimentation and shared research infrastructure.
1 Introduction
- Closed commercial systems limit researchers’ access to model behavior, experimental controls, and interaction telemetry.Prior studies often required bespoke, single-purpose tools, restricting infrastructure reuse.Source: §1 · p. 1
- Code4Me V2 is an open-source JetBrains plugin with inline completion and chat, built as a research platform with a configurable data-collection and inference backend.Its architecture separates user-facing components from the backend.Source: §1 · p. 1
- The authors propose Code4Me V2 as shared infrastructure so researchers can focus on experimental design and data analysis rather than building and maintaining collection systems.They invite the AI4SE community to adopt and contribute to the tool.Source: §1 · p. 1
2 Domain Analysis
Commercial coding assistants provide integrated experiences but restrict access to their suggestion logic, model updates, and interaction data. Existing open-source alternatives offer more transparency, yet often lack the modularity needed for systematic research.
2 Domain Analysis
- Commercial tools such as GitHub Copilot, Cursor, and JetBrains AI are proprietary black boxes that limit inspection, control of model updates, and access to raw interaction data.The passage notes that model updates can act as confounding variables.Source: §2 · p. 2
- Open-source alternatives such as Cody and Code4Me [6] were not designed primarily for research extensibility, and adapting them can require substantial core-code modification.This complicates systematic studies of developer–AI interaction.Source: §2 · p. 2
- After analyzing seven coding assistants, the authors designed Code4Me V2 as an open-source platform that is transparent, controllable through modular data collection, and extensible with new telemetry or context providers.The named platforms include GitHub Copilot, Cursor, Cody, Devin, Replit Ghostwriter, Code4Me [6], and JetBrains AI.Source: §2 · p. 2
3 Code4Me V2
Code4Me V2 uses modular components across its client, backend, frontend, and analytics platform. Its architecture separates concerns, supports configurable research data collection, and provides tools for analyzing telemetry and experiments.
3 Code4Me V2
- Code4Me V2 is organized around modularity and separation of concerns; Figure 1 depicts the backend, IntelliJ plugin, and admin dashboard.The figure’s panels (a), (b), and (c) correspond to those components, respectively.Source: §3 · p. 2, Figure 1 · p. 3
3.1 General Architecture
- The client–server architecture lets components be modified separately and supports extensions such as a VSCode client or a new model.The JetBrains plugin renders ghost text and chat, gathers configurable editor context and telemetry, and dispatches requests; the server handles authentication, storage, and model inference, while the analysis platform supports researcher analysis and study setup.Source: §3.1 · p. 2
3.2 Backend
- The Python backend uses an asynchronous API and a Celery task queue with Redis so inference and persistence run outside the main request-response cycle.The system supports WebSocket and HTTP requests; PostgreSQL stores user profiles, telemetry, and suggestion metadata.Source: §3.2 · p. 2
3.3 Frontend
- The frontend’s hierarchical modules let a manager dispatch requests to aggregators and self-contained modules for specific data classes or processes.Researchers can add telemetry by implementing and registering a module; actions run in data-collection and after-insertion stages when a user submits a completion request.Source: §3.3 · p. 2
3.4 Analytics Platform
- The analytics platform summarizes stored event streams into time-series aggregates, acceptance statistics, latency percentiles, and calibration analyses.It supports A/B testing and configuration management, with role-based access and admin-only cross-user comparisons and study endpoints.Source: §3.4 · p. 3
4 Preliminary Evaluation
The preliminary evaluation combined server-performance measurements with formative feedback from four expert researchers and a second evaluation with four daily users. Completion latency was reported as within a non-disruptive threshold, and feedback highlighted extensibility alongside areas for improvement.
4 Preliminary Evaluation
- 186.31 ms (± 139.50) was the end-to-end code-completion latency, averaging 18.66 tokens; chat latency was 8369.78 ms (± 840.48), averaging 277.08 tokens.The completion measure ran from the IDE trigger to ghost-text rendering and was reported as within the threshold for a non-disruptive user experience.Source: §4 · p. 4
- Four expert researchers unanimously praised modularity and extensibility, while identifying the module-management UI’s intuitiveness and suggestion timing as improvement areas.They completed a multi-file programming task using both inline completion and chat; feedback came from interviews and a questionnaire.Source: §4 · p. 4
- In the second evaluation, four daily users did not echo the experts’ previously identified issues, and fewer mentioned timing problems, though one wanted slightly faster suggestions.A participant also requested an agent feature, citing the rapidly evolving domain and academia’s need to adapt quickly.Source: §4 · p. 4
5 Limitations and Future Work
The platform has usability and performance gaps relative to non-expert configuration and well-resourced commercial tools. Future work includes improving context retrieval, particularly through project-wide retrieval.
5 Limitations and Future Work
- The configuration interface can have a steep learning curve for non-expert users, despite improvements in a second iteration.The authors also note room to improve how settings are presented, or whether they should be shown.Source: §5 · p. 4
- Project-wide context retrieval remains a future objective; current retrieval is described as effective in single-file and multi-file settings but still needing improvement.The paper identifies project-wide retrieval as a primary objective for improving suggestion accuracy.Source: §5 · p. 4
- Completion speed and model quality do not yet match those of well-resourced commercial tools.Source: §5 · p. 4
6 Conclusion
Code4Me V2 is an open-source, modular platform intended to lower barriers to empirical research in AI-assisted software development. Preliminary evaluations indicate practical day-to-day use and adaptability for experimental needs.
6 Conclusion
- Code4Me V2 is an open-source, modular platform designed to lower the barrier to empirical research in AI-assisted software development.Source: §6 · p. 4
- Preliminary evaluations found the platform practical for day-to-day use and adaptable to experimental needs.Source: §6 · p. 4
- The authors present Code4Me V2 as transparent, controllable, and extensible infrastructure for academic investigation of human–AI collaboration in programming.Source: §6 · p. 4