Source-linked AI summary
Qlippy: A Retrieval-Augmented GenAI Assistant for Reproducible Quantum Workflows and Experiment Tracking
Mahee Gamage, Vlad Stirbu
TL;DR
Quantum developers struggle to adopt reproducibility practices because of unstable hardware, specialized tooling, and limited domain grounding in general-purpose language models. Qlippy addresses this gap with a retrieval-augmented assistant that explains concepts and injects MLflow tracking aligned with QProv into development workflows. Preliminary evaluation indicates that a locally hosted model can approach a commercial model on answer relevancy and factual coverage within a RAG pipeline.
Problem
Novice quantum developers face complex experiment-tracking and provenance practices, while generic language models lack grounding in domain-specific materials and may hallucinate.
Method
Qlippy retrieves curated quantum-software knowledge to explain reproducibility and provenance concepts and augment programs with MLflow tracking aligned to QProv.
Results
A locally hosted model can approach a commercial model on answer relevancy and factual coverage when used within a RAG pipeline.
Takeaways & Limitations
Grounding separates knowledge from model parameters, giving explicit control over response scope and provenance while supporting low-cost, privacy-preserving local deployment.
Takeaways & Limitations
The evaluation is based on a prototype, depends on curated knowledge sources, lacks extensive real-user validation, and retains residual LLM limitations.
Abstract
from arXiv · showhide
Quantum software development is iterative and error-prone. Noisy hardware and repeated re-execution make experiment tracking, provenance, and reproducibility essential, yet these practices are hard to adopt because of tooling complexity and the specialized knowledge they demand. General-purpose language models can help but tend to hallucinate and lack grounding in domain-specific tooling. We present Qlippy, a retrieval-augmented GenAI assistant embedded in the development environment that grounds its responses in a curated corpus of quantum-software-engineering knowledge. Qlippy explains reproducibility and provenance concepts in context and augments existing Qiskit programs with MLflow-based experiment tracking aligned to the QProv schema. By separating knowledge from model parameters, grounding gives explicit control over the scope and provenance of the assistant's responses and reduces reliance on model scale, which points toward low-cost, privacy-preserving local deployment.
I. INTRODUCTION
Quantum software developers face difficulty adopting experiment tracking, provenance, and reproducibility practices, while generic language models lack domain-specific grounding. Qlippy addresses these challenges through a retrieval-augmented assistant integrated into Visual Studio Code.
- Quantum software development requires software-engineering practices such as experiment tracking, reproducibility, and structured data management.
- Practitioners, especially novices, often have limited exposure to reproducibility and lifecycle-management concepts.
- Experiment tracking and provenance are frequently applied superficially or omitted, producing ad hoc practices and less transparent experimental results.
- Integrating external tools increases cognitive load, while generic language models may generate explanations or code misaligned with specialized domain practices.
- Qlippy uses curated tools documentation and research literature to provide context-aware explanations and augment code for experiment tracking and structured provenance collection.
II. BACKGROUND
Quantum experiments require systematic tracking because unstable hardware and iterative execution affect results, yet the associated practices remain difficult for developers to learn and apply. Tool-supported workflows and grounded generative AI motivate more accessible assistance.
- Hardware noise, calibration drift, and backend variability can change quantum results across executions, making experiment tracking important.
- Quantum development is inherently iterative, requiring repeated refinement of circuits, parameters, and execution strategies.
- Existing approaches to experiment structuring and reporting set a high bar for novice developers.
- Developers must coordinate quantum algorithms, hybrid execution environments, and infrastructure while handling abstractions such as MLflow and QProv.
- Large language models can provide relevant responses but suffer from hallucinations, outdated knowledge, and weak grounding in domain-specific facts.
III. DESIGN
Qlippy separates local developer interaction from research services connected through a REST API. Its RAG service retrieves curated quantum-software documentation and generates responses from that external knowledge.
- The architecture connects a Visual Studio Code extension on the developer workstation to research services through a REST API.
- The research environment contains an MLflow tracking server and a Python-based RAG service.
- Fig. 1 presents the system architecture and identifies newly developed components in gray.
- The RAG service stores curated documentation in a ChromaDB vector database, retrieves relevant context, and synthesizes responses from it.
B. Developer interaction model
Qlippy integrates explanation, code assistance, and source grounding into the developer’s existing quantum-programming workflow. This links conceptual learning directly to implementation.
- Developers ask natural-language questions while writing or exploring quantum code in the development environment.
- The assistant explains concepts, augments or modifies code, and references relevant sources in response.
- Coupling explanations with immediate code changes reinforces conceptual understanding and practical implementation within the workflow.
C. Integration with Tracking
Qlippy integrates directly with MLflow, augmenting user code with tracking calls that record experiment runs, parameters, metrics, and artifacts.
- C. Integration with Tracking: Qlippy automatically inserts MLflow-based tracking calls into user code.The integration records experiment runs, parameters, metrics, and artifacts using a professional research and industry tracking system.
A. Advisory Scenario: In-Context Explanation of Experiment Tracking
Qlippy supports both advisory explanation and active code augmentation for experiment tracking. Users can explore provenance concepts interactively, while the assistant translates tracking requirements into executable MLflow instrumentation.
- A. Advisory Scenario: In-Context Explanation of Experiment Tracking: The advisory scenario retrieves curated MLflow and QProv knowledge to explain experiment tracking concepts in context.Users interact through natural-language queries embedded in the development environment.
- A. Advisory Scenario: In-Context Explanation of Experiment Tracking: Interactive questioning connects abstract provenance concepts with concrete implementation practices.Users can refine questions and request examples while learning what to track, including circuit properties and execution parameters.
- B. Augmentation Scenario: AI-Assisted Injection of Experiment Tracking Code: The augmentation scenario modifies existing Qiskit programs by adding MLflow logging aligned with stated tracking requirements.The system retrieves relevant knowledge and generates a modified program with integrated tracking functionality.
- B. Augmentation Scenario: AI-Assisted Injection of Experiment Tracking Code: Successful augmentation preserves the original program logic while logging parameters, metrics, and provenance data consistent with QProv guidelines.The modified program is expected to execute correctly and record experiment data in MLflow.
- B. Augmentation Scenario: AI-Assisted Injection of Experiment Tracking Code: The augmentation scenario shifts Qlippy from passive advice to active development assistance.This lowers the operational barrier to adopting structured experiment tracking in quantum workflows.
A. Preliminary evaluation
A preliminary RAGAS evaluation examined Qlippy’s advisory scenario across experiment-tracking questions and multiple response qualities. The reported benefits include grounded, context-controlled assistance for adopting reproducible quantum software practices.
- A. Preliminary evaluation: The preliminary evaluation used RAGAS on 20 question–answer pairs spanning five experiment-tracking subdomains and four question types.The question types were factual, summary, reasoning, and unanswerable; GPT-5.4-mini served as the judge.
- A. Preliminary evaluation: Responses were scored for faithfulness, context recall, answer relevancy, and factual correctness.Unanswerable questions were excluded from the means.
- A. Preliminary evaluation: Grounding in a curated corpus gives Qlippy explicit control over the scope and provenance of its responses.The assistant is intended to keep explanations aligned with reproducible quantum software practices rather than open-ended model behavior.
C. Operational concerns
Local deployment can improve cost, privacy, and execution control, but it requires infrastructure and operational maintenance. The study also limits its claims because the prototype and curated knowledge base have not been extensively validated in real-world settings.
- C. Operational concerns: Local models such as qwen3.5:9b can approach GPT-5.4 on answer relevancy and factual coverage within a RAG pipeline.RAG shifts knowledge responsibilities from model parameters to curated external sources, reducing the advantage of larger models in constrained domains.
- C. Operational concerns: Local deployment eliminates recurring API usage fees and provides greater control over data flows.These properties matter when proprietary code or unpublished research artifacts require cost or confidentiality control.
- C. Operational concerns: Local deployment requires adequate computational infrastructure, model-update management, and system-reliability measures.Choosing between commercial and local models is therefore a broader socio-technical decision involving cost, governance, and operational capabilities.
- D. Threats to validity: The evaluation uses a prototype that may not reflect performance and usability in real-world deployments.This limits how broadly the preliminary findings should be interpreted.
- D. Threats to validity: The system depends on curated knowledge sources, has limited real-user validation, and retains occasional hallucinations and response variability.These limitations are particularly relevant for gaps or biases in the corpus and complex queries.
VI. CONCLUSION
Qlippy embeds experiment tracking and provenance into quantum software workflows through curated-source grounding and MLflow instrumentation aligned with QProv. Preliminary evaluation suggests locally hosted models can approach commercial models on answer relevancy and factual coverage, while further validation remains planned.
- Qlippy couples curated-source grounding with automated MLflow instrumentation aligned to the QProv schema.
- The assistant supports reproducible quantum software practices directly within developers’ workflows, including for novice practitioners learning these concepts.
- Within a RAG pipeline, a locally hosted model can approach a commercial model on answer relevancy and factual coverage.
- The evaluation is preliminary, with future work targeting multi-expert validation, provenance-completeness assessment, and comparisons against ungrounded and general-purpose assistants.