QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

1 Introduction

• Existing QC evaluations focus on specific aspects, leaving the broader workflow from circuit construction through simulation without comprehensive benchmark coverage.The stated workflow also includes compilation, debugging, and error correction.Source: §1 · p. 1
• LLM capability may not generalize across QC tasks, with models excelling at oracle synthesis yet scoring near zero on hardware routing.The paper notes that a model ranked first overall may score 0% on a challenging task.Source: §1 · p. 1
• QC-Stark covers 11 QC tasks with five difficulty levels, seed-based procedural generation, and automatic verification through code execution.These design elements target workflow coverage, contamination resistance, and verification without manual judgment.Source: §1 · p. 1

2 Related Work

• Quantum-Audit [4] evaluates conceptual knowledge, Qiskit QuantumKatas [2] tests educational exercises, and QCoder uses contest problems with simulator feedback.Their reported best performances are 84%, 83%, and 78%, respectively.Source: §2 · p. 2
• QCircuitBench [5] provides large-scale algorithm-design data, while QC-Stark spans the QC operational spectrum with five difficulty levels.Source: §2 · p. 2
• Outside QC, SciCode [6] and CMT-Benchmark [7] report frontier-model best performances of 4.6% and 30%, respectively.The paper presents these results as implying a need for continuously developing benchmarks distinct from existing ones.Source: §2 · p. 2

3 Benchmark Design

• The 11 tasks cover circuit construction, code understanding, verification, compilation, simulation, and error correction.They include tasks such as debugging, equivalence checking, hardware routing, VQE, and syndrome decoding.Source: §3 · p. 2
• Each task has five difficulty levels, from Textbook (L1) to Open (L5), controlled by problem size, constraint complexity, and domain-specific parameters.Source: §3 · p. 2
• Each instance is generated deterministically from a task, level, and seed, and a verifier executes the model’s solve() function against ground truth.Correctness checks include state fidelity, functional equivalence, and correct identification; human judgment is not required.Source: §3 · p. 2
• The evaluation includes 10 models selected under availability constraints to cover different parameter sizes, with the authors noting possible bias toward Claude models.Claude Code, using Opus 4.6 and higher, was used to design the benchmark.Source: §3 · p. 2
• Table 1 reports mean accuracy per model and task at maximum supported token budgets, with an overall mean and standard deviation across seeds.Bold and gray mark the best and worst per-task scores, respectively.Source: Table 1 · p. 3

4.1 Overall Performance and Rank Inversions

• Claude Sonnet 5 scores 0.662 overall, while performance drops 59% from L1 to L5 and task difficulty ranges 5.9×, from Equivalence at 0.78 to Debugging at 0.13.The results summarize 2,750 evaluations.Source: §4.1 · p. 3
• On 4 of 11 tasks, overall rankings do not predict per-task performance, with statistically non-significant Spearman correlations.The affected tasks are Routing, Trotterization, Debugging, and Equivalence.Source: §4.1 · p. 3

4.2 The “Debugging” Conundrum

• Six of 10 models score 0.00 on debugging despite running at their maximum supported token budgets.Claude Sonnet 5 leads at 60% (15/25), while o4-mini scores 0% at a 65,536-token budget.Source: §4.2 · p. 3

4.3 Routing: A Universal Bottleneck

• Routing averages 0.21 accuracy across models, with Gemini Flash Lite leading at 0.44 and GPT-5.4 scoring 0.04.Gemini Flash Lite’s stronger performance is concentrated in L1/L2; many evaluations produce circuits not functionally equivalent to the input.Source: §4.3 · p. 4

4.4 Measurement Validation

• The 2PL IRT model yields marginal reliability of 0.985, and 216 of 275 items (78.5%) have discrimination a > 1.0.Difficulty progresses monotonically from L1 (b = −0.88) to L5 (b = +0.89), and IRT and raw-accuracy rankings correlate at ρ = 0.976 (p < 0.001).Source: §4.4 · p. 4
• Prompt-condition rankings are largely preserved, with ρ = 0.915 and Kendall τ = 0.778 across 2,745 paired evaluations per condition.The top four models remain in the top four; the bottom four do not.Source: §4.4 · p. 5
• The benchmark requires Qiskit 2.x Python output, which may conflate quantum-domain knowledge with framework-specific API proficiency.The authors note that framework-agnostic output or retrieval-augmented generation could further isolate reasoning ability, but each has trade-offs or is future work.Source: §4.4 · p. 5

5 Conclusion

• Dedicated reasoning models o4-mini and o3-mini score 0% on debugging even at 64K-token budgets, while Gemini Flash Lite, ranked 8th overall, leads routing at 44%.Best task accuracies range from 13.2% in Debugging to 78.0% in Equivalence.Source: §5 · p. 5
• Performance drops 59% from difficulty level L1 to L5.Source: §5 · p. 5
• Prompt rankings are robust (ρ = 0.915), and IRT analysis reports reliability r = 0.985 with monotonic difficulty progression.Future work includes RLVR-based fine-tuning, framework-agnostic representations, and retrieval-augmented generation.Source: §5 · p. 5

• Spearman ρ = 0.915 and Kendall τ = 0.778 between model rankings under the two prompt conditions.Source: A Prompt Sensitivity Details · p. 7
• Overall accuracy rises 0.403 →0.446 with the structured prompt.Source: A Prompt Sensitivity Details · p. 7
• The top-4 grouping is unchanged across prompt types, while the bottom-4 ranks change.Source: A Prompt Sensitivity Details · p. 7

B.1 System Prompts

• The system prompt casts the model as a quantum computing expert writing Python code.Source: §B.1 · p. 7
• The canonical prompt specifies Python 3.13, Qiskit 2.x, NumPy, and SciPy, with updated Qiskit API guidance.It directs users to qiskit.qasm2.dumps, Statevector or StatevectorSimulator, and qiskit.quantum_info in place of removed or deprecated APIs.Source: §B.1 · p. 7
• The canonical output instructions require executable Python code only and a solve() function that returns the answer.Source: §B.1 · p. 7
• The minimal prompt retains expert framing, code-only output, Qiskit 2.x, and the solve() function while shortening the API guidance.It specifically warns against deprecated .qasm() and recommends qiskit.qasm2.dumps() when needed.Source: §B.1 · p. 7

B.2 Task Prompt Examples

• The state-preparation example asks for a circuit producing a two-qubit target state with equal amplitudes on |00> and |11>.Source: §B.2 · pp. 7–8
• The state-preparation instructions permit initialization or standard gates, require a normalized statevector and fidelity above 0.999, and request only the circuit object.Source: §B.2 · p. 8
• The debugging example specifies exactly one circuit bug and asks for its identification and a corrected circuit using the intended basis-state mapping.Source: §B.2 · p. 8

• Performance drops 59% from L1 to L5, with accuracy decreasing monotonically as difficulty increases.Source: C Difficulty-Level Breakdown · p. 8
• State Preparation falls from 1.00 to 0.14, VQE from 0.96 to 0.12, and Oracle Synthesis from 0.88 to 0.24 between L1 and L5.These are the steepest reported L1-to-L5 drops.Source: C Difficulty-Level Breakdown · p. 8
• Noise Fidelity stays roughly flat from 0.70 to 0.66, while Debugging has the smallest absolute decline, from 0.16 to 0.02.Source: C Difficulty-Level Breakdown · p. 8

D Task Descriptions

• Table 2 summarizes a 2PL IRT fit over 55 task-level cells, five seeds, and 2,750 responses, including model abilities, item difficulties, and discriminations.State Preparation L1 and L2 are excluded from difficulty means because every model passed them under every seed.Source: Table 2 · p. 9
• Table 3 reports mean binarized-accuracy differences between structured and minimal prompts by model and task across 2,745 paired evaluations per condition.Five Gemma-4-31B State-Preparation L5 calls were excluded because minimal-condition results could not be obtained after six retries.Source: Table 3 · p. 10
• Table 4 reports mean accuracy by difficulty level across all models and tasks.Source: Table 4 · p. 10
• Table 5 lists QC-Stark task descriptions alongside their scoring criteria and difficulty-scaling parameters.Source: Table 5 · p. 11