InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

Motivation

• InteractBench targets the unresolved question of whether model-generated programs can solve interactive problems when key information must be acquired through runtime queries rather than provided upfront.Interactive problems require online querying under strict protocols and problem-specific query budgets, unlike batch-style problems with static inputs.Source: §1 · p. 1

Challenges

• Existing benchmarks underrepresent interactive problems, often rely on remote judges, and provide limited diagnostics beyond overall success metrics.These limitations hinder interactive coverage, fully offline evaluation, and separation of algorithmic errors from interaction-specific failures.Source: §1 · p. 2

Contributions

• InteractBench introduces a curated benchmark of interactive competitive-programming problems with consolidated categories, difficulty tiers, versioned snapshots, and temporal-contamination diagnostics.The problems are drawn from Codeforces, AtCoder, IOI, and ICPC, with semi-annual refreshes planned to maintain freshness.Source: §1 · p. 2
• Each problem includes an executable local interactor, enabling fully offline evaluation without external judge submissions and supporting repeatable experimentation.The harness packages the interaction environment with every problem rather than requiring remote execution infrastructure.Source: §1 · p. 2
• The evaluation harness supplements pass@k with protocol checks, query-budget enforcement, and a failure taxonomy covering wrong answers, protocol violations, query-budget overruns, and deadlocks or timeouts.These diagnostics distinguish algorithmic errors from failures specific to interaction.Source: §1 · p. 2

General code benchmarks

• InteractBench evaluates one-shot, self-contained programs whose success depends on implementing a sound strategy during program–interactor interaction, rather than iterative debugging.Few general code benchmarks cover strict multi-round protocols with explicit query budgets.Source: §2 · p. 3

Competitive programming benchmarks

• Competitive-programming benchmarks commonly provide full inputs upfront, rely on external judges, include few interactive problems, or offer only pass/fail diagnostics.These limitations constrain evaluation of unrevealed-information reasoning and protocol adherence.Source: §2 · p. 3
• InteractBench addresses these gaps through offline-executable local interactors and protocol-aware diagnostics for interactive problems.It complements existing benchmark axes spanning general code synthesis and competitive programming.Source: §2 · pp. 2–3

3.1. Problem Definition

• An execution is Accepted only when its transcript conforms to the protocol, stays within the query budget, and produces the correct final output.The framework models solver queries and interactor responses over T rounds, with the evaluator checking protocol compliance, Q_Π(τ) ≤ B, and correctness for hidden test case x.Source: §3.1 · p. 3

3.2. Task Selection

• Tasks are selected from Codeforces, AtCoder, IOI, and ICPC when they require multi-round interaction, enforceable protocol constraints, and explicit or monitorable query budgets.InteractBench supports both stdio and grader-linked interfaces, while excluding essentially batch-style single-call settings; tasks receive expert-verified difficulty tiers and one or more of seven categories.Source: §3.2 · pp. 3–4

3.3. Construction Pipeline

• The construction pipeline proposes generators and interactors, validates them against official verdicts and protocol constraints, and adjudicates failures through trace-grounded feedback before expert escalation.It generates a fixed, seeded suite of 100 test cases per task, uses adaptive interactors only when the statement permits history-dependent replies, and caps the loop at three iterations; 31 tasks required expert intervention.Source: §3.3 · pp. 4–5
• Offline validation runs the solver and local interactor as isolated processes, enforcing protocol and budget counters while logging transcripts, termination signals, and structured rejection reasons.Each task uses 15 accepted and 30 rejected submissions in its verification pool, and candidates must match official outcomes on all human submissions while rejecting mutation-derived probes.Source: §3.3 · p. 5

3.4. Post-hoc Audit

• A supplementary post-hoc audit compares offline and official Accepted/Rejected outcomes using at most 5 manually submitted solutions per task where permitted.The audit is restricted because platforms can prohibit automated submissions and impose strict rate limits.Source: §3.4 · pp. 5–6

4.1. Experimental Setup

• The benchmark evaluates 16 model configurations using C++ programs, standardized zero-shot interactive-I/O prompts, and 10 independent samples per problem at temperature 0.8.Tasks run locally with executable interactors, while reasoning and non-reasoning variants are reported separately for within-family comparisons.Source: §4.1 · p. 6

4.2. Metrics

• Performance is summarized with pass@1 and pass@5, averaged over the evaluation set, and unsuccessful executions receive mutually exclusive labels including IDLE, PE, QLE, CE, RE, TLE, MLE, and WA.The taxonomy distinguishes interaction-specific failures from algorithmic, compilation, runtime, and resource-limit failures.Source: §4.2 · p. 6

4.3. Experimental Results

• Hard interactive tasks remain unsolved: the best model reaches 0.597 hard pass@5, while only Gemini-3-Pro-Preview and GPT-5.2 exceed 0.400.Gemini-3-Pro-Preview reaches 0.961 easy pass@5, but hard tasks remain strongly discriminative.Source: §4.3 · p. 7, Table 2 · p. 7
• Reasoning improves performance but does not remove the Hard barrier: DeepSeek-V3.2-Thinking reaches 0.513 overall pass@5 versus 0.262 for DeepSeek-V3.2, while Qwen3-14B reaches 0.000 hard pass@5.Resampling gains are uneven; Gemini-3-Pro-Preview rises from 0.367 pass@1 to 0.597 pass@5 on Hard, whereas GPT-5.2 rises from 0.340 to 0.492.Source: §4.3 · p. 7
• Graph is the dominant category bottleneck: the best model stays below 0.500 pass@1, while Gemini-3-Pro-Preview reaches 0.613 on Bit versus 0.475 on Graph.Graph stresses information acquisition and state tracking; Data Structures additionally require long-horizon invariant maintenance, while frontier models specialize across categories.Source: §4.3 · pp. 7–8, Table 3 · p. 8
• Failure analysis finds WA dominant for many configurations, while PE and IDLE disrupt query-loop establishment and QLE reflects entering the protocol but exhausting the query budget.QLE is more prevalent among frontier models, including 0.119 for Gemini-3-Pro-Preview and 0.101 for GPT-5.2; runtime and resource failures are secondary.Source: §4.3 · p. 8, Table 4 · p. 8

• InteractBench comprises 322 interactive tasks with executable local interactors for fully offline, reproducible evaluation under unrevealed information, strict protocols, and limited query budgets.Source: §5 · p. 9
• The evaluation harness supplements pass@k with protocol-compliance checks, query-budget enforcement, and a fine-grained taxonomy separating algorithmic errors from interaction breakdowns.Source: §5 · p. 9
• Across 16 model configurations, performance consistently degrades as problem difficulty increases, revealing a gap between full-information coding benchmarks and interactive problem solving.Source: §5 · p. 9
• Algorithmic logic errors dominate failures, while protocol violations and query-budget overruns remain frequent even for frontier reasoning models.Source: §5 · p. 9
• Future work will expand coverage through versioned releases and develop prompting and training strategies targeting interactive querying, state maintenance, and protocol adherence.Source: §5 · p. 9

Scale limitation

• InteractBench’s scale is limited because interactive tasks are less common than batch-style competitive-programming problems and require reliable local interactors for offline evaluation.This constrains current benchmark coverage.Source: §6 · p. 9

Future expansion

• Future versions will expand coverage by collecting more eligible tasks and constructing new tasks with verified interactors.The proposed expansion addresses both task scarcity and evaluator reliability.Source: §6 · p. 9

Evaluation scope

• The experiments use zero-shot model-generated solvers, so follow-up studies should examine prompting and training for interactive querying, state maintenance, and protocol adherence.These strategies are not evaluated in the current experiments.Source: §6 · p. 9

Impact Statement

• InteractBench evaluates generated solvers that acquire information through multi-round queries under a strict interaction protocol.It targets program synthesis when key information is not revealed upfront.Source: Impact Statement · p. 9
• The benchmark supports research on reliable code generation, state tracking, and protocol adherence.Source: Impact Statement · p. 9
• InteractBench enables fine-grained, reproducible evaluation of LLM-based coding systems.Source: Impact Statement · p. 9
• The paper presents an evaluation methodology intended to advance machine learning by measuring model capabilities.Source: Impact Statement · p. 9
• The authors encourage responsible use of the work in accordance with applicable rules and norms.Source: Impact Statement · p. 9

A. InteractBench Specification

• The appendix collects benchmark specifications and supplementary materials for InteractBench.Source: A. InteractBench Specification · p. 13
• The appendix is organized into specification, evaluation setup, supplementary experiments, and diagnostics and artifacts.Source: A. InteractBench Specification · p. 13

A.1. Task Card

• Each InteractBench task includes a structured task card that specifies the interface required by the offline evaluation harness.The task-card design makes the benchmark interface explicit and auditable.Source: §A.1 · p. 13
• Table 5 documents the fields included in InteractBench task cards.Source: Table 5 · p. 13

B.1. Dataset Statistics

• InteractBench tasks come from Codeforces, AtCoder, IOI, and ICPC, with task counts broken down by source and difficulty in Table 6.The sources include online judges and competition series.Source: §B.1 · p. 13, Table 6 · p. 14
• Category counts are stratified by difficulty in Table 7, and multi-label assignment allows one task to contribute to multiple categories.The appendix defines categories for label assignment and category-level analyses.Source: §B.1 · p. 13, Table 7 · p. 14
• The category definitions cover Graph, Search, and Greedy tasks involving hidden structures, feasible-region reduction, or extendable committed constructions.These categories emphasize relational queries, maintained feasible sets, and invariants preventing dead ends.Source: §B.1 · p. 13
• Additional categories are Bit, Data Structures, Math, and Game, covering low-bandwidth decoding, online state maintenance, structured deduction, and worst-case reply planning.The definitions highlight bookkeeping, uniqueness and degeneracy handling, and minimax-style reasoning over knowledge states.Source: §B.1 · p. 14

B.2. Model Card

• The model card groups evaluated models by provider and access mode, listing publicly specified parameter counts and official hyperlinks when available.Table 8 presents the model-card information.Source: Table 8 · p. 14

B.3. Evaluation Prompt for Code Generation

• Models use a standardized minimal zero-shot wrapper instantiated with the target language, reducing prompt-engineering confounds while enforcing protocol-compliant interactive I/O.Generated programs are compiled and executed against task interactors, with correctness determined from the interactive transcript.Source: §B.3 · p. 15
• The prompt requires a complete single-file program that deduces hidden parameters through the defined protocol and outputs only a language-tagged fenced code block.It also requires standard stdin/stdout entry points, flushing after every query, immediate termination on invalid responses, and no debug output.Source: §B.3 · p. 15

C. Supplementary Experiments

• The supplementary experiments section collects additional experiments intended to complement the main experiments in Section 4.No supplementary findings are specified in the supplied passage.Source: C. Supplementary Experiments · p. 15

C.1. Batch vs. Interactive Evaluation

• The batch comparison uses 150 matched problems, reveals hidden test data through standard input, and preserves the same final validation logic while removing query-based information acquisition.This isolates the difficulty of unrevealed information from the batch algorithmic core.Source: §C.1 · p. 15
• Revealing hidden test data improves pass@1 for every configuration, reduces protocol and budget failures, and exposes a Qwen3-14B rank reversal between batch and interactive evaluation.Qwen3-14B-NonThinking scores 0.180 vs. 0.093 in batch evaluation, but 0.011 vs. 0.056 interactively; wrong answers remain dominant among unsuccessful runs.Source: Table 9 · p. 15, §C.1 · p. 16

C.2. Resampling, Few-Shot Prompting, and Iterative Refinement

• Iterative refinement improves over one attempt for stronger models but remains below resampling and few-shot prompting in final scores.Each failed attempt returns execution feedback to the model, which revises its program for the next round; refine@k is cumulative solve rate within k rounds.Source: §C.2 · p. 16
• Few-shot prompting is competitive with independent resampling and slightly higher for most configurations, with the clearest gains for Qwen3-14B and Qwen3-14B-NonThinking.The comparison uses pass@5 for independent resampling and refine@5 for iterative refinement.Source: Table 10 · p. 16, §C.2 · p. 16
• Refinement reduces execution-level failures, including GPT-5.2 compilation errors from 0.040 to 0.010 and protocol errors from 0.139 to 0.038 between rounds 1 and 5.These values are failure-type fractions among still-unsolved executions.Source: Table 11 · p. 16, §C.2 · p. 16
• Wrong answers remain the dominant residual failure type across all five configurations, indicating that feedback repairs execution issues but does not resolve the underlying partially observed interactive reasoning problem.The failure composition is reported across rounds 1, 3, and 5.Source: §C.2 · pp. 16–17

Method

• The diagnostic partitions problems at cut year Y into pre-release P<Y and post-release P≥Y sets, then reports mean pass@1 on P≥Y as Y varies.A sharp change could indicate contamination concentrated in a temporal window, whereas smoother curves suggest no single sharp temporal boundary drives performance.Source: §C.3 · p. 17

Results

• Gradual post-set mean pass@1 curves across release-year cutoffs show no cliff-like discontinuity for six representative model configurations.The configurations are Claude-Opus-4.5, GPT-5.2, Gemini-3-Pro-Preview, DeepSeek-V3.2-Thinking, Qwen3-32B, and Qwen3-30B-A3B-Thinking.Source: §C.3 · p. 17

C.4. Multi-language and I/O Template Evaluation

• Template effects are mixed: DeepSeek-V3.2 on Python rises from 0.060 to 0.131 with the template, whereas DeepSeek-V3.2-Thinking falls from 0.309 to 0.289.The evaluation covers C++, Python, Java, and Go with packaged local interactors and compares zero-shot prompts against prompts containing the interactive I/O template.Source: §C.4 · pp. 17–18, Table 12 · p. 18
• Templates mainly reduce low-level I/O failures, while wrong answers remain the main unsuccessful-run category after language-specific failures are separated.Python without templates more often produces idle or deadlocked runs; Java and Go introduce more compile failures, especially for weaker models.Source: §C.4 · p. 18
• C++ shows the smallest template-versus-no-template difference, supporting its use without templates as the default evaluation configuration.The main experiments use C++ without templates, and the section checks whether model ordering persists across other language and template settings.Source: §C.4 · pp. 17–18

D.1. Qualitative Evidence for Key Findings

• Resampling can rescue runs whose plausible high-level strategy fails because of small protocol or robustness errors, explaining why pass@1 can lag noticeably behind pass@5.The qualitative examples are illustrative rather than exhaustive and focus on alternative approaches and avoidance of implementation brittleness.Source: §D.1 · pp. 18–19
• A DeepSeek-V3.2 sample fails before any valid query because an uninitialized index pair causes a format error, while a fallback to valid indices avoids this protocol failure.The failing harness output reports wrong output format, invalid query indices, zero queries, and a query limit of two.Source: §D.1 · pp. 18–19
• GPT-OSS-120B exhibits unusually frequent IDLE failures, often issuing no query because generations misinterpret interactive tasks as offline input-output problems.This contrasts with frontier models that more often execute the protocol and fail as WA.Source: §D.1 · p. 19
• On cf2181C, reading an unavailable edge list blocks immediately because the true interactive protocol provides no such input.The saved program begins by reading m edges, so it never enters a valid query loop.Source: §D.1 · p. 19
• On cf1979F, brittle sentinel handling can cause a late IDLE after thousands of queries when the solver reads two integers before checking a single -1 termination token.If the judge emits only -1 after an invalid query or budget violation, the second read can block.Source: §D.1 · p. 19

• Each case study combines the interaction protocol, a model-generated code excerpt, harness evidence, and a diagnosis of the underlying failure mode.The taxonomy examples are designed to distinguish compilation, interaction, resource, and algorithmic failures.Source: §D.2 · p. 19
• Compilation and protocol failures arise from flushing the input stream, issuing a query during the answer phase, or omitting a required answer flush.These cases produce CE, PE, and IDLE: invalid stream operations fail compilation, phase mixing violates the output grammar, and missing flushes deadlock the interactor.Source: §D.2 · pp. 20–22
• Query and stream-management failures include exhaustive enumeration beyond the query budget and skipping judge feedback, causing QLE or RE through input desynchronization.Enumerating all action strings scales as k · 2^n, while failing to read a Correct/Wrong verdict shifts the input stream and can crash the interactor.Source: §D.2 · pp. 23–24
• Uncontrolled randomized preprocessing can consume the entire time budget, while empty partitions leave recursion without a base case, causing expansion, stack growth, and eventual crashes.Such runs are classified as MLE when peak RSS exceeds the task’s memory limit, and the interactor may subsequently report WA after premature termination.Source: §D.2 · pp. 24–25
• A protocol-compliant submission can still receive WA when its comparisons do not uniquely certify the true top-K set, revealing a reasoning-level failure.The example stays within the query budget and follows the protocol but outputs an incorrect applicant set because its selection logic merely guesses a plausible result.Source: §D.2 · p. 26

D.3. End-to-End Task Showcase: cf2036G

• cf2036G is packaged with a protocol, generator, local interactor, hidden offline cases, and recorded evaluation artifacts.The showcase uses a non-adaptive Codeforces task whose hidden input is fixed before interaction.Source: §D.3 · pp. 26, 31
• The task hides three distinct values among duplicated values, requires XOR range queries, allows at most 150 queries per test case, and accepts the three values in any order.Queries are flushed and answered interactively; receiving -1 requires solver termination.Source: §D.3 · p. 26
• The generator combines fixed edge cases, power-of-two and XOR-zero adversarial patterns, random stress cases, and values spread across the range.It also supports an adaptive schema in which only n is provided and the interactor chooses a,b,c adversarially within [1..n].Source: §D.3 · pp. 27–29
• The local interactor enforces query limits, validates query bounds and answer values, returns XOR responses, compares sorted answers, and logs query metrics.It sets a 150-per-test-case limit, applies a hard cap, and terminates with verdicts for protocol or execution failures.Source: §D.3 · pp. 30–31
• A representative model-generated passing solver issues flushed XOR queries, exits on -1, uses binary searches to locate stolen values, and recovers s2 from the prefix before s3.Source: §D.3 · pp. 32–33
• 100-case evaluation shows the passing cf2036G solver succeeds on every case, whereas the failure solver gets WA on cases/003.in after exhausting its query budget.The failure stems from sparse xor i i probing that misses two large target values around 258, producing an incorrect triple.Source: §D.3 · p. 36

D.4. Adaptive Interactor Example: cf2096G

• cf2096G requires exactly the minimal query count f(n), with queries encoding L/R/N replies and one ignored position before the solver guesses the hidden x.The minimum is implemented as the smallest q satisfying 3^(q-1) >= n.Source: §D.4 · pp. 37–38
• The adaptive interactor may choose a consistent hidden value and ignored query after observing all queries, so the solver must uniquely determine x under history-dependent choices.The offline harness provides only public parameters, while its deterministic worst-case policy maximizes remaining candidate ambiguity with fixed tie-breaking.Source: §D.4 · p. 37
• The adaptive interactor validates the submitted final answer’s range and rejects interactions that do not uniquely determine an answer.It also reports successful validation only after these checks pass.Source: §D.4 · p. 40

D.5. Mutation-based Stress Tests

• Stress testing generates 15 mutated submissions per task, divided into wrong-answer protocol-valid cases and protocol- or budget-invalid cases.The two sets separately test semantic checking and consistent enforcement of interaction rules, parsing, accounting, and termination.Source: §D.5 · p. 40
• Protocol-valid mutations perturb state updates, stopping behavior, feedback interpretation, query strategy, or final outputs while preserving well-formed interaction.These mutations target failures such as broken invariants, incomplete querying, misread responses, and invalid uniquely checkable answers.Source: §D.5 · pp. 40–41
• Invalid mutations target parameter boundaries, malformed commands, one-step budget overruns, and output after interactor termination.Mutations derive primarily from verified accepted solutions, with rejected submissions optionally added for failure-mode diversity; every stress case must be rejected, and mistaken acceptance triggers refinement and human-subset revalidation.Source: §D.5 · pp. 40–41

D.6. Expert Intervention Case Study

• For n = 4, ⟨1, 1, 1, 2⟩ and ⟨1, 1, 1, 4⟩ yield identical replies to every query, proving that ! -1 can be required even after exhausting all four triples.A second example shows that non-zero replies also do not establish uniqueness: ⟨1, 2, 3, 4⟩ and ⟨1, 2, 4, 3⟩ are indistinguishable.Source: §D.6 · p. 42
• Expert review corrected cf1847E’s interactor so ! -1 is accepted exactly when multiple hidden arrays produce identical replies, rather than when any non-degenerate triangle exists.The pre-fix heuristic incorrectly rejected ambiguous arrays; the revision checks alternative arrays over the small value domain and preserves rejection for uniquely identifiable arrays.Source: §D.6 · pp. 42–43

D.7. Interactive I/O Templates

• The benchmark supplies language-specific interactive I/O templates emphasizing immediate flushing, reply handling, graceful termination on EOF or invalid replies, and protocol-only stdout.Templates are provided for C++, Python, Java, and Go.Source: §D.7 · pp. 44–46

D.8. Prompt Templates for the Propose–Validate–Adjudicate Loop

• The construction prompts define proposer and adjudicator roles for generators, interactors, mutated submissions, and candidate selection within the propose–validate–adjudicate loop.Generator instructions require deterministic seeded testlib cases, corner and adversarial coverage, and stress near worst-case query efficiency; interactor prompts specify a non-adaptive C++17 judge execution model.Source: §D.8 · pp. 47–48
• The interactor templates enforce fixed hidden inputs, explicit protocol communication, query-budget accounting, robust parsing and flushing, and logging of queries and query_limit on termination paths.The harness validates budget adherence from logged counters, while hard caps prevent infinite loops or output spam and early logging mitigates stalled-process failures.Source: §D.8 · pp. 48–49
• The propose–validate–adjudicate loop uses strict prompts to generate minimally mutated submissions, select compatible generators and interactors, and choose solver–interactor pairs using correctness, protocol compliance, and query efficiency.Mutations target either subtle semantic errors that preserve valid interaction or exactly one protocol/budget violation; adjudicators prioritize passing standard checks, rejecting representative incorrect solvers, and selecting tournament-passing combinations.Source: §D.8 · pp. 50–52

D.9. Model Performance by Category and Difficulty

• The benchmark reports each model’s mean pass@1 and pass@5 by category and Easy/Medium/Hard difficulty tier, alongside overall results.These per-model tables complement Table 3; the supplied passages identify the tables’ scopes but do not provide their cell values.Source: §D.9 · p. 52