PaperlayerView exact revision on arXiv

Source-linked AI summary

Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

Han Li, Jinyu Tian, Rili Feng, Yuqiao Du, Chong Zheng, Chenyu Wang, Chenchen Liu, Shihao Li, Xinping Lei, Yifan Yao, Weihao Xie, Letian Zhu, Jiaheng Liu

arXiv:2605.15301v1cs.AI

TL;DR

Hard competitive programming remains difficult for LLMs, while existing multi-agent systems discard experience across problems. Solvita adds trainable graph-structured knowledge networks to a closed-loop multi-agent framework, raising CodeContests pass@1 from 40.0% to 82.4% with GPT-5.4.

  • Problem

    Existing coding pipelines are stateless, discarding experience from prior mistakes, while retrieval augmentation relies on static similarity lookup rather than learned reasoning improvement.

  • Method

    Solvita couples Planner, Solver, Oracle, and Hacker agents with trainable graph-structured knowledge networks updated by reinforcement signals from execution and adversarial testing.

  • Results

    82.4% pass@1 accuracy on CodeContests with GPT-5.4, up from 40.0% for single-pass generation, nearly doubling the baseline.

  • Takeaways & Limitations

    Solvita enables frozen LLMs to accumulate algorithmic intuition, strategy routing, and debugging experience across competitive-programming tasks.

  • Takeaways & Limitations

    The agentic loop has a substantial cold-start cost, requiring about 5,000 training problems before per-problem costs are amortized into accuracy gains.

Abstract

from arXiv · show

Large language models (LLMs) still struggle with the rigorous reasoning demands of hard competitive programming. While recent multi-agent frameworks attempt to bridge this reliability gap, they remain fundamentally stateless: they rely on static retrieval and discard the valuable problem-solving and debugging experience gained from previous tasks. To address this, we present Solvita, an agentic evolution framework that enables continuous learning without requiring weight updates to the underlying LLM. Solvita reorganizes problem-solving into a closed-loop system of strategy selection, program synthesis, certified supervision, and targeted hacking, executed by four specialized agents: Planner, Solver, Oracle, and Hacker. Crucially, each agent is paired with a trainable, graph-structured knowledge network. As the system operates, outcome signals, such as pass/fail verdicts, test certification quality, and adversarial vulnerabilities discovered by the Hacker, are recast as reinforcement learning updates to these network weights. This allows the agents to dynamically route future queries based on past successes and failures, effectively accumulating transferable reasoning experience over time. Evaluated across CodeContests, APPS, AetherCode, and live Codeforces rounds, Solvita establishes a new state-of-the-art among code-generation agents, outperforming existing multi-agent pipelines and nearly doubling the accuracy of single-pass baselines.

1 Nanjing University 2 Tsinghua University 3 Independent Researcher · 1 Introduction

Solvita addresses the statelessness of current LLM coding pipelines with a closed-loop, four-agent framework whose trainable knowledge networks accumulate problem-solving and debugging experience without updating the frozen LLM. It reports state-of-the-art performance, raising CodeContests pass@1 accuracy from 40.0% (single-pass) to 82.4% with GPT-5.4.

  • 1 Introduction: Competitive programming tests LLMs’ ability to formalize specifications, select strategies, and rigorously verify efficient programs.These benchmarks provide a controlled setting for evaluating structured reasoning.
  • 1 Introduction: Single-shot generation conflates understanding, planning, coding, and verification, while newer pipelines remain stateless and discard experience from prior mistakes.AlphaCodium and MapCoder introduce hierarchical planning and iterative refinement but still solve each new problem from scratch.
  • 1 Introduction: Solvita uses Planner, Solver, Oracle, and Hacker agents in a dynamic closed loop for strategy selection, program synthesis, repair, supervision, and adversarial testing.Failure signals propagate across agents, enabling collaboration and cross-correction.
  • 1 Nanjing University 2 Tsinghua University 3 Independent Researcher: The paper’s listed affiliations are Nanjing University, Tsinghua University, and an Independent Researcher.These affiliations appear in the section heading.
  • 1 Introduction: Each agent’s trainable graph-structured knowledge network links queries, metacognitive analyses, and reusable skills, with edge weights updated by outcomes for learned strategy routing.This replaces passive semantic retrieval with memory that expands where the agent struggles.
  • 1 Introduction: Oracle certification quality and Hacker-discovered vulnerabilities become reinforcement learning signals while the underlying LLM remains entirely frozen.The knowledge networks learn to route problems toward suitable strategies and failure-prevention tactics.
  • 1 Introduction: 82.4% pass@1 accuracy is achieved on CodeContests with GPT-5.4, up from 40.0% (single-pass).The result nearly doubles the cold-start baseline while maintaining a similar token-consumption footprint to existing multi-agent pipelines.

2 Data

Solvita builds its cold-start corpus from heterogeneous competitive-programming platforms, unifies the data into a canonical JSON schema, and applies four sequential filters. The process reduces 30,018 starting problems to a final corpus of 8,017 problems.

  • Collection: The corpus is collected directly from Codeforces, AtCoder, Aizu Online Judge, and smaller platforms, with public datasets used to recover missing editorials and verdict labels.The collection unit is the platform rather than any single pre-packaged dataset.
  • Schema unification: All artifacts are normalized into a unified JSON schema containing problem statements, typed variables and constraints, tests, editorials, submissions, verdicts, execution times, and algorithmic tags.The schema uses canonical fields and a controlled tag vocabulary.
  • Filtering: The corpus starts with 30,018 problems and undergoes four sequential filters for completeness, tag balance, redundancy, and difficulty.Tag load balancing precedes embedding-based deduplication, while difficulty pruning is applied last against the post-deduplication distribution.
  • Filtering: 8,017 problems remain in the final corpus after the filtering pipeline.The retained corpus follows the sequential filtering process applied to the unified collection.

3 Solvita

Solvita closes competitive-programming problem solving into a four-agent solve–certify–attack loop whose failure signals update trainable knowledge networks. Its agents formalize and plan problems, synthesize and patch solutions, certify supervision, and search for adversarial vulnerabilities using complementary reusable strategies.

  • Architecture: Solvita couples Planner, Solver, Oracle, and Hacker in one closed loop, propagating failure signals across all four agents’ knowledge networks.The Planner selects paradigms, the Solver performs patch-based repair, the Oracle constructs certified tests, and the Hacker launches adversarial attacks.
  • Planner: The Planner strips narrative context into a formal specification, then proposes algorithmic tags, an implementation sketch, and a complexity estimate.After failure, its verdict classification guides replanning; tag rewards teach the network which formalized structures admit particular paradigms.
  • Solver: The Solver preserves passing behavior by applying search-and-replace patches that are accepted only when all regression tests continue to pass.Its heterogeneous graph stores problem descriptions, solution decompositions, contrastive analyses, and annotated C++ skills; contrastive REINFORCE trains the network against the pass-rate difference between augmented and bare-LLM runs.
  • Oracle and Hacker: Oracle strategies favor reliable supervision through DP/Search and Enumeration, whereas Hacker strategies target latent bugs through routes including semantic, stress, and antihash attacks.Both agents factorize the shared algorithm space into distinct reusable strategy families, with the Hacker searching for vulnerabilities that survive Oracle certification.
  • Oracle: The Oracle generates and independently certifies supervision, producing a certification ratio ρ = Ncert/Ntarget ∈[0, 1] and discarding artifacts that fail its gate.Its strategy-family bandit reward combines partial credit, a full-certification judge bonus, and failure penalties.

4 Experiments

Solvita is evaluated across three competitive-programming benchmarks and live Codeforces rounds using multiple frontier backbones. It achieves broad performance gains, while analyses attribute them to trained agentic components, adversarial validation, and efficient patch-based repair.

  • Main results: Solvita attains the best pass@1 in 14 of 15 backbone–benchmark cells, with the sole exception being AetherCode under Claude Opus 4.6.Open-source frameworks trail Solvita on every cell, and the gap widens on the harder AetherCode benchmark.
  • Cost and failure profile: Solvita remains in the token-consumption band of open-source agent frameworks while reducing algorithmic, specification-level, complexity, memory, and runtime failures versus single-pass generation.The cost and failure analysis indicates that gains are distributed across multiple failure categories rather than concentrated in one easy category.
  • Component ablations: Trained knowledge networks distinguish the Full system from stateless multi-agent and Single-pass baselines across the 5,318-problem training trajectory.Single-pass is monolithic, without training is stateless multi-agent, and Full uses all four agents’ fully trained networks.
  • Solver inner loop: Under matched Nmax = 8 iteration budgets, patch-based repair is compared with full regeneration using identical retrieval and decoding.Both strategies save tokens under the shared reference, but patch repair saves substantially more because later iterations modify drafts rather than regenerating full solutions.
  • Diagnostic modules: The Oracle preserves correct solutions well but misses subtle bugs, whereas the Hacker detects more wrong solutions and exposes accepted-solution disagreements requiring stronger tests.Figure 6a evaluates wrong-solution detection, correct-solution preservation, and confirmed stronger-test rates across three backbones.
  • Codeforces evaluation: All Solvita variants reach the Legendary Grandmaster band (≥3000) within roughly a dozen Codeforces rounds, while bare backbones plateau in the high Grandmaster band.The evaluation uses K = 12 post-cutoff contests, totaling 76 problems, under uninterrupted official-time-limit sessions without post-contest corrections.

5 Related Work

Prior work advances code generation through structured multi-agent and self-improving pipelines, while memory-augmented agents and adversarial validation address experience reuse and program reliability. However, flat retrieval and limited role specialization remain reported bottlenecks.

  • Code generation and self-improving agents: Code generation has progressed from single-shot synthesis to structured multi-agent pipelines with planning, retrieval, role separation, decomposition, reranking, and self-debugging.These pipelines also include repository-level interfaces, self-repair, general-purpose orchestration, and debate.
  • Code generation and self-improving agents: Self-improving agents update prompts, rationales, or pipelines through execution feedback.
  • Memory and adversarial validation: Memory-augmented agents store and retrieve past experience through skill libraries, episodic reflection, virtual memory, or graph-structured reasoning.Flat retrieval and the absence of role specialization remain reported bottlenecks.
  • Memory and adversarial validation: Systematic test generation spans fuzzing, equivalence modulo inputs, coverage-guided mutation, LLM-based fuzzing, certified validators, and dedicated hacking pipelines.The passage frames these methods as parallel approaches to adversarial validation.

6 Conclusion · Appendix

Solvita enables continuous, experience-driven learning for frozen LLMs by coupling four specialized agents with dynamic graph-structured knowledge networks and updating them from execution and adversarial-testing signals. The conclusion identifies cold-start cost, bounded Hacker coverage, and patch-repair drift as limitations, and proposes warm-starting and transferring the decomposition to other verifiable reasoning domains.

  • 6 Conclusion: Solvita couples Planner, Solver, Oracle, and Hacker agents with graph-structured knowledge networks for continuous learning without updating the underlying LLM.The framework uses execution verdicts and adversarial testing as REINFORCE updates, accumulating algorithmic intuition, strategy routing, and debugging experience over time.
  • 6 Conclusion: About 5,000 training problems are needed before knowledge-network training costs are amortized into per-problem accuracy gains.The agentic loop is more expensive per problem than direct generation during cold start.
  • 6 Conclusion: Hacker anti-hash and lattice-based attacks are bounded by the backbone’s reasoning horizon, leaving number-theoretic invariants and geometric tolerance bugs under-covered.The limitation concerns heavily mathflavored failure modes.
  • 6 Conclusion: On globally flawed candidates, Solver patch repair can mislabel systemic flaws as localized and accumulate inconsistent edits before the iteration budget is exhausted.Section 4.3’s regression-rate signal catches this drift only post hoc.
  • 6 Conclusion: Warm-starting knowledge networks from editorials, accepted submissions, and debugging traces could shrink the cold-start window.The conclusion identifies open-source experience corpora as a source for initialization.
  • 6 Conclusion: The four-agent decomposition could transfer to formal theorem proving, with the Oracle as a proof checker and the Hacker searching for counter-models.The passage also introduces mathematical olympiad problems as another prospective verifiable reasoning domain.

A Data Pipeline Configuration · B Contextual Bandit Policy Details · C Oracle Reward: Failure-Path Details

The appendix specifies the filtering pipeline, contextual-bandit configuration, and Oracle failure-path rewards. Together, these details define normalized difficulty and tag balancing, persistent agent adaptation, and informative penalties when certification fails.

  • A Data Pipeline Configuration: Difficulty is normalized across Codeforces, AtCoder, LeetCode, and CodeContests onto a shared Codeforces rating scale for filtering.The normalized tier supports the difficulty signal and per-tag floor.
  • A Data Pipeline Configuration: Tag balancing caps each tag at Cmax = 2300 surviving problems and uniformly subsamples tags above the cap.Subsampling preserves the difficulty distribution and keeps per-tag counts within a constant factor of the smallest surviving tag.
  • A Data Pipeline Configuration: Embedding-based deduplication uses text-embedding-3-large within tag buckets with cosine-similarity threshold δ = 0.93.The threshold was chosen to maximize precision on a manually labeled validation set of 500 candidate pairs.
  • B Contextual Bandit Policy Details: Each knowledge network uses α = 0.01, rewards r ∈[−1, 1], and a +0.05 tag-overlap bonus, with feature keys encoding state, prior failures, and problem tags.Parameters are persisted in JSON using atomic file-locking writes.
  • C Oracle Reward: Failure-Path Details: The Oracle represents each instance as x = (d, c, p, κ) and produces y = (F, f ∗, T, V, A, m) through bandit-based family selection and an acceptance gate.The artifact contains candidate and selected families, certified tests, verifier provenance, acceptance, and metadata.
  • C Oracle Reward: Failure-Path Details: Failure-path penalties are −1.0 for crashes or severe errors with Ncert = 0, −0.7 for self-check failures, and −0.6 when tests are invalid or state is unready.These penalties preserve an informative bandit signal when no certified test survives.
  • C Oracle Reward: Failure-Path Details: For full certification, ρ = 1, the verification bonus is +1.0, −0.2, or −0.5 according to independent-judge agreement, partial agreement, or contradiction.The reward distinguishes complete agreement from partial or conflicting verification.

D Hacker Reward: Degenerate-Round Details · E Prompt Details

The Hacker reward combines validity, breakage, severity, and compilation-failure signals, with a bounded correction for degenerate rounds. Prompt templates enforce structured agent outputs and guide planning, skill selection, code generation, verification, and repair through explicit contracts and resource checks.

  • D Hacker Reward: Degenerate-Round Details: On degenerate Gen_Failed rounds with no valid verdict, the reward is r = −0.6 −min(0.3, 0.1 c), replacing the default composition.This correction supplies a usable gradient despite repeated generator failures.
  • D Hacker Reward: Degenerate-Round Details: The Hacker reward weights valid-input rate, break rate, and average severity while subtracting a compile-failure penalty, prioritizing candidate-breaking inputs without rewarding invalid tests.The break component receives most of the budget, while the validity component prevents routing toward high-severity inputs rejected by the validator.
  • D Hacker Reward: Degenerate-Round Details: When |Vvalid| = 0, the fixed −0.6 baseline and linear compile penalty keep the bandit signal bounded and nonzero, preventing route-weight updates from freezing.The severity term averages fixed per-verdict weights over Vbreak, calibrated once on a 200-problem development split and held constant across reported runs.
  • E Prompt Details: All agent prompts are stored in config/prompt_template.yaml and rendered by substituting <KEY> placeholders, while shared boilerplate preserves output schemas and implementation reminders.The conventions require strict JSON without fences or outside commentary, escaped control characters, and C++17 with explicit headers and fast I/O.
  • E.1 Planner: The Planner converts each problem into a self-contained canonical form with coarse and optional fine-grained algorithmic tags, confidence, rationale, warnings, constraints, and edge cases.The output is deliberately short and JSON-bound so downstream agents can parse it deterministically.
  • E.2 Solver: skill selection: The Solver selects a small number of exactly copied skill identifiers and emits an acyclic subproblem DAG using the problem summary, graph context, and candidate relevance ranks.Strict JSON and exact identifier copying are required because the result directly indexes the skill table.
  • E.3 Solver: code generation and patch repair: The Solver alternates initial full-program generation with localized patch repair for up to Nmax = 8 iterations, routing modes through a patch-decision prompt and requiring resource-feasibility audits.The generation workflow requires design before code, sample-based verification, an implementation plan, explicit C++17 constraints, and time, memory, and edge-case checks.
  • E.3 Solver: code generation and patch repair: The Solver’s reasoning prompt requires executable verification before declaring progress, including brute-force comparison on small random cases and worst-case C++ runtime checks.The recommended workflow declares PROCEED only after agreement on at least 5 random tests and completion within the runtime budget.

E.4 Solver: failure analysis … F.3 Knowledge-network defaults

The paper specifies a structured Solver failure-analysis loop and fixed experimental settings spanning backbones, budgets, and trainable knowledge-network defaults. These configurations standardize comparisons while enabling categorized debugging feedback and persistent policy memory.

  • E.4 Solver: failure analysis: The Solver step-traces the simplest failing case, identifies the root cause, proposes concrete code-level fixes, and feeds categorical error patterns back into bandit features.Supported categories include overflow, off-by-one, wrong formula, missing edge case, and TLE.
  • F Experimental Configuration: The appendix records the exact backbone, infrastructure, and budget settings used across all reported experiments.These settings define the experimental configuration rather than introducing a separate algorithmic component.
  • F.1 Backbones and inference: All five backbones use a unified Azure OpenAI–compatible gateway, with each comparison row sharing one deployment to isolate agent-framework effects.The listed models are GPT-5.4, Claude Opus 4.6, Qwen3.6, DeepSeek V4 Pro, and Grok.
  • F.1 Backbones and inference: Decoding uses temperature = 0.1 and max_tokens = 16,384 by default, increasing to 64,000 for long-context backbones.Runtime retrieval uses text-embedding-3-small with a 32,768-entry LRU cache and up to 5 HTTP retries.
  • F.2 Pipeline budgets: The closed-loop pipeline uses fixed iteration budgets so per-problem cost remains comparable across baselines.This budget discipline applies to the reported pipeline comparisons.
  • F.2 Pipeline budgets: 8 patch/regenerate rounds cap the Solver inner loop, while 3 Hacker rounds cap the outer loop and each Hacker round runs the full cascade.Oracle certification targets Ntarget certified tests and accepts when ρ ≥τ; rejected cases trigger retry with a different bandit-selected solver family.
  • F.3 Knowledge-network defaults: Each agent uses a contextual-bandit policy over a typed item store, with the Solver additionally carrying a Solver knowledge network.The default Solver network retrieves 4 similar Q nodes, samples 5 skills from a top-20 pool, and enforces 1–5 selected skills.
  • F.3 Knowledge-network defaults: The trainable memory injects 3 advice items per namespace, learns with α = 0.01, and auto-deprecates low-reward items after sufficient use.Items averaging below −0.3 after at least 20 uses are deprecated; persistence uses a SQLite-backed JSON store with atomic file-locking writes.

F.4 Datasets and benchmarks · F.5 Codeforces Rating Estimation Protocol

Solvita is evaluated on four competitive-programming benchmarks and Codeforces rounds, using fixed, uninterrupted contest sessions. Codeforces performance is converted into a contest-local rating estimate from official standings and aggregated across contests, rather than treated as an official account rating.

  • F.4 Datasets and benchmarks: The evaluation uses 165 CodeContests problems, 1,000 APPS problems, 400 AetherCode problems, recent Codeforces rounds, and an 8,017-problem Codeforces cold-start corpus.AetherCode combines novel algorithmic tasks with verified Oracle-generated test suites; Codeforces rounds are attempted under official time limits in single uninterrupted sessions.
  • F.5 Codeforces Rating Estimation Protocol: The rating estimate is not an official Codeforces account rating; it is a contest-local, human-comparable estimate based on accepted submissions, solved counts, penalties, ranks, and pre-contest human ratings.The procedure follows a rating-inversion view using the same public ingredients as Codeforces standings.
  • F.5 Codeforces Rating Estimation Protocol: The Codeforces protocol starts each agent/backbone pair from contest statements at time zero and forbids manual corrections, prompt edits, or post-window submissions.Each round uses its official duration and division setting, and a problem counts as solved only when the official judge returns Accepted.
  • F.5 Codeforces Rating Estimation Protocol: Agents are inserted into each contest’s retained human standings using solved problems, penalties, and deterministic last-accepted-submission tie-breaking.More solves rank higher, lower penalties rank higher among equal solve counts, and the final tie-breaker is used only when necessary.
  • F.5 Codeforces Rating Estimation Protocol: The contest-local rating is the latent Elo rating whose expected number of outperforming human participants matches the agent’s inserted rank.The estimate is obtained by solving the Elo-based inversion equation; the implementation uses binary search over [−500, 5000] until residual error is below 10−6.
  • F.5 Codeforces Rating Estimation Protocol: Each contest is treated independently, producing a low-variance contest-local estimate rather than an online Codeforces account history.This is appropriate because all compared agents participate in the same fixed contest set.
  • F.5 Codeforces Rating Estimation Protocol: Figure 6b plots the running mean of contest-local estimates over ordered contest prefixes, with uncertainty reported using across-contest standard error.Canonical Codeforces color/rating bands are applied only for interpretability and do not imply official Codeforces users.

G Knowledge-Network and Pipeline Implementation Details … G.5 Sandbox and judge resolution

The appendix specifies Solvita’s implementation as a closed-loop pipeline built on shared SQLite-backed knowledge storage, sparse feature-based retrieval, role-specific learning updates, and sandboxed judging. Failure events propagate across all four namespaces, while strict verdict resolution supplies the feedback signals for Oracle certification and Hacker break-rate metrics.

  • G Knowledge-Network and Pipeline Implementation Details: The implementation details cover four knowledge networks and their closed-loop control flow.These networks support the pipeline described in Section 3.
  • G.1 Shared item schema: All namespaces share an SQLite item table storing identifiers, role-specific payloads, searchable tags, rewards, usage, lifecycle flags, and timestamps.Reads use an in-memory index, while atomic file-locked writes protect concurrent benchmark workers.
  • G.2 Featurizer and bandit scoring: At inference, per-namespace featurizers encode FSM position, prior failure type, and problem-level tags into sparse feature keys.Examples include solve and hack states, verdict failures, and tags such as dp, graphs, and strings.
  • G.2 Featurizer and bandit scoring: Items are ranked by a bias plus feature-weight sum, receive a +0.05 matching-tag bonus, and are selected with ε-greedy top-k retrieval into <MEMORY_ADVICE>.The typical selection size is k = 3.
  • G.3 Solver knowledge network storage and dynamics: The Solver network persists graph nodes and learned w_qm and w_ms matrices, representing problems, analysis trajectories or contrastive solutions, and annotated skills.M nodes can encode either a function-block DAG trajectory or a correct–incorrect pair with an annotated divergence point.
  • G.4 Failure-event propagation: When the Hacker breaks a candidate, the failure is broadcast across namespaces to penalize the plan, create a Solver contrastive pair, record an Oracle generator hint, and boost the successful hack route.This mechanism transfers one discovery across all four role-specific knowledge networks.
  • G.5 Sandbox and judge resolution: Compilation and execution run in per-process sandboxes using g++ -std=c++17 -O2 with wall-time and memory limits.The sandbox wraps the compilation and execution process.
  • G.5 Sandbox and judge resolution: Judge resolution prioritizes a custom checker, certified-reference token comparison, then exact canned-output matching, producing verdicts for ρ(x, f) and g_break.These verdicts feed the Oracle certification ratio and Hacker break-rate metrics.

H Additional Ablations

The appendix defines diagnostic accounting and validates design choices through ablations of stronger-test confirmation, repair mode, skill selection, Oracle threshold, Hacker budget, and REINFORCE rewards. These analyses clarify how Solvita balances detection, preservation, computational cost, attack coverage, and learning stability.

  • Diagnostic metrics: Figure 6a evaluates diagnostic configurations on held-out candidates using TP, TN, NP, and NF outcomes against official verdicts.TP and TN are Solvita accepts; NP and NF are Solvita rejects, distinguished by official correctness.
  • Diagnostic metrics: Str. Rate counts only Solvita-rejects/official-accepts disagreement problems confirmed by accepted-solution cross-checking and manual validation as genuinely stronger tests.Validation excludes isolated candidate failures, invalid inputs, incorrect expected outputs, checker mistakes, and Oracle or certification errors.
  • Repair ablation: Both patch and regenerate use Nmax = 8 and the same failure-analysis prompt, while patch emits SEARCH/REPLACE blocks and regenerate rewrites the full program.Patch is the default Solver repair mode; the passage introduces regression-rate measurement but does not provide its result.
  • Oracle and Hacker ablations: The Oracle threshold sweep tests τ ∈{0.6, 0.75, 0.9, 1.0}, with default τ = 0.9 identified as the precision/recall trade-off knee.Lower thresholds admit more tests but alter the downstream detection–preservation trade-off.
  • Oracle and Hacker ablations: Hacker budgets {1, 2, 3, 5} show most break events in rounds 1–2, small but non-zero gains from round 3, and rare new bugs after round 4.The default max_hack_rounds = 3 preserves an anti-hash attack opportunity while avoiding wasted later budget.
  • Learning ablation: Replacing contrastive ∆R = Rwith − Rwithout with absolute Rwith removes variance reduction and slows convergence across all three Tab. 2 checkpoints.The contrastive update is described as making the Solver knowledge network the dominant component in Tab. 2.

I Failure Cases

Evaluation exposed five representative failure modes that shaped Solvita’s design and prompting: misleading cold-start retrieval, Oracle false certification, limited Hacker coverage, and patch-repair drift. These failures arise from semantic misrouting, shared implementation bugs, insufficient mathematical reasoning, and misclassification of global flaws.

  • Evaluation failure cases: These failure categories were observed during evaluation and informed design choices in Sections 3.5–3.6 and prompts in Appendix E.The documented categories include cold-start retrieval misfires, Oracle false certification, Hacker scope limitations, and patch repair drift on global flaws.
  • Cold-start retrieval misfires: Cold-start retrieval can select structurally similar but semantically misleading Solver skills, biasing the initial draft toward the wrong paradigm.The selection LLM may return an empty list, but does not always do so; contrastive REINFORCE is used as mitigation.
  • Oracle false certification: The Oracle occasionally false-certifies subtly buggy reference solutions, causing certified tests and Solver rewards to inherit the same bug.The Hacker catches most cases, but pathological agreements between independently buggy implementations remain a residual failure mode.
  • Hacker scope limitations: The Hacker frequently misses bugs in number-theory and combinatorial-identity problems because the Code Analyst cannot identify deep mathematical bug classes.Its reward distribution is bimodal, with high break-ratio for implementation-level bugs and near-zero performance on math-heavy problems.
  • Patch repair drift on global flaws: Patch repair can misclassify global flaws as localized, producing accumulating state inconsistencies and exhausting max_iterations without convergence.The system currently uses regression-rate signals to identify these runs after the fact.
Loading 2605.15301v1…