PaperlayerView exact revision on arXiv

Source-linked AI summary

A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever

Sietse Schelpe

arXiv:2607.23806v1cs.CLcs.AIcs.IRcs.LGcs.PF

TL;DR

Retraining language models is costly, opaque, and non-repeatable, limiting their suitability for reproducible, auditable work. This paper freezes the model and adds persistent memory containing independently verified solutions, enabling subsequent instances to be answered deterministically without generation. The system reports measured verified reuse, including correct answers to compound problems never deposited in memory.

  • Problem

    Retraining language models is costly, opaque, and non-repeatable, creating a gap for reproducible and auditable answers.

  • Method

    The system freezes model weights and stores independently verified parameterized methods in persistent memory for deterministic reuse on new instances.

  • Results

    Verified memory reuse produces correct answers to compound problems whose solutions were never deposited, while subsequent family instances require zero generation tokens and return bit-exact outputs.

  • Takeaways & Limitations

    The measurements support a model–memory contract in which verified knowledge can be reused deterministically rather than regenerated for each instance.

  • Takeaways & Limitations

    Behavior beyond the tested upper bound of simultaneous items remains unmeasured.

Abstract

from arXiv · show

Improving a language model today means retraining it: enormous compute, a new opaque model each cycle, non-deterministic output. We take the opposite path: the model stays frozen, and a persistent memory of verified solutions grows beside it. Once a problem family is solved and has passed an independent verification step that never consults the answer key, every new instance of that family is answered at zero generation tokens, bit-exact, deterministically. Across 180 fresh instances spanning nine problem families, four architectures from four vendors - dense and mixture-of-experts - each score 180/180 at zero generation tokens per answer: execution-bound capability decoupled from parameter scaling. A negative control attributes the capability fully to the memory: emptied, it solves nothing. The same verify-before-store contract holds for open-ended reasoning: 88/88 consistency-gated acceptances across all four models, machine-checked formal proof, and reasoning-method transfer at 77/80. Memory selection takes 1.4 microseconds; a full reuse completes in 6-23 ms at 36 mWh. Approximate similarity retrieval selects the wrong item 94.3% of the time on a 4,500-item verified store where exact addressing makes zero errors. The store also serves as working context at a scale no shipped engine matches: a 6,000,000-token movable window on a single 46 GB GPU at flat memory, where vLLM stops at 30,399 tokens and SGLang silently truncates past 32,000. On published benchmarks, frontier models remain far ahead of any 12B at raw from-scratch reasoning; on everything this system has solved and verified, the comparison inverts: a frontier API call pays a fresh generation pass on every query, forever, while verified reuse costs zero tokens and returns the identical bits every time. A public testbench with free, rate-limited access accompanies this report: https://corbenic-galahad-bench.hf.space

1. Introduction

The paper introduces Galahad, which keeps the language model frozen while growing a persistent memory of independently verified solutions. Once a problem family is stored, subsequent instances are answered from memory with zero generation tokens and bit-exact, deterministic output.

  • Motivation: Retraining capability is costly, opaque, and unrepeatable, requiring GPU-months and re-auditing behavioral changes after each cycle.The same question does not reliably produce the same answer, undermining audit trails and reproducibility.
  • Method: Galahad freezes the model permanently, with no fine-tuning or weight changes, and grows capability through persistent solution memory.Its integrity layer, Merlin, stores only solutions that pass independent verification without seeing the benchmark answer key.
  • Contribution: Once a problem family is solved and stored, every subsequent instance is answered from memory at zero generation tokens with bit-exact, deterministic output.The contract persists for as long as the memory exists.

2. Measured results … 2.4 Stored knowledge combines into new answers

Verified memory delivers exact, zero-generation-token reuse across fresh problem instances, remains effective across four model architectures, and attributes capability to stored knowledge rather than frozen weights. The memory also supports independently verified composition of stored methods into new answers, with measured energy costs and clear failure under ablation.

  • 2.1 Core result: verified reuse at zero generation cost: 180/180 fresh instances across nine problem families are solved at 0 generation tokens, with identical outcomes across four model architectures.The instances use new parameter values within known logical families, requiring fresh computation from stored verified methods rather than exact-answer lookup.
  • 2.1 Core result: verified reuse at zero generation cost: N* = 17–34 queries when fully reasoned answers cost 500–1,000 tokens, after a one-time deposit of 16,579 generation tokens.Reuse thereafter costs 0 generation tokens, so the break-even gap widens beyond the repayment point.
  • 2.2 The capability is model-independent: 180/180 core verified reuse results are reproduced across four vendors’ dense and mixture-of-experts architectures.The memory operates as a layer beside the model, so the verified-knowledge investment survives model swaps.
  • 2.2b Energy accounting — measured, not estimated: 6.3–6.5 Wh per model powers the full 180-answer reuse battery, or about 36 mWh per verified answer.The figures are measured on actual runs; the one-time solve-and-verify campaign measured 81.1 Wh, while the composition run measured 0.09 Wh.
  • 2.3 The knowledge lives in the memory — negative control: 180/180 with full memory falls to solving nothing when memory is emptied, while frozen weights remain unchanged.This negative control attributes the measured capability to stored, verified knowledge rather than the model weights.
  • 2.3 The knowledge lives in the memory — negative control: 30/30 fresh instances of an externally authored, independently verified family are answered at 0 generation tokens beyond the frozen model’s own reach.The model emits nothing; the memory carries the capability.
  • 2.4 Stored knowledge combines into new answers: 0/5 compound answers remain when a required memory item is removed, versus 5/5 at 0 generation tokens with all three stored solutions.The system combines verified parts to compute new results, then stores the combined result as verified knowledge.

2.4b Merged knowledge in one working context — the nine-block result · 2.5 Beyond exact computation: verified open-ended reasoning

Verified knowledge can be merged into a single working context that supports source-attributed identification, planning, and cross-item computation, with nine simultaneous items as the tested upper bound. The same verify-before-store contract extends to open-ended reasoning, yielding 88/88 accepted-and-reused frameworks, 63 of 66 first-try passes, and zero generation tokens for 143 measured reuses.

  • 2.4b Merged knowledge in one working context — the nine-block result: 2/2 items and 3/3 items were correctly identified, while seven of nine items were reached and correctly named in about 1,300 tokens of merged context.At five items, the four items reached by the output window were each correctly identified; descriptions remained accurate as far as the fixed output-length cap allowed.
  • 2.4b Merged knowledge in one working context — the nine-block result: Nine simultaneous items is the tested upper bound; behavior beyond it is unmeasured.Without the system’s merge procedure, the model could not read the combined context at all, while the procedure enabled movable, mergeable knowledge units.
  • 2.4b Merged knowledge in one working context — the nine-block result: Six independently verified items merged into a 14,134-token working context produced a composite answer drawing on all six, with every element traceable to its source.No cross-source misattribution occurred, and the model correctly named three required items and combined their procedures for an unseen compound problem.
  • 2.4b Merged knowledge in one working context — the nine-block result: 44 N was the correct computed answer when the model combined physical constants from one stored item with a computational rule from another.Neither item contained the answer, demonstrating computation across stored items from different domains.
  • 2.5 Beyond exact computation: verified open-ended reasoning: The verify-before-store contract extends to open-ended domains through machine-checked proof, consistency-checked reasoning, and reasoning-method transfer.A formal checker accepted a proof and rejected 2+2=5; a consistency gate accepted self-consistent reasoning and rejected a self-contradicting variant.
  • 2.5 Beyond exact computation: verified open-ended reasoning: 88/88 reasoning frameworks were accepted and reused across all four models, while a verified method transferred to medical triage with the adaptation itself passing the consistency gate.The transfer result was measured across the four models, and the adaptation passed on the first attempt.
  • 2.5 Beyond exact computation: verified open-ended reasoning: 0 of 3 fabricated claims were admitted while 5 of 6 genuine claims were accepted by an answer-key-free declarative gate.The gate rejected a contradictory variant and held accompanying reasoning to three independent consistency invariances, all of which passed.
  • 2.5 Beyond exact computation: verified open-ended reasoning: 63 of 66 gated reasoning frameworks passed on the first try, and all 143 measured reuses thereafter cost zero generation tokens.Depositing the verified set was a one-time cost, while a full restore-and-reverify cycle completed in 6–23 ms median per model.

2.6 The memory scales safely — the Merlin integrity layer · 2.7 Reuse overhead is negligible — and cannot silently fail · 2.8 The frontier comparison — on public numbers only

The Merlin layer admits only independently verified, unique content and remains deterministic and recoverable under failure, while exact addressing avoids the silent substitutions of approximate matching. Against public frontier-model numbers, the frozen 12B is weaker at cold-start reasoning but advantageous on already verified work because reuse is bit-exact and has zero marginal generation cost.

  • 2.6 The memory scales safely — the Merlin integrity layer: 0 incorrect candidates were admitted on all tested inputs after layered verification, including 150 boundary cases per candidate.The gate never consults the answer key and rejected a subtly incorrect candidate that a naive single check accepted.
  • 2.6 The memory scales safely — the Merlin integrity layer: 2,000 unique items were admitted from 6,000 candidates, while all 4,000 byte-identical duplicates were filtered.A single-byte difference is treated as new, so memory growth tracks unique, verified content.
  • 2.6 The memory scales safely — the Merlin integrity layer: 14/14 measured Merlin guarantees passed against the naive alternative, with content-key computation at 3.7 nanoseconds per key.The benchmark also reports 271 million keys/second, identical output across 100,000 repeated resolutions, and addressing throughput of 16 GB/s single-core and 63 GB/s across sixteen cores.
  • 2.6 The memory scales safely — the Merlin integrity layer: Under identical caps and workloads, naive storage crashes, whereas Merlin refuses over-demand, survives torn writes without corruption, and recovers entries byte-identically after restart.The conventional file-based store became entirely unreadable under the same interruption.
  • 2.7 Reuse overhead is negligible — and cannot silently fail: 94.3% of approximate similarity retrievals were wrong at 4,500 stored items, versus 0% for exact content addressing.The figure reports zero exact-addressing collisions and warns that approximate selection silently substitutes the wrong verified item.
  • 2.8 The frontier comparison — on public numbers only: Publicly reported benchmarks show frontier models outperform Gemma-4-12B on cold-start reasoning, including AIME 2026 77.5, LiveCodeBench-v6 72.0, and MMLU-Pro 77.2.The comparison uses official benchmark results, API rate cards, and the 12B’s officially published results, with no private frontier-model measurements.
  • 2.8 The frontier comparison — on public numbers only: On solved, verified families, frontier systems regenerate and execute code per query, while the frozen 12B executes previously paid computation at zero marginal generation cost.The deposited set grows, and the right-hand-column outcomes are architectural guarantees rather than statistical outcomes.
  • 2.8 The frontier comparison — on public numbers only: N* ≈ tens of queries marks full repayment; on already solved territory, the frozen 12B beats a frontier model on cost, latency, determinism, and verification.The claim is framed as an open, falsifiable benchmark inviting counterexamples through the public testbench.

2.9 Taxonomy of execution caching and verification

This section distinguishes the system from adjacent caching, retrieval, and agent-memory approaches by combining exact selection, bit-exact deterministic reuse, and independent answer-key-free verification before storage. That combination enables verified executable solutions to be reused at zero generation tokens, unlike approaches that approximate matching, reuse only partial computation, or still decode fresh tokens.

  • Adjacent approaches: Semantic caches and vector-retrieval systems match queries approximately, while prefix/KV caching reuses partial computation and agent skill libraries accumulate reusable tools.The section explicitly contrasts these behaviors with exact selection, deterministic reuse, and independent verification.
  • Verification: Answer-key-free verification rejects subtly-wrong candidates before storage, rather than relying on another model’s judgment or a task-specific oracle.The verification principles build on differential and metamorphic testing, test-based selection, and machine-checked proof.
  • Defining combination: The system combines exact selection, bit-exact deterministic reuse, and independent pre-storage verification in one chain.A systematic review across six categories of shipped systems found none combining all three properties.
  • Execution reuse: Verified executable solutions are reused at zero generation tokens, distinguishing the system from approaches that still decode fresh tokens per answer.The comparison is framed around which system returns a pre-verified, bit-identical answer at zero generation tokens on a repeated verifiable task.

2.10 The window: six million tokens on one GPU

The verified store provides a bounded, movable 6,000,000-token working-context window on one 46 GB GPU with flat memory, depth-independent probe retrieval, and deterministic reuse. It exceeds the tested serving-engine ceilings while relying on storage bandwidth rather than global attention or GPU-memory scaling.

  • The window: six million tokens on one GPU: 6,000,000-token movable window held on one 46 GB GPU with flat memory, retrieving 5 of 5 probes in 0.55–0.59 seconds regardless of depth.Memory drift was +263 MB across the run, and access at token 5,970,000 cost the same as access at token 0.
  • The window: six million tokens on one GPU: 30,399 tokens is vLLM’s hard-error ceiling, while SGLang silently truncates input beyond ~32,000 tokens instead of returning service.The comparison used identical hardware, a single 46 GB GPU.
  • The window: six million tokens on one GPU: The control returns a wrong answer when the deep probe item is held elsewhere, showing the system uses a bounded movable window rather than global attention.The window covers arbitrarily large stored context, while no shipped engine on this hardware can operate past ~32k.
  • The window: six million tokens on one GPU: 13×, 41×, and 55× speedups at 4k, 16k, and 30k tokens of stored context respectively, with byte-exact deterministic reuse.The speedup grows with context size, while competitor caches are neither byte-exact nor deterministic.
  • The window: six million tokens on one GPU: Qwen2.5-7B served as the engine baseline because neither comparator loads the report’s 12B architecture; capacity is instead bounded by storage bandwidth.Serving engines remain preferable for batched throughput on novel prompts, whereas the window targets context they cannot hold.

3. What the numbers mean

Verified reuse eliminates generation-token costs on solved tasks, becomes cheaper after tens of queries, and returns bit-exact answers independently of the underlying model. Its auditability depends on independent verification that rejects subtly wrong candidates, false proofs, and self-contradictions.

  • Zero generation tokens on solved tasks make reuse cheaper, with one-time verification amortizing toward zero within tens of queries.The cost ratio improves with every additional query.
  • Bit-exact reuse is deterministic and auditable, providing the same verified answer every time.The passage identifies audit, reproducibility, and regulatory settings as contexts where this property matters.
  • Four vendors and four architectures produce one identical outcome for exact reuse and gated open-ended reasoning, making the memory model-independent.The memory survives model changes.
  • Acceptance requires an independent check that never consults the answer key and rejects subtly wrong candidates, false proofs, and self-contradicting outputs.Nothing is stored solely on the model’s say-so.

4. Target operating envelope and boundary conditions

The system’s advantage is scoped to recurring, execution-bound workflows whose outputs can be independently verified and reproduced bit-exactly. Its permanent verified accumulation enables zero-cost reuse, while fresh open-ended invention, closed APIs, movable-window attention, and behavior beyond tested context ceilings remain explicit boundaries.

  • Operating envelope: Recurring verifiable workflows are the system’s maximum-advantage zone, with measured 100% accuracy, zero marginal tokens, millisecond latency, bit-exact output, and ~36 mWh per answer.The stated envelope includes recurring computation, parametric analysis, formal procedures, structured extraction, and batch scoring.
  • Operating envelope: Permanent verified accumulation retains self-sourced or externally sourced knowledge and reuses it at zero cost, forever, without losing or re-deriving verified items.Autonomous invention of genuinely new hard knowledge is explicitly treated as roadmap capability rather than a present claim.
  • Scope and verification: The scope is execution-bound, verifiable work, while published benchmarks still place frontier models ahead of any 12B on fresh open-ended problems.Here, “verified” means answer-key-free acceptance checks plus bit-exact reproduction, not mathematical proof of every free-form output.
  • System boundaries: Closed text-in/text-out commercial APIs cannot provide zero-cost reuse unless the provider implements it, and six million stored tokens remain subject to bounded-window attention.The movable window uses flat GPU memory, but capacity is storage-bandwidth-bound and the model attends over only a bounded window at any instant.
  • Measured ceilings: Nine simultaneous small items and six larger items totaling 14,134 merged-context tokens are measured ceilings; behavior beyond them is unmeasured.Full source attribution was demonstrated for the merged contexts, while computed answers across merged items were demonstrated for two.

5. Public testbench

A public testbench lets anyone compare the same frozen model answering cold versus from verified memory, with key operational metrics and verification status shown per query. Free, time-limited, request-limited access makes the report’s zero-token, bit-exact behavior directly observable.

  • Public testbench: The public testbench compares cold and verified-memory answers from the same frozen model while displaying tokens, energy, determinism, and verification status per query.It is available at corbenic-galahad-bench.hf.space, with the repository at github.com/corbenicai/galahad-bench.
  • Public testbench: Free access is time-limited and request-limited, with a queue under load to keep the instance responsive for everyone.The access controls are part of the public deployment’s operating conditions.
  • Public testbench: The bench makes the report’s zero-token, bit-exact behavior directly observable rather than requiring readers to take the claim on trust.Users can inspect the relevant behavior through the public comparison itself.

6. Roadmap (explicitly future, not claimed)

The roadmap identifies future work to scale verified memory, broaden stored-knowledge composition, measure production hit rates, and extend public-benchmark evaluation.

  • Roadmap: Scale the memory to thousands of verified items across many domains using the demonstrated safe-deposit gate and uniqueness ledger.The required mechanisms are already demonstrated in §2.6.
  • Roadmap: Broaden stored-knowledge combination into larger computed results, including merged-context composition beyond the tested nine-item bound.The demonstrated core is in §2.4.
  • Roadmap: Study hit rates on production traffic and convert the per-family break-even N* into a measured system-wide savings figure.The roadmap explicitly proposes this measurement.
  • Roadmap: Extend evaluation through a public-benchmark study.The supplied roadmap passage begins a fourth item on public-benchmark evaluation but is truncated.

Appendix A — Provenance

The report anchors every headline result to SHA-256-identified artifacts, with a public manifest for byte-level verification. Methodology, architecture, configuration, and some measurement details are withheld or available only under NDA.

  • Artifact provenance: Every headline result maps to a banked raw-result artifact identified by a SHA-256 hash, and the evidence package covers every cited artifact.The public provenance manifest lists the SHA-256 digests of all 41 cited artifacts.
  • Artifact provenance: The companion repository publishes the provenance manifest so artifacts later released under NDA can be verified byte-for-byte against publication-time digests.The repository is identified as github.com/corbenicai/galahad-bench.
  • Disclosure limits: Methodology, architecture, and configuration are intentionally omitted, while artifacts, energy and token measurements, and the reproduction environment are held on file under NDA.No implementation detail, algorithm, or configuration is contained in the document by design.
  • Disclosure limits: Frontier-model figures are presented only as quotations of officially published results and rate cards, with citations provided in place.This treatment is part of the report’s related-work provenance.
Loading 2607.23806v1…