VReAct: Verification as an Architectural Layer for LLM Agents: A V-Model Design, and a Pilot Study of Its Deterministic Core

1 Introduction

• Without an external rejection mechanism, a stalled ReAct loop continues until an external budget intervenes, leaving callers unable to distinguish failure from a plausible answer or silence.Prior approaches change planning, efficiency, or prompting, while self-checking remains inside the agent loop.Source: §1 · p. 1

1.1 What “verification” means here

• VReAct adapts the software-engineering V-model by pairing each specification level with a verifier, separating deterministic gates from model judges, and making verdicts binding through a controller.Verification is external to the artifact-producing component, paired to one specification level, and bounded so exhausted rejection budgets produce a level-specific abstention.Source: §1 · p. 1, §1.1 · pp. 1–2

1.2 Objectives

• The paper’s long-term objective is an architectural commitment in which agents return either a verifier-accepted answer or a level-specific abstention, rather than stopping at a token limit.Its near-term objective is a minimal set of deterministic checks, described as cheap and primarily aimed at stopping for a stated reason.Source: §1.2 · p. 2

1.3 The pilot, and its scope

• The pilot tests acceptance- and unit-level verifier pairs in five configurations across 47 executions, but silent-error behavior was untested at this scale and the middle pair and memory schema were not implemented.The paper also identifies gates-versus-guards confounding and seeks feedback on study design and the proposed integration-test domain.Source: §1.3 · pp. 2–3

2 Background

• The V-model pairs each design stage with a test stage of matching scope, from unit tests for detailed design to integration tests for architecture and acceptance tests for requirements.The paper frames an agent’s inference-time control loop as a delivery process that lacks corresponding certification structure.Source: §2 · p. 3
• The deterministic grounding check tests whether an answer occurs in a returned observation, while MuSiQue is selected because its construction defeats single-hop shortcuts.The passage places this check among methods for detecting fluent but unsupported answers and contrasts MuSiQue with HotpotQA.Source: §2 · p. 3

3.1 Levels and their paired verifiers

• Each specification level has a verifier paired to its scope, so the level that catches a failure identifies where it was introduced.L1 covers requirements, L2 decomposition and data flow, and L3 an individual tool step; R3, R2, and R1 inspect those respective levels.Source: §3.1 · p. 3

3.2 Runtime properties

• The controller enforces every verdict, routes rejection to the paired level, and terminates with an answer or an abstention naming the failed level.Gates run before optional judges, which are invoked only on artifacts that pass the gates; the pilot implements R3 and R1 only.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Memory: what each component may remember

• Memory is written only by verifier verdicts: accepted work enters a success ledger, while rejection reasons and retry outcomes enter a component-keyed failure index.The proposed context is bounded by the largest subtask and preserves banked work after a component fails, but these memory mechanisms are not implemented in the pilot.Source: §3.3 · p. 4

3.4 What is new, and what the pilot omits

• The proposed novelty is the arrangement of level-specific, externally binding verification with separately measurable gates and judges, coupled to verdict-written memory.The pilot omits the L2/R2 integration pair because retrieval subtasks return strings rather than assembled artifacts for that verifier to inspect.Source: §3.4 · pp. 4–5

3.5 Worked example: a four-hop question

• In the four-hop example, individually valid retrieval steps propagate a mis-bound subtask into an invalid chain that neither implemented verifier catches.An R2 type gate could reject a person-to-region mismatch without a model call; a judge would address residual cases where types agree but entities do not.Source: §3.5 · p. 5

3.6 Worked example: fixing a repository issue

• In the repository example, a patch can pass its own test yet break three tests for the old format, so control returns to L2 for re-decomposition.The executor receives one edit with its contract and tests; after three rejections by the same gate, the run stops with the failure named and previously accepted work kept.Source: §3.6 · p. 5

3.7 The pilot implementation

• The pilot implementation puts retries and termination in bounded program logic, runs inexpensive checks before costly judges, and reports failures rather than silently weakening objectives.It omits R2 and the context store; its gates inspect plans and answers, while the plan and answer judges assess question-answering relevance and evidential support.Source: §3.7 · p. 5
• VG is the minimal pilot configuration that retains verification without a planner, using the question itself as one task with a completion condition derived from its interrogative form.The five configurations share an executor, tool set, retriever, scorer, and backbone, with adjacent configurations differing by one component.Source: §3.7 · p. 5, Table 3 · p. 5

• The implemented pipeline omits the R2 row and context store, and uses bounded rejection paths with fallback to single-task execution after plan regeneration is exhausted.Deterministic checks precede optional LLM judges, and program logic controls retries and termination.Source: Figure 4 · p. 6
• A broader evaluation requires assembled artifacts, executable ground truth, multiple model tiers, abstention-aware measures, and switchable memory; the pilot supplies only some of these.The pilot is framed as a feasibility study of whether architectural differences can be measured while other components are held constant.Source: §4 · pp. 5–6
• The study uses five four-hop MuSiQue items, two paragraph tools, and the same 8B model at temperature 0 across configurations.The executor budget is 12 steps unless stated otherwise, over two passes.Source: §4 · p. 6
• The pilot measures correct answers, silent errors, abstentions, and coverage, defining a correction as a rejection followed by a different accepted answer.Scoring follows MuSiQue’s SQuAD-style procedure.Source: §4 · p. 6
• Of 47 executions, 37 were scoreable and 10 were not; uncapped runs of N and NP continued until the provider’s token limit.Repeated passes of VG and VPG reproduced their first-pass outcomes; the plan judge rejected all five enabled plans.Source: §4 · p. 6

5.1 Execution outcomes (RQ1)

• Table 4 accounts for 47 runs; VG answered eight of ten questions and abstained on the two it could not resolve, while N and NP answered none.N and NP’s ten capped runs hit the step cap without invoking finish, and their ten uncapped runs continued until the provider refused them.Source: Table 4 · p. 6, §5.1 · p. 6

5.2 A confound between guards and gates

• The difference cannot be attributed to verification alone because verified configurations also include guards, and the duplicate-action guard could prevent repeated-tool collapse by itself.The authors identify separating guards from gates as the first ablation in the planned work.Source: §5.2 · p. 6

5.3 The effect of planning (RQ3)

• Planning cost 43% more steps for roughly one tenth of the success rate; planned runs averaged 15.6 steps with 0.08 success, versus 10.9 steps and 0.80 without a plan.The plan judge rejected all five plans, causing fallback to single-task execution; the trace analysis identifies budget fragmentation and error propagation as mechanisms.Source: §5.3 · pp. 6–7, Table 5 · p. 7

5.4 Cost (RQ4)

• VG ranged from 3 to 21 steps, while N and NP sat at the cap in every scoreable run, showing different termination patterns across configurations.VG resolved one item in three steps and used its full allocation plus a retry on an item it declined.Source: §5.4 · p. 7

5.5 Attribution of corrections (RQ2)

• Gates fired 24 times and judges three times, with no judge reversal; the judge rejected every plan after recurring objections that retrieval could not satisfy.A correction required a retry to produce a different answer that was then accepted.Source: §5.5 · p. 7, Table 6 · p. 7

5.6 Retrieval ceiling

• The retriever reached 67% of required paragraphs on average and all required paragraphs in 25% of runs, limiting interpretation of every configuration’s results.VG answered eight of ten despite complete evidence in only one of five runs; the study observed no judge false negatives or false positives, with the latter evidence limited because no confident wrong answer occurred.Source: §5.6 · p. 7

6.1 Unverified agents do not abstain

• Every verified termination recorded a decision and cause, while unverified runs did not refuse; their recorded abstentions followed the harness step cap.The uncapped collapse was produced by the authors’ artifact, not ReAct itself.Source: §6.1 · p. 7

6.2 Behavior under a stronger backbone

• No confidently wrong answers occurred, so the silent-error hypothesis could not be tested; the 8B experiment establishes a lower bound, not the judge’s target failure class.The authors chose a low-capacity backbone to isolate the deterministic architecture, but the judge’s target failures did not arise.Source: §6.2 · p. 7
• On stronger backbones outside the controlled set, plausible but unsupported answers appeared, while gates caught malformed or otherwise syntactic failures.The authors say these exploratory runs require confirmation and recommend selective judging for anticipated semantic failures.Source: §6.2 · p. 7

6.3 Planning and verification are partial substitutes

• 8 of 10 versus 1 of 10: NP’s advantage over N on stronger backbones did not survive verification.This comparison is outside the controlled set; the authors tentatively attribute the pattern to structural discipline from guards and gates, while planning without integration verification may fragment retrieval-task budgets and propagate errors.Source: §6.3 · p. 8

6.4 Why self-assessment is structurally limited

• At 8B, self-assessment judges shared the executor’s blind spots, making agreement uninformative and unanchored disagreement noisy.The authors distinguish nominal externality from effective independence: the same model judged the same model’s work.Source: §6.4 · p. 8
• A judge caught a literal finish placeholder that all seven gates admitted; expressing that catch as a gate took three lines of code.The authors suggest promoting recurring judge catches into gates and reserving runtime judging for semantic failures.Source: §6.4 · p. 8

• Separate configuration directories risk implementation divergence, although the executor, gates, tools, and scorer were byte-identical and runs were logged.The authors detected and corrected divergence more than once during development.Source: §7 · p. 8
• Exact-match scoring under-credits paraphrases, while the retrieval ceiling bounds reported figures and may produce false negatives for the judge.Source: §7 · p. 8
• The study covers one backbone, benchmark, task family, and small sample; it reports qualitative patterns on stronger backbones and tests only part of the architecture.The integration row and proposed memory schema were not tested.Source: §7 · p. 8
• With 37 scoreable runs and cells of 7 to 10, the authors perform no statistical inference; absent runs are crashes, not missing at random.N and NP appear in outcome columns only in capped configurations.Source: §7 · p. 8
• One author implemented all five configurations, and scoring was not blind.Source: §7 · p. 8

• Four deficiencies drive the plan: insufficient statistical power, one model tier, a task family unable to exercise integration, and separate configuration directories.Source: §8 · p. 8
• Half of the twelve-month schedule goes to code, where deterministic verifiers are rich and the L2/R2 row can be exercised.Source: Table 7 · p. 8

• Across five items, 47 executions, and one 8B backbone, unverified runs never terminated by decision, while verified runs ended in an accepted answer or reasoned abstention.Eight of nine observed corrections were free; planning without its paired verifier made the system worse.Source: §9 · p. 8
• Three uncertainties remain: whether the effect belongs to verification or bundled guards, whether judges improve when separated from the executor, and whether integration verification matters for code.The authors note that the question-answering task cannot exercise the integration level.Source: §9 · p. 8