Source-linked AI summary
REA: From Correct LLM Judgments to Safe Decision Delivery
Jingxiang Meng, Aofan Liu, Yaoshuai Ma, Fangxin Liu, Yongbiao Chen
TL;DR
Generative safety guards must produce judgments applications can parse, but evaluations leave open how much classification performance survives consumer contracts and what happens when parsing fails. REA follows judgments through delivery and fallback; experiments show that output requirements affect retained correct decisions and resulting actions.
Problem
Safety evaluations measure judgment quality and output compliance, but leave open how much a guard's classification advantage survives the consumer contract and what action follows parsing failure.
Source: §1 · p. 1Method
REA traces an expressed guard judgment through the consumer contract, extraction, and fallback, separating semantic accuracy from conditional delivery.
Source: §1 · p. 2Results
93.0% correct and executable decisions under GuardReasoner's strict native-answer consumer fell to 80.1% under a structured-trace contract, with most of the difference recovered by accepting trace-format variants.
Source: §1 · p. 2Takeaways & Limitations
Guard replacement should evaluate verdict format, extractor, and fallback together because parser rejection can change the application action despite a correct guard judgment.
Source: §1 · p. 2The findings concern decision delivery under explicit verdict contracts in evaluated model–consumer configurations; their relevance to open-ended decisions and multistep interactions remains an open question.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Generative safety guards must classify requests correctly and return outputs that applications can parse. Reasoning guards can improve classification accuracy while producing outputs that applications cannot parse. When applications cannot extract a correct prediction, fallback handling can allow harmful requests or block safe ones. We introduce Resolve--Express--Act (REA), a framework that traces judgments through parsing and fallback to application actions. Experiments on classifiers and safety guards show that correct predictions can appear in incorrectly formatted outputs. Parsing the safety verdict preserves the reasoning guard's accuracy advantage, while stricter explanation requirements reject additional correct predictions. Accepting formatting variations recovers most rejected correct predictions. Held-out tests confirm that output requirements and generation limits affect whether correct predictions can be parsed. In our fail-open replay, matching the parser to the output and fixing formatting reduced the fraction of harmful requests allowed despite correct guard judgments from 95.5% to 0%. A fixed-output comparison shows that model-based extraction can preserve guard verdicts, but is not automatically reliable; deterministic normalization already repairs the observed native-format failures. Fallback replay shows that the same correct prediction can lead to different safety actions depending on the parser and fallback policy, including allowing requests the guard correctly identifies as harmful.
1 Introduction
Generative guards must deliver their judgments in forms applications can parse, because contract failures can separate a correct classification from the action taken. The introduction presents REA as a way to trace that path and shows that consumer requirements affect retained guard accuracy.
1 Introduction
- A bare-label parser can reject a correct harmful-request verdict placed after an explanation, allowing the request under fail-open or blocking it under fail-closed.Reasoning guards may express their verdicts after reasoning, while a verdict-only consumer expects a different output form.Source: §1 · p. 1
- 93.0% correct and executable decisions under GuardReasoner's strict native-answer consumer fell to 80.1% under a structured-trace contract.Accepting trace-format variants recovers most of the difference.Source: §1 · p. 2
- Fallback replay shows that parser rejection can lose correct Block decisions, while native-answer parsing and format normalization recover decisions lost at the interface.The work identifies verdict format, extractor, and fallback as components to evaluate together when replacing a guard.Source: §1 · p. 2
- REA formalizes decision delivery by separating semantic accuracy from conditional contract compliance and measuring their joint success.The framework follows the expressed judgment through the consumer contract and fallback to the application action.Source: §1 · p. 2
2 Related Work
Related work addresses reasoning-based moderation, safety and instruction-following evaluation, constrained generation, and parsing failures in applications. The paper positions its contribution as tracing an expressed guard judgment through extraction and fallback to the resulting action.
2.1 Reasoning Guards and Moderation Tasks
- Reasoning and rationale methods report classification or robustness gains, while moderation distinguishes request harm, response harm, and refusal as separate prediction targets.The paper motivates field-aware extraction to preserve which target a verdict concerns.Source: §2.1 · p. 2
2.2 Safety Evaluation and Instruction Following
- Prior evaluations assess safety judgments, instruction following, and prompt sensitivity; this work asks how much a correct expressed decision survives extraction and fallback.Conditional delivery measurement distinguishes classification errors from failures to satisfy the consumer's requirements.Source: §2.2 · p. 3
2.3 Constrained Generation and Output Contracts
- Constrained generation guides outputs into formal languages, but syntax guarantees neither verdict accuracy nor correct application actions.REA compares free-text and constrained interfaces while separating semantic accuracy from decision delivery.Source: §2.3 · pp. 3–4
2.4 Parsing Differentials and Application Actions
- Prior work examines parsing and action failures across applications; this paper focuses on how postgeneration interfaces can reverse the effect of a correct decision.It treats the model's probabilistic interpretation and the application's deterministic code as two interpreters of the same output.Source: §2.4 · p. 4
3 Framework and Metrics
REA tracks a response from its expressed decision through contract-compliant delivery to the executed action. It separates semantic correctness from delivery loss and fallback effects.
3 Framework and Metrics
- REA follows a response from its expressed decision through contract-compliant delivery to the executed action.The stages are defined over the same evaluation population.Source: §3 · p. 4
3.1 Interface Model
- The interface model distinguishes a semantic reader of expressed judgments from an application parser governed by an accepted-representation contract.A response can express “This request is harmful” while a bare-label contract rejects it.Source: §3.1 · p. 4
3.2 Semantic Accuracy and Decision Delivery
- Semantic accuracy A measures correct expressed decisions, while conditional contract compliance κ measures delivery among those correct decisions.Aκ is the fraction of responses with a correct, executable decision; A(1−κ) is the fraction of correct decisions lost in delivery.Source: §3.2 · p. 4
3.3 Decision Delivery and Executed Actions
- Decision-delivery analysis separates wrong or missing judgments, rejection of correct judgments, and acceptance with the wrong extracted decision.Fail-open maps rejected outputs to Allow, whereas fail-closed maps them to Block; thus a rejected correct Block becomes Allow only under fail-open.Source: §3.3 · p. 5
3.4 Response Modes and Consumer Recovery
- Response modes distinguish bare native verdicts, expanded native-vocabulary verdicts, other decision vocabulary, and outputs with no unique task verdict.Punctuation variants and labels embedded in sentences or rationales are distinct from other decision vocabulary; instruction changes can shift mode frequencies, while parser changes on fixed responses affect recovery.Source: §3.4 · p. 5
4 Experiments
The experiments test how instruction wording, output contracts, extractors, and guard specifications affect whether correct judgments reach applications. Results show that semantic accuracy can exceed decision delivery, and that stricter explanation requirements reject some otherwise correct outputs.
4.1 Characterizing the Accuracy–Delivery Gap
- 93 of 480 correct responses violated the bare-label contract, leaving correct decision delivery at 80.6% despite 100% semantic accuracy.The violations were 91 sentence-or-rationale outputs and two with added punctuation; the consumer required the entire response to match a valid label.Source: §4.1 · p. 6, Table 1 · p. 6
- 91 of 120 responses without an output-form clause expanded beyond a bare label, compared with two punctuation-only violations among 360 responses with such a clause.The association is specific to these configurations and does not estimate deployment prevalence.Source: §4.1 · p. 6, Table 2 · p. 6
- 80.6% to 91.7% acceptance, with whole-word containment reaching 100%, shows that more permissive extraction recovers rejected labels in this sample.Containment alone cannot distinguish a selected verdict from a label mentioned in an explanation.Source: §4.1 · p. 6
- 93.4% versus 87.2% accuracy favored GuardReasoner over Llama Guard 3 on the same XSTest requests, but none of its structured outputs met the tested legacy contract.Better unsafe-request recognition outweighed GuardReasoner’s lower accuracy on safe requests.Source: §4.1 · p. 6
- 93.0% correct-and-executable decisions under strict native-answer parsing fell to 80.1% with strict trace requirements; accepting trace variants raised delivery to 91.8%.The remaining rejections lacked separate step bodies; these checks measure structural compliance, not explanation quality.Source: §4.1 · p. 6
A. Model accuracy
The replay links parsing and fallback choices to safety actions, while held-out tests examine generation budgets and output interfaces. Extraction comparisons show that normalization can repair observed format failures, whereas a valid extracted label does not necessarily preserve the guard’s verdict.
A. Model accuracy
- The legacy bare-label parser rejected all 588 GuardReasoner responses: fail-open allowed every unsafe request, while fail-closed blocked every request, including safe ones.Native-answer recovery executes the guard’s expressed request-level verdict; normalization removes remaining delivery losses without changing expressed decisions.Source: Table 4 · p. 7, A. Model accuracy · p. 7
4.2 Representative Factors Affecting Decision Delivery
- Reducing the generation budget from 700 to 256 tokens lowered semantic accuracy by 1.2 percentage points and correct decision delivery by 25.2 points.Under B192, correct delivery fell to 3.1%; responses often hit the length limit before completing the required answer structure.Source: §4.2 · p. 8, Table 5 · p. 8
- Typed JSON achieved 99.6% acceptance but 57.1% semantic accuracy, showing that syntactic validity did not guarantee a correct verdict.The tested interface configuration changed output instructions, schema, and decoding together, so the accuracy difference cannot be attributed to constrained decoding alone.Source: §4.2 · p. 8, Table 5 · p. 8
4.3 Repairing Verdict Extraction
- On 762 explicit-verdict outputs, normalization recovered all three strict-parser rejections without changing verdicts, while Qwen returned usable labels but changed 344 decisions.Luna faithfully extracted all 762 verdicts; this fixed-configuration comparison does not establish recovery on truncated or unclear outputs.Source: §4.3 · p. 8, Table 6 · p. 8
5 Conclusion
The paper uses REA and Aκ to distinguish correct guard judgments from decisions successfully delivered under a consumer contract. It finds that extraction failures can turn correct judgments into unintended safety actions, while normalization repaired all observed extraction failures in the evaluated explicit-verdict outputs.
5 Conclusion
- REA and Aκ distinguish correct judgments from those successfully delivered under consumer requirements, generation budgets, and output interfaces.Fallback replay showed that extraction failures can turn correct judgments into unintended safety actions.Source: §5 · p. 8
- Lightweight normalization repaired all observed extraction failures, while model-based extractors either preserved or misread the original verdicts.The results support evaluating the guard, extractor, and fallback policy together, since practical value depends on the application faithfully receiving and acting on a judgment.Source: §5 · p. 9
Limitations
The paper bounds its conclusions to decision delivery under explicit verdict contracts in evaluated model–consumer configurations, while identifying open-ended decisions and multistep interactions as questions for future research. Its appendix details assumptions, study materials, and response patterns relevant to those findings.
Limitations
- The findings concern explicit verdict contracts in evaluated model–consumer configurations; whether they extend to open-ended decisions or multistep interactions remains a future question.The break-even identity requires per-response correctness and contract-compliance measurements conditioned on the same responses; benchmark-suite weighted-average F1 scores do not meet that requirement.Source: Limitations · p. 9, A Conditions on the Break-Even… · p. 11
Appendix Contents
- The appendix follows the main argument with framework conditions, comparisons, held-out validation, extraction repair, supplementary analyses, and reproducibility details.This describes the appendix’s organization rather than an additional result.Source: Appendix Contents · p. 11
A Conditions on the Break-Even Substitution
- Equation 6 is valid as a deployed-reliability threshold only when both guards’ per-response accuracy and contract compliance are measured on the same request set.Section 4.1 uses 196 requests, with κ measured on responses in the fraction counted as correct.Source: A Conditions on the Break-Even… · p. 11
B.1 Instruction Renderings
- Table 7 lists twelve instruction renderings, pooled off-contract rates, and whether each instruction explicitly restricts output form.The coding treats any output-form restriction as a constraint, regardless of wording.Source: Table 7 · p. 11, §B.1 · p. 11
B.2 Response-Mode Examples
- Table 8 illustrates response modes classified mechanically by exact-label match, line breaks, sentence-level labels, or remaining single-token responses.The study’s two surface-formatting violations were the only off-contract responses under an instruction with an output-form clause.Source: §B.2 · p. 12, Table 8 · p. 12
B.3 Model–Rendering Cell Results
- Across six renderings, 70 of 80 record–model pairs had at least one violation (87.5% overall; 100% for topic classification and 75% for sentiment).Under deterministic decoding, 46 of 48 model–rendering cells had either zero or ten off-contract responses.Source: §B.3 · p. 12, Table 9 · p. 12
B.4 Extractor Definitions and Per-Model Results
The extractor ladder shows that permissive parsing recovers more fixed responses, but recovery varies by response form and may misread label mentions as decisions. The appendix specifies guard output contracts and parsers, including the fields and trace structures they require.
B.4 Extractor Definitions and Per-Model Results
- Pooled acceptance rises from 80.6% to 91.7% and then to 100% as extractor rules become more permissive, without regenerating responses.First-line extraction recovers Claude’s rationale-style responses but none of GPT’s sentence-style answers; whole-word containment can mistake a label mention for the model’s choice.Source: §B.4 · p. 12
- Contract compliance depends on response mode and parser together, and higher acceptance alone does not establish correct semantic extraction.E4 reaches 100% in the described study with A = 1 and small, non-overlapping label sets.Source: §B.4 · p. 12, §C.1 · p. 13
C.2 Native-Answer and Trace Parsers
- Native-answer parsers require one valid Request, Completion, and Response field, while trace parsers additionally require ordered tasks and consecutively numbered steps with separate, nonempty bodies.The Request field supplies the user-request classification; reading the last safety word could instead conflate it with the assistant-response classification. Relaxing trace-heading variants recovers 72 of 83 rejections and lifts Aκ to 91.8%, while 11 responses without separate step bodies remain rejected.Source: §C.2 · pp. 13–14
C.3 Results Across Guard Specifications
Across guard specifications, reasoning and output-form requirements produce different response modes and delivery outcomes. Under fail-open replay, parser mismatch can lose correct Block decisions, while native-answer or trace parsing is incompatible with Llama Guard’s response structure.
C.3 Results Across Guard Specifications
- GuardReasoner retains a three-task trace under form-only instructions (median 171 words; accuracy 93.9%), while Llama Guard retains a one-word verdict under reason instructions (accuracy 87.2%).Fill-in and reason-first produce 25 hazard-code-only responses, 21 concerning harmful requests; all 1,764 responses express a unique verdict.Source: §C.3 · p. 14
- 92 of 96 unsafe requests (95.8%) lost correct Block decisions under fail-open with the legacy parser at GuardReasoner’s official specification.At the official specifications, the accuracy ratio is 87.2/93.4 = 93.4%, and the unsafe-subset ratio is 80.2/95.8 = 83.7%; native-answer and trace parsers reject all 1,176 Llama Guard responses.Source: §C.3 · p. 14, Table 12 · p. 14, Table 13 · p. 14
D Held-Out Validation of Consumer-Contract Effects
Held-out tests preserve the distinction between guard accuracy and the share of correct judgments delivered under a consumer’s contract. Verdict-only parsing preserves GuardReasoner G1’s accuracy advantage, while stricter trace requirements reject more of the same outputs; the extraction-repair evaluation is limited to fixed configurations and explicit native verdicts.
D Held-Out Validation of Consumer-Contract Effects
- The held-out evaluation used 254 XSTest requests absent from both released guard caches, and rescoring against adjudicated labels preserved the qualitative conclusions.Two annotators labeled request text while model outputs were sealed; an earlier smoke test’s two responses for item 1 were excluded, but item 1 remained in the formal evaluation.Source: D Held-Out Validation of… · p. 15
- Under the official W1 specification, Llama Guard 3 had no contract loss, while strict native-answer parsing preserved GuardReasoner G1’s higher semantic accuracy without losing correct judgments.Across GuardReasoner specifications, strict native parsing caused a small delivery loss from noncanonical answer blocks, which deterministic normalization recovered.Source: D Held-Out Validation of… · p. 15
- The complete-trace contract rejected more of the same G1 outputs than verdict-only parsing, while accepting descriptive headings and Markdown variants recovered much of the loss.Empty reasoning-step bodies remained rejected; because outputs were identical, the difference reflects consumer requirements rather than changed model judgments.Source: D Held-Out Validation of… · p. 15
E Verdict-Extraction Repair Protocol
- The extraction-repair comparison used frozen configurations and cached outputs with explicit final Request verdicts, so it does not establish performance on truncated or ambiguous outputs.The two model configurations differ in instruction wording, evidence requirements, and serving stack, preventing attribution of outcome differences to model size or architecture.Source: E Verdict-Extraction Repair Protocol · pp. 15–16
F Supplementary Explanation and Recovery Analyses
The supplementary analyses find that deterministic normalization repaired the observed strict native-answer rejections without changing verdicts or making another model call. An exploratory rationale audit also shows that schema acceptance does not guarantee useful support for a verdict.
Normalization, Retry, and Escalation Volume
- Deterministic normalization recovered the observed strict native-answer rejections without another model call or a change in the expressed verdict.A single retry recovered only part of this small failure set; normalization eliminated native-format rejections that would otherwise require fallback in this sample.Source: F Supplementary Explanation and… · p. 16
Blinded Explanation Audit
- Typed JSON had lower descriptive rationale-support scores than free text and terminal-answer outputs despite near-complete schema acceptance.The exploratory audit scored support for the expressed verdict, not verdict correctness or rationale faithfulness; its coupled interface and prompt conditions do not identify the cause of the difference.Source: F Supplementary Explanation and… · p. 16
Normalization, Retry, and Escalation Volume
- The small failure set does not establish a general ranking of recovery strategies, and escalation estimates describe routing demand rather than staffed-review latency, cost, or effectiveness.Tighter generation budgets left many more requests unresolved in this sample.Source: F Supplementary Explanation and… · p. 16
G Resources and Reproducibility
The resource and reproducibility discussion specifies the evaluated data, model-run settings, and intended use, while delimiting what the experiments can establish. The studies use English texts and do not establish coverage of other languages or demographic populations.
G Resources and Reproducibility
- The evaluated texts and model instructions are English, and the samples do not establish coverage of other languages or demographic populations.The general-classification study is a repeated small-sample interface probe, while XSTest contains constructed safe requests and unsafe contrasts across 18 types.Source: G Resources and Reproducibility · p. 17
- The guard experiments evaluate harmfulness of user requests, and the stated intended use is offline decision-delivery analysis rather than production safety-system certification.The text also notes that these samples do not establish coverage of other languages or demographic populations.Source: G Resources and Reproducibility · p. 17
- Held-out guard runs used an NVIDIA H100 PCIe GPU and Ollama 0.34.4 with Q4_K_M artifacts, while the recorded parameter counts were approximately 8.03B for Llama Guard 3 and 3.21B for GuardReasoner.Normal output limits were 256 and 700 tokens, respectively; these are local guard-run details, not provider infrastructure details for general classifiers or Luna.Source: G Resources and Reproducibility · p. 17
- The controlled-generation comparison changed only the output-token limit across budget conditions, while the interface comparison reused G1 outputs for free text and generated outputs for terminal-answer and typed-JSON conditions.A complete total-compute accounting across all experiments is unavailable, so reported durations are not total research GPU-hours.Source: G Resources and Reproducibility · p. 17