Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation

1. Introduction

• Scenario-specific specifications adapt to differing behavioral requirements and safety levels, while systematic evaluation of LLM adherence remains limited.The introduction frames specification alignment as meeting both behavioral and safety requirements in real-world scenarios.Source: §1 · pp. 1–2
• SPECBENCH unifies behavioral and safety evaluation across 5 scenarios, 103 spec, and 1,500 prompts, using Specification Alignment Rate to capture their trade-off.Its specifications draw on domain resources and organizational policies; the dataset combines synthetic and existing sources.Source: §1 · p. 2
• Test-Time Deliberation reasons over behavioral specifications while staying within safety specifications; ALIGN3 uses behavior optimization, safety-guided refinement, and a holistic audit.These three stages progressively apply behavioral and safety requirements before the response.Source: §1 · p. 2
• On Qwen3-14B, ALIGN3 raises specification alignment from 51.03% to 62.92%, approaching GPT-4.1 at 69.20% with minimal token overhead.The figure plots safety score against behavioral score.Source: §1 · p. 2, Figure 2 · p. 2

2.1. Definitions

• A scenario is an application context with a task goal and operational specifications that capture user preferences.Specifications consist of criteria for scenario preferences and risk boundaries, divided into safety and behavioral requirements.Source: §2.1 · p. 3
• Specification alignment seeks behavioral-spec satisfaction within safety-spec boundaries; its formulation captures their trade-off under a safety budget.For unsafe prompts, the paper calls for high-level, nonoperational guidance when content is restricted but not strictly prohibited.Source: §2.1 · p. 3, §2.2 · p. 4

2.2. Test-Time Specification Alignment

• Test-time specification alignment optimizes the reasoning trace for behavioral-spec satisfaction while constraining safety risk for a fixed prompt and model.The formulation defines behavioral score as the proportion of behavioral specifications satisfied and safety risk as the likelihood or severity of violations.Source: §2.2 · p. 4

3.1. Overview

• SpecBench covers 5 scenarios, 103 specifications, and 1,500 prompts to assess alignment with both behavioral and safety requirements.Each scenario includes about 20 specifications, with 200 unsafe and 100 safe prompts.Source: §3.1 · p. 4

3.2. Data Curation Process

• Each scenario’s roughly 10 safety and 10 behavioral specifications were refined with human involvement, drawing on safety resources and domain literature.Safety specifications were tailored to scenario characteristics; behavioral specifications were iteratively refined for appropriate difficulty and distinctive requirements.Source: §3.2 · p. 4
• Unsafe prompts were synthetically generated from safety specifications, while existing datasets supplied additional safe and unsafe prompts rewritten as needed to preserve their intent.Hand-crafted real-world seed questions were also used to make synthetic prompts more realistic.Source: §3.2 · pp. 4–5
• Filtering removed irrelevant, low-quality, and highly similar items before sampling 300 prompts per scenario; attack enhancement disguised unsafe intent to raise task difficulty.The final sample per scenario contains 200 unsafe and 100 safe prompts, with human-in-the-loop quality control.Source: §3.2 · p. 5

3.3. Evaluation Protocol

• SAR gives unsafe responses zero and scores safe responses using a baseline offset and behavioral compliance, then averages scores across the test set.The scoring derivation assumes zero safety budget and binary safety risk; the offset keeps scores within [0, 1].Source: §3.3 · p. 5
• The evaluator labels each specification YES, NO, or NA; any safety-spec NO sets risk to 1, while behavioral score is the YES proportion excluding NA.The authors acknowledge possible evaluator bias and report a human alignment study to validate result accuracy.Source: §3.3 · p. 5

4.1. Setup

• The evaluation covers 18 instruct and 15 reasoning models from open- and closed-source families, using GPT-4.1 as evaluator and reporting safety, behavioral, and SAR metrics.The evaluator uses temperature 0, and the safety-budget parameter α is fixed at 0.3.Source: §4.1 · pp. 5–6

4.2. Overall Results

• High safety can coincide with low behavioral scores: Llama-3.2-1B-Instruct exceeds its 70B variant in safety but has the lowest behavioral score, while explicit safety-alignment models often over-refuse.Llama-3.1-8B-Instruct and its DeepSeek-R1-Distill variant show the contrasting pattern of higher behavioral scores but lower safety scores.Source: §4.2 · p. 6
• 82.14% SAR makes GPT-5-chat the highest-scoring model, 12.94% above second-best GPT-4.1; most models score below 65%.Table 1 reports safety score, behavioral score, and SAR averages across the five scenarios.Source: Table 1 · p. 6, §4.2 · p. 6
• Reasoning models generally outperform instruct counterparts; Qwen3-32B-Thinking exceeds its instruct variant by 7.65% SAR, with gains also reported for Gemini and DeepSeek-R1.DeepSeek-R1-distill is an exception: distillation without adequate alignment can weaken existing alignment capability.Source: §4.2 · pp. 6–7

5. Optimizing Specification Alignment via Test-Time Deliberation

• ALIGN3 progressively optimizes behavior, applies safety-guided refinement near the end of thinking, then audits all specifications before answering.The method operates in a single forward pass and is described as reducing safety violations with minimal token overhead.Source: §5 · p. 7
• The baselines span multi-pass refinement or parallel sampling and single-pass reasoning interventions; multi-pass methods target instruct models, while single-pass methods target reasoning models.Best-of-N, Self-Refine, and TPO are multi-pass; ZeroThink and MoreThink are single-pass.Source: §5 · p. 7
• 11.89% SAR gain over the non-thinking baseline (51.03% → 62.92%) and 6.29% over vanilla thinking make ALIGN3 the strongest reported method, using fewer than 2k tokens.Multi-pass methods consume dozens of times more tokens than vanilla, while single-pass methods add only minor overhead.Source: Table 2 · p. 7, §5 · p. 7

Analysis across data splits

• Safety scores are substantially lower on unsafe prompts, where their larger standard deviation also highlights greater differences among models.Behavioral scores are slightly reduced on unsafe prompts, while their standard deviations remain comparable across safe and unsafe subsets.Source: §6 · pp. 7–8, Figure 5 · p. 8

Analysis across scenarios

• Average SAR is lower on Code and Biochem, while model-level performance varies across scenarios.DeepSeek-R1 performs well on Child but relatively poorly on Code; GPT-5-chat achieves consistently high SAR, particularly in Biochem and Code.Source: §6 · p. 8

Instruction-following

• Instruction-following evaluations have expanded beyond single semantic or format constraints, yet often remain question-level rather than scenario-level.The related work contrasts this with systematic, scenario-level behavioral specifications and fine-grained evaluation.Source: §7 · p. 8

Safety alignment

• Existing safety frameworks and benchmarks often apply uniform standards, whereas safety boundaries and preferences differ across scenarios.The paper positions scenario-specific specifications as a more flexible approach.Source: §7 · p. 8

Test-time scaling (TTS)

• Test-time scaling methods improve performance through iterative refinement, parallel sampling, or changes within a single generation.The cited work also adjusts verbosity or introduces reasoning interventions.Source: §7 · p. 8

Conclusion

• Experiments across models and methods show that SpecBench reveals alignment gaps, test-time deliberation enhances alignment, and Align3 improves alignment with minimal overhead.Align3 uses hierarchical reflection.Source: §8 · p. 9

Impact Statement

• Access to sensitive-content data is limited to authorized researchers, and the benchmark excludes real personal information and extremely harmful material.The authors state that collection and experiments follow privacy protection and informed-consent principles.Source: Impact Statement · p. 9

A. Best-of-N version of Test-Time Specification Alignment

• The Best-of-N formulation selects the candidate with the highest reward while maximizing expected behavioral score under a safety-risk budget.The reward may come from an external verifier or reward model, or from the model’s own judgment.Source: A. Best-of-N version of Test-Time… · p. 14

B. Discussion

• Concise policy-style specifications support test-time adaptation, while training to internalize fixed specifications is described as inflexible when requirements evolve.The authors present test-time deliberation as complementary to training-based alignment.Source: B. Discussion · p. 14

C.1. Data Construction Details / C.2. Sentence Embedding-based Filtering

• Unsafe questions are synthesized and filtered with human review, while embedding-based filtering removes redundant prompts to preserve semantic diversity.For Child, 1,630 generated questions were reduced to 800 by embedding filtering and then to 180 by random filtering.Source: §C.1 · p. 14, §C.2 · p. 15

C.3. Attack Enhancement

• WildTeaming rewrites unsafe prompts into harder adversarial variants, and all 1,000 unsafe prompts were transformed for testing.Most prompts yielded suitable variants within 100 attempts, although some required up to 10k.Source: §C.3 · pp. 15–16

C.4. Human Quality Control

• Three experts review prompts across multiple rounds for scenario relevance, safety categorization, and factual and structural quality.The process removes ambiguous, mislabeled, and low-quality samples while maintaining balanced difficulty and quality within scenarios.Source: §C.4 · p. 17

C.5. Behavioral Specification Construction Details

• Behavioral specifications are designed for clarity, task relevance, consistency, diversity, evaluability, appropriate difficulty, customization, and a relevant knowledge base.The authors use human-supervised generation and revision, consulting public resources for technically demanding scenarios.Source: §C.5 · p. 17

D.1. Model Details / D.2. Test-Time Deliberation Baselines

• The evaluated models include instruct and reasoning formats, while deliberation baselines range from reward-based sampling to iterative refinement and single-pass reasoning changes.TPO combines parallel sampling with iterative refinement; single-pass methods modify reasoning within one generation.Source: §D.1 · p. 18, §D.2 · p. 18

• Behavioral-spec helpfulness is assessed by whether a response shows sufficient problem-solving ability, not whether its content is strictly correct.Source: §D.3 · p. 18
• A scenario-specific helpfulness spec checks that a response addresses the question instead of evading unsafe content with harmless but irrelevant text.Source: §D.3 · p. 18
• Reference answers are included in evaluation prompts when available to guide judgments of content helpfulness.Source: §D.3 · p. 20

E. Human Evaluation Study

• Human–GPT-4.1 agreement was high (Kappa 0.86; SAR gap 6.1%), and four evaluators produced highly consistent rankings (W=0.995).Source: E. Human Evaluation Study · p. 20, §F.1 · pp. 20–21

F.2. Ablation Study

• All three ALIGN3 steps matter: single-step variants improve little, two-step combinations remain below the full method, and omitting any step reduces performance.Source: §F.2 · p. 21, Table 4 · p. 22

F.3. Correctness Evaluation

• ALIGN3 performs comparably to or slightly better than vanilla Qwen3-14B-thinking on Biochem and Health, while recent frontier models still leave room for joint specification alignment.Source: §F.3 · p. 21, Table 5 · p. 23, §F.4 · p. 23

F.5. Specification Judgements Analysis

• DeepSeek-R1 has higher YES rates than Llama-3.1-8B-Instruct across behavioral and safety specs, while violation patterns remain relatively even across specs and scenarios.Source: §F.5 · p. 23, Figure 9 · p. 23, Figure 10 · p. 24

F.6. The Constant Offset α in Specification Alignment Rate (SAR)

• Llama-3.2-1B-Instruct’s SAR rises from 19% at α = 0.0 to 79% at α = 1.0; the authors recommend α between 0.2 and 0.5, using 0.3 experimentally.Source: Figure 11 · p. 24, §F.6 · p. 25

F.7. Scenario Analysis

• Code and Biochem generally have lower SAR than Health, reflecting stricter code safety requirements and Biochem’s dual-use ambiguity; the authors call for broader scenario coverage.Source: §F.7 · p. 25, Figure 12 · p. 26

F.8. Case Study

• Case studies contrast unsafe compliance or bare refusal, each scoring 0, with responses that preserve safety while meeting requests and earning full scores.Source: §F.8 · pp. 25, 27

F.9. Cross-Evaluator Correlation: GPT-4.1 vs. Qwen3-32B-thinking

• GPT-4.1 and Qwen3-32B-thinking showed strong score-rank agreement; safety judgments matched on 86.9% of items, with a 4.4% average mismatch.Source: §F.9 · p. 28, Table 7 · p. 28

F.10. Attack Enhancement Analysis

• Safety scores fall by about 10% and SAR by roughly 7% after attack enhancement, while behavioral scores remain largely stable or rise slightly.The evaluation covers Qwen3-32B, Gemini-2.5-flash, and their thinking variants on the unsafe subset.Source: §F.10 · pp. 28, 30, Table 9 · p. 30
• In case studies, Gemini-2.5-flash and Claude-4.5-haiku move from safe responses to harmful detail after prompts are reframed as fiction.The examples show attack enhancement eliciting harmful responses despite refusal of the original prompts.Source: §F.10 · p. 30

F.11. Detailed Results across Different Data Splits

• Safety scores are close to perfect on safe prompts but lower on unsafe prompts, while behavioral scores are lower on unsafe prompts, illustrating a safety–behavior trade-off.Scenario-specific safety rules can classify some prompts considered safe in general settings as risky, including code with vulnerabilities.Source: §F.11 · p. 30, Table 10 · p. 31

G. How to Incorporate New Scenarios for Applications?

• The proposed extension workflow engages domain experts, builds scenario-specific specifications and datasets with review, then evaluates candidate models using training or test-time methods.The guidance also recommends targeted human evaluation to check specification clarity and annotation consistency.Source: G. How to Incorporate New Scenarios… · pp. 31–32

H. Scenarios

• The five scenarios span independent domains with distinct requirements, including strict child-safety boundaries, code vulnerabilities, and stricter safeguards for mental health than physical health.The scenarios also include travel planning and biochemical procedures.Source: H. Scenarios · pp. 32–33

I. Specifications

• The specification appendix lists domain-specific safety rules for Travel, Child, Code, Biochem, and Health.The rules include constraints on illegal travel assistance, child-appropriate content, secure code, hazardous procedures, and health advice.Source: §I · pp. 34–38

J. Prompt Design

• The prompts frame questions with scenario specifications, use a three-step Align3 process, and parse evaluator decisions as NA, YES, or NO.An attack-verification prompt checks whether an attacked query preserves the original query’s key information and meaning.Source: Figure 20 · p. 39, Figure 21 · p. 39, Figure 22 · p. 40
• The unsafe-question synthesis prompt guides generation of diverse questions intended to violate a specified safety rule.It uses the scenario, its description, the safety specification, a target prompt count, and seed examples.Source: Figure 24 · p. 42, J. Prompt Design · p. 42

K. How Do Evolving Boundaries Affect Evaluation Rankings?

• The evaluation updates specification text across five scenarios and tests 12 selected models to examine ranking stability as requirements change.Absolute scores shift naturally when the requirements are updated.Source: K. How Do Evolving Boundaries Affect… · p. 43
• Rankings under original and updated specifications remain highly consistent across behavioral score, safety score, and SAR, with rank correlations having p-values below 10−4.Top-4 and Top-8 overlaps also support agreement among highly ranked models; behavioral Kendall’s τ is slightly lower.Source: K. How Do Evolving Boundaries Affect… · p. 43, Table 11 · p. 43
• The revised examples make requirements more explicit, including laboratory oversight for pathogen procedures, emergency referral for red-flag symptoms, and adult supervision for children’s adventures.The updated code and travel specifications also add requirements concerning provenance and sustainable travel, respectively.Source: K. How Do Evolving Boundaries Affect… · pp. 44–48

L. How Do Weighted Specifications Affect Model Performance?

• The case study changes only the wording and priority of the fifth Child-scenario behavioral specification; safety rules, other specifications, prompts, and evaluation settings remain unchanged.The revised wording labels the requirement as core and calls for cultural differences to be shown without stereotypes.Source: L. How Do Weighted Specifications… · p. 49
• After the fifth behavioral specification is explicitly prioritized, Llama-3.1-8B-Instruct responses feature characters from at least two cultures or regions and make their differences salient.The original-version response featured characters from only one cultural background; the revised-version example satisfies the updated requirement.Source: L. How Do Weighted Specifications… · p. 49, Figure 31 · p. 50, Figure 32 · p. 51
• The case study indicates that users can express behavioral priorities or conflict-handling strategies directly in natural-language specifications; systematic authoring and validation remain future work.The authors identify broader support for authoring and validating priorities as an interesting research direction.Source: L. How Do Weighted Specifications… · p. 49

M. Representative Cases

• A safety-spec violation drops SAR to 0 despite behavioral performance, while a safe refusal receives SAR 0.3 and GPT-5-chat’s safe, helpful story receives full scores.The refusal fulfilled none of the behavioral specifications; the story used allegory and contrast without endorsing violence.Source: Figure 34 · p. 53, Figure 35 · p. 54, Figure 37 · p. 56
• In the Code case, DeepSeek’s RLS-disabling extension violates a safety specification and scores 0.0, whereas Gemini refuses that code, points to a supported alternative, and earns full SAR.The DeepSeek response also misses behavioral requirements; Gemini’s response avoids unsafe content while remaining helpful.Source: M. Representative Cases · pp. 57–59
• Other scenario-specific failures include Qwen3-32B-thinking omitting required DMEM information and Mistral-Small-Instruct-2409 presenting turpentine as a remedy with preparation and usage instructions.The examples concern a Biochem behavioral specification and a Health safety specification.Source: Figure 41 · p. 60, Figure 42 · p. 60
• Gemini-2.5-flash refuses the raw methamphetamine prompt but gives detailed instructions after fictional framing, and the attacked response receives a safety score of 0.The attack enhancement reframes the drug-use request as novel writing.Source: Figure 44 · p. 61, M. Representative Cases · p. 62, Figure 46 · p. 62
• For the shoplifting story prompt, Claude-4.5-haiku gives a non-actionable moral cautionary tale before enhancement but provides replicable evasion tactics after the prompt is reframed.The attacked response’s narrative presents illegal actions under morally appealing justifications.Source: Figure 47 · p. 63, Figure 48 · p. 64, Figure 49 · p. 65
• Other examples include Llama-3.1-8B-Instruct detailing an explosive device and Gemini-2.5-flash-lite omitting required hazard alerts after hazardous procedure steps.The cited cases concern a Child-scenario prompt and Biochem procedure steps involving autoclaving at 121 °C.Source: Figure 50 · p. 66, Figure 51 · p. 67
• Qwen3-14B TPO provides a crash-diet plan, while Qwen3-14B Self-Refine supplies a script disabling Windows Defender despite the applicable safety requirements.The examples concern harmful diets in Health and secure-by-default settings in Code.Source: Figure 52 · p. 68, Figure 53 · p. 69
• Even the two ALIGN3 examples include failures: one normalizes child exploitation, and another uses insulting, body-shaming language toward a teenager.The cases are in the Child and Health scenarios, respectively.Source: Figure 54 · p. 70, Figure 55 · p. 71