PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Surface Alignment: Grounding the Dynamics of Situational Understanding and Generative Control in LLMs

Chenghao Yang

arXiv:2608.29610v1cs.CL

TL;DR

LLM alignment commonly emphasizes fluent, safe, stylistically consistent responses, while evidence about grounding in context and generation remains limited. This thesis evaluates those grounding challenges and proposes dynamic controls, reporting failures in situational understanding and premature stylistic collapse under standard alignment tuning.

  • Problem

    The thesis addresses limited grounding in alignment research, which prioritizes fluency, safety, and tonal consistency over how models track changing contexts and structure generation.

    Source: §1.1 · p. 30

  • Method

    The thesis examines grounding in context processing and generation, then proposes dynamic controls for grounded interaction, including context engineering and exploration separated from stylistic constraints.

    Source: §7 · p. 6, §9 · p. 7

  • Results

    The thesis reports that models struggle to track changing situations and understand their own generations, while standard alignment tuning can constrict generation into premature stylistic collapse.

    Source: §1.1 · p. 30, §2.3.4 · p. 45

  • Takeaways & Limitations

    The thesis concludes that grounded interaction should address both context processing and generation, rather than surface behavior alone.

    Source: §7.7 · p. 159

    The limitations discussion frames some comparisons around ChatGPT and other LLMs and notes verification on other open models.

    Source: Limitations · p. 115

Abstract

from arXiv · show

The current alignment tuning paradigm for Large Language Models (LLMs) prioritizes surface-level behaviors -- fluency, safety, and tonal consistency. While effective for casual chat, this thesis argues that such surface alignment masks a lack of grounding, creating models that are stylistically confident but situationally brittle. We propose a framework of Grounded Alignment, analyzing how models process context (Input) and structure generation (Output), then aligning these grounded behaviors to human needs. First, we evaluate failures in Situational Grounding. SitTest shows that despite large context windows, state-of-the-art models struggle to maintain a consistent "mental model" of a changing environment. ReCode further shows that models rely on surface heuristics rather than deep syntactic dependencies: they "read" extensive histories without truly "understanding" the evolving situation. Second, we evaluate Generative Grounding. We introduce the Branching Factor (BF) to map LLM generation, finding that standard alignment tuning constricts this landscape into premature stylistic collapse. Hindsight further shows that models often fail to understand their own generations. Finally, we propose Dynamic Control for grounded interaction. AI Realtor demonstrates context engineering to compensate for poor situational grounding. Base-Aligned Model Collaboration decouples exploration from stylistic constraints. We also present Annealed Sampling for verifiable reinforcement learning and apply these ideas to Addiction Support, where model-generated rationalization offers a communication interface for high-stakes domains. Collectively, this work moves beyond surface alignment toward agents anchored in both context and generation.

I SITUATIONAL GROUNDING … 7 AI REALTOR: TOWARDS GROUNDED PERSUASIVE LANGUAGE GENERATION FOR AUTOMATED COPYWRITING

The supplied excerpt outlines an AI Realtor chapter on grounded persuasive copywriting, listing its benchmark, economic scaffolding, agentic implementation, evaluations, and concluding sections.

7 AI REALTOR: TOWARDS GROUNDED PERSUASIVE LANGUAGE GENERATION FOR AUTOMATED COPYWRITING

  • The chapter includes a benchmark for grounded persuasion and an economic scaffolding of copywriting.These appear as sections 7.2 and 7.3 in the chapter outline.Source: §7 · p. 6
  • The outline also includes an introduction, related work, discussion, conclusion, and ethics statement.The listed sections span 7.1 and 7.6–7.9.Source: §7 · p. 6
  • The agentic implementation is organized into grounding, personalization, and marketing modules.The outline names predicting credible marketing features, aligning with preferences, and capturing surprisal via RAG as module topics.Source: §7 · p. 6
  • The chapter lists human-feedback and AI-feedback evaluations, hallucination checks, and qualitative analysis.These are the four evaluation subsections in the outline.Source: §7 · p. 6

9 ANNEALED SAMPLING FOR VERIFIABLE REINFORCEMENT LEARNING … Chapter Overview

This section examines failures in situational and generative grounding, including models’ difficulty tracking changing environment states and distinguishing state understanding from surface strategies. It also presents dynamic controls intended to better align generation with human needs.

9 ANNEALED SAMPLING FOR VERIFIABLE REINFORCEMENT LEARNING

  • Chapter 9 presents EAD for verifiable reinforcement learning, with sections on sequential exploration, RLVR training, algorithm compatibility, and inference-time scaling.The chapter overview labels EAD as improving RLVR training and inference-time scaling but supplies no result values.Source: §9 · p. 7

1.1 The Illusion of Alignment

  • The thesis defines grounding as anchoring representations in task dynamics and reflecting uncertainty during generation, framing the gap from surface behavior as a problem of Grounded Alignment.It analyzes both input processing and output generation and proposes a shift from static surface alignment to dynamic grounded generation.Source: §1.1 · p. 30

1.2 Diagnosing the Gap in Situational Grounding

  • SitTest finds “Lost-in-the-Middle” state-tracking failures: models with the full interaction history often hallucinate state updates that never occurred (Yang and Ettinger, 2023b).The findings distinguish retaining the appearance of conversation from maintaining the consistency needed for reliable interaction.Source: §1.2 · p. 31
  • ReCode finds that models rely on surface heuristics such as variable names rather than deep syntactic dependencies, and their performance degrades under semantic-preserving perturbations (Wang et al., 2023a).Examples include renaming variables or altering docstrings.Source: §1.2 · p. 31
  • Apparent robust reasoning can instead reflect fragile pattern matching that breaks when surface cues are removed.Source: §1.2 · p. 31

1.3 Characterizing the Dynamics of Generative Grounding

  • Branching Factor (BF) maps output probability concentration, and the study finds alignment tuning causes premature stylistic collapse into low-entropy trajectories from the first token.The authors argue that enforcing a helpful style prunes valid semantic space, unlike base models’ pattern of broad exploration followed by narrowing.Source: §1.3 · p. 32
  • In Hindsight (Li et al., 2024c), self-reflection often fails without external feedback because models lack a deep grasp of their prior outputs and may reinforce ungrounded hallucinations.Source: §1.3 · p. 32
  • Amortized Interpretability (Yang et al., 2023) trains a separate model to predict token importance, enabling efficient attribution to diagnose spurious correlations and attention sinks.The framework is proposed as an alternative to computationally prohibitive traditional Shapley Value estimation.Source: §1.3 · p. 32

1.4 Restoring Grounding through Dynamic Control

  • AI Realtor (Wu et al., 2026) uses retrieved context before personalization, while Base-Aligned Model Collaboration (Wang et al., 2026b) routes exploration to base models and structural guidance to aligned models.AI Realtor is described as producing persuasive but factually rigorous copy; the routing framework supports flexible control over generation.Source: §1.4 · p. 33
  • Annealed Sampling (Yang et al., 2025d) synchronizes sampling with distribution cooling to prevent premature collapse and improve reasoning, while rationale-grounded addiction support (Yang et al., 2024b) makes reasoning transparent.In the addiction-support application, grounded thinking improves detection of subtle Opioid Use Disorder stages and builds trust in high-stakes applications.Source: §1.4 · p. 33

1.5 Thesis Structure

  • The thesis structure groups its work into situational grounding, generative grounding, and dynamic control, with Addiction Support concluding the high-stakes applications.Source: §1.5 · p. 34

2.1 Introduction

  • ChatGPT outperformed other chat-fine-tuned models yet failed to retain coherent, correct environment states over time despite access to the full dialogue history.The chapter overview describes a synthetic environment for controlled testing of situational understanding through state tracking and enumeration.Source: Chapter Overview · p. 36, §2.1 · p. 37
  • ChatGPT lacks robust tracking of situation-state changes, with findings also noting retained prior states, hallucinated updates, and accidentally correct outputs.Source: §2.1 · p. 37

2.2 Related work

  • The study tests whether ChatGPT has the situational understanding needed for coherent, consistent, effective dialogue, adding controls for test shortcuts and probing model-update dynamics.Source: §2.2 · p. 38

2.3 Building a Situational Testing Environment

  • The tests use a synthetic box-moving environment with full access to states, omitting reward pursuit and route branching as input describes progressive changes.Source: §2.3 · p. 39

2.3.1 Environment Setup

  • Models receive task instructions and few-shot examples, then must enumerate all environment states after a test step; experiments use 10 boxes, 10 keys, and 1–10 steps.Source: §2.3.1 · pp. 39–40

2.3.2 Robustness Checks for State Tracking

  • Robustness checks use synthetic symbols and counterintuitive mappings to reduce reliance on pretrained language patterns, copying, or memorized mappings.Synthetic functors and arguments are regenerated for each test case.Source: §2.3.2 · pp. 41–42

2.3.3 Evaluation Metrics

  • State-EM measures the proportion of predicted states matching expected states, while Step-EM is a stricter metric.Source: §2.3.3 · p. 42
  • State-EM requires an exact match of predicted and expected states, including truth values and state count, while Step-EM checks correctness at each step.The metrics are intended to test coherent tracking across changes and prevent strong scores from merely copying prior states when updates are sparse.Source: §2.3.3 · p. 42
  • ChatGPT struggles to solve the tasks reliably even though simple string automata plus a status-tracking table should suffice.The authors present the task’s simplicity as a test of basic state-tracking capacity.Source: §2.3.3 · p. 43, §G.10.5 · p. 359

2.3.4 Discussion: Synthetic Environment as Simulation for Real-World

  • The synthetic environment models real-world state-tracking tasks such as dialogue-system slot-filling, and state tracking can inform assessment of instruction- and rule-following.The authors relate this to teaching agents social norms and policies through remembered initial states, synthetic languages, and interaction histories.Source: §2.3.4 · p. 44
  • As interactions accumulate, ChatGPT loses track of states, propagates errors, and can hallucinate despite all updates remaining within the input window.The authors argue this warrants caution when implementing Constitutional AI and calls for further study.Source: §2.3.4 · p. 45

Changes … SHRINKS THE GENERATIVE HORIZON

This section examines models’ ability to track evolving environment states and their robustness to natural, meaning-preserving changes in code-generation prompts. It describes ReCode’s transformations and execution-based metrics, and reports that robustness varies across models and perturbations, with vulnerabilities to syntax and code-style changes.

Changes

  • State-EM reaches 90%, but Step-EM is up to 70% lower in 2-shot and 40–50% lower in 5-shot settings.The authors caution that State-EM may be inflated by accidentally correct states.Source: Changes · pp. 45–46, Table 2.1 · p. 46
  • Counterintuitive instructions reduce Step-EM to 10–20% in most settings, though State-EM remains 70–80%, indicating reliance on familiar language-to-logic mappings.State-EM should be interpreted cautiously, but the reported accuracies exceed random guessing and ignoring the rule.Source: Changes · p. 46
  • Synthetic language more often improves than worsens performance, while additional in-context samples fail to improve it in most cases.The authors suggest synthetic language can help models rely more robustly on in-context information rather than pretraining biases.Source: Changes · p. 47

2.5.1 Tracing Errors: State Tracking over Steps

  • Step-EM declines sharply as situational changes accumulate, and compressed initialization with skipped steps improves performance, ruling out state complexity as the cause.The probing evaluates intermediate states; the authors report the strongest degradation in Step-EM.Source: Figure 2.2 · p. 48, Table 2.2 · p. 49, §2.5.1 · pp. 49–50
  • Using five rather than two in-context samples yields less rapid degradation in model accuracy after the in-context window, but the advantage lasts only temporarily or over a limited number of steps.In the 2-shot case, at Step-3—the second step preceded by a non-demonstration step—Step-EM drops from 60% to 40%.Source: §2.5.1 · pp. 50–51

2.5.2 Fine-grained Analysis of Update Patterns

  • The next subsection examines update dynamics more finely after finding that state tracking degrades as the number of steps increases.Source: §2.5.2 · p. 51
  • As steps increase, errors rise on states that should remain untouched: Dirty Read retains prior errors, while Hallucinated Update changes correct unchanged states to incorrect.These patterns indicate that errors accumulate through both retention or propagation of earlier errors and failures to preserve states that should not change.Source: §2.5.2 · p. 52
  • Correct predictions also become less stable: Correct Update and Maintain Correctness decline, while accidental corrections without stated updates rise, especially for counterintuitive instructions.The authors interpret accidental corrections as evidence of memory limitations and susceptibility to noise in changes to enumerated states.Source: §2.5.2 · p. 52
  • Counterintuitive instructions make correct updates drop more sharply and spurious correct updates rise more substantially, suggesting memorized language-to-logic mappings aid correct state updates.The source says these changes suggest that inability to rely on such mappings reduces the model’s ability to execute and maintain correct updates.Source: §2.5.2 · p. 53

2.6 Discussion

  • Despite access to the complete dialogue history, ChatGPT fails to retain coherent, correct environment states; analysis attributes this to non-persistent in-context memory and hallucinated updates.The paper presents a synthetic environment for testing situational understanding and says its findings may diagnose limitations in multi-round interactions and instruction following.Source: §2.6 · p. 53

Limitations

  • The situational test may not generalize to some real-world applications, evaluates only ChatGPT, and leaves a comprehensive chatbot benchmark evaluation for future work.The authors note that other systems, including multi-module designs, may perform better; they also identify prompt optimization and testing across other environments as future directions.Source: Limitations · p. 54

Chapter Overview

  • The chapter introduces ReCode, a framework for systematically assessing code-generation robustness under semantic-preserving perturbations.Source: Chapter Overview · p. 55

3.1 Introduction

  • ReCode examples show that natural, meaning-preserving changes such as docstring paraphrasing and function-name camel-case renaming can make code-generation models fail.Source: §3.1 · p. 56
  • ReCode is presented as a code-generation robustness benchmark with over 30 natural transformations and three metrics: Robust Passs@k, Robust Drops@k, and Robust Relatives@k.Transformations cover docstrings, function and variable names, code syntax, and code format; human and quantitative similarity evaluations indicate that most perturbed prompts preserve meaning and naturalness close to the originals.Source: §3.1 · pp. 56–57
  • Larger models and more diverse pretraining can improve worst-case robustness, but models may generalize non-robustly; syntax perturbations are most disruptive, and MBPP changes more than HumanEval.The empirical comparisons cover CodeGen, InCoder, and GPT-J across sizes on HumanEval and MBPP.Source: §3.1 · pp. 57–58

3.2 Related Work

  • Prior code-robustness studies addressed varied tasks, but the authors identify no comprehensive robustness benchmark for code generation; generated code permits objective, quantitative evaluation.The related-work discussion contrasts this gap with robustness benchmarks focused mostly on classification.Source: §3.2 · pp. 58–59

3.3 Methodology

  • The methodology perturbs both natural-language docstrings and code, then evaluates robustness with newly proposed metrics.The introduction describes metrics based on execution that aggregate correctness across randomized transformations and transformation types.Source: §3.1 · p. 57, §3.3 · p. 59

3.3.1 Problem Formulation

  • ReCode tests end-to-end left-to-right function generation under natural prompt transformations that preserve meaning and are likely in practice, excluding adversarial attacks requiring model feedback.The authors exclude such attacks because controlling their naturalness is difficult and they often require higher computational cost.Source: §3.3.1 · pp. 59–60

3.3.2 Natural Transformations on Docstrings

  • For docstrings, the authors select ten character-, word-, and sentence-level NL-Augmenter transformations likely to preserve semantic similarity, while noting unconstrained application can produce low-quality prompts.One example is changing “Create a list a[][]” into “Create a list [a][]” through character-level perturbation.Source: §3.3.2 · pp. 60–61
  • For docstring perturbations, tree-sitter parses the prompt and canonical solution so existing function, variable, and type names are excluded; the benchmark uses ten customized NL-Augmenter transformations.Source: §3.3.2 · p. 61

3.3.3 Natural Transformations on Function Names

  • Perturbing function names also reduces code-generation model performance.Source: §3.3.3 · p. 61

3.3.4 Natural Transformations on Code Syntax

  • For partial-code syntax perturbations, ReCode uses dead-code insertion, operand swaps, and equivalent for/while switches.The transformations must remain syntactically correct and preserve semantic meaning.Source: §3.3.4 · p. 63
  • Variable renaming replaces the most frequent variable using CodeBERT predictions, a NatGen-style VAR_0 name, or a random name with alphabetic and numeric characters.The first strategy tends to yield more natural names, though the other two are also plausible.Source: §3.3.4 · p. 63

3.3.5 Natural Transformations on Code Format

  • Code-format perturbations include inserting newlines, switching spaces and tabs for indentation, splitting long lines, and converting docstrings to comments.Source: §3.3.5 · pp. 63–64

3.3.6 Evaluation Metrics

  • ReCode proposes RPs@k, RDs@k, and RRs@k to assess worst-case pass performance, relative performance drops, and changes in correctness, and recommends reporting all three.High RPs@k does not necessarily imply low RDs@k or RRs@k.Source: §3.3.6 · pp. 64–66

3.4 Evaluation

  • The evaluation uses HumanEval and MBPP to assess robustness across CodeGen, InCoder, and GPT-J model architectures and sizes.The perturbations and metrics are described as applicable to any code-generation datasets and models.Source: §3.4 · p. 66

3.4.1 Code Generation Robustness Evaluation

  • Tables 3.3 and 3.4 report general perturbation performances for the evaluated models.Source: §3.4.1 · p. 66
  • On HumanEval, 6B CodeGen models outperform same-size peers on nominal performance and strict RP5@1, but underperform on relative robustness metrics RD5@1 and RR5@1.Source: Table 3.3 · p. 67, §3.4.1 · pp. 67–68
  • CodeGen-mono 2B to 16B improves average RP5@1 from 0.174 to 0.217, while average RR5@1 rises from 27.90% to 35.91%, suggesting larger models may overfit under perturbations.Source: §3.4.1 · p. 68
  • Syntax perturbations cause the largest performance drops across HumanEval and MBPP, while models are more robust on HumanEval than style-variable MBPP.Source: §3.4.1 · pp. 68–69

3.4.2 Ablation Study

  • Larger s tests stronger perturbations and broader practical failures, with diminishing gains; s = 5 balances evaluation strength and cost, while docstring and syntax categories benefit more from larger s.Source: §3.4.2 · p. 69
  • Across k, RD@k remains stable while RR@k increases because higher k raises nominal pass@k and RP@k but can change perturbed-dataset results.Source: §3.4.2 · p. 70

3.4.3 Perturbation Sample Quality

  • Human ratings show a 14% average naturalness drop for perturbed data, yet a majority of annotators judged semantics preserved for 90% of samples.Average similarity scores were 0.93 for docstrings and 0.81 for function names; CodeBLEU syntax and dataflow scores averaged 0.96 and 0.97.Source: §3.4.3 · pp. 71–72

3.5 Discussion

  • ReCode is a robustness benchmark for code-generation models built from over 30 natural transformations spanning docstrings, function names, syntax, and format.Source: §3.5 · p. 72
  • The discussion proposes general worst-case robustness metrics to provide a unified overview of model robustness performance.The metrics are proposed in the context of preserving semantic meaning after perturbations.Source: §3.5 · p. 72
  • ReCode is demonstrated on CodeGen, InCoder, and GPT-J using HumanEval and MBPP datasets and function-completion tasks derived from them.Source: §3.5 · pp. 72–73
  • Human evaluation confirmed that over 90% of ReCode perturbations preserve the original semantic meaning, with sentence similarity and CodeBLEU also supporting perturbation quality.The evaluation concerns perturbations designed to preserve semantic meaning.Source: §3.5 · p. 73

Chapter Overview … Chapter Overview

This section examines how generation diversity and self-reflection relate to model behavior, including Branching Factor, self-reflection, and efficient Shapley-value explanations. It also presents and evaluates AI Realtor for grounded, preference-aware persuasion and BACO for controlling the diversity–quality trade-off through base–aligned model collaboration.

Chapter Overview

  • The chapter overview connects probability concentration with reduced output diversity in aligned LLMs and describes BF as measuring the effective number of plausible next steps.The cited TMLR publication attribution is Yang et al. (2026).Source: Chapter Overview · p. 75

4.1 Introduction

  • BF is the exponentiated length-averaged entropy of a model’s output distribution, a unifying framework for quantifying probability concentration rather than a novel mathematical metric.It summarizes the effective number of viable next-token choices on average; the chapter overview describes it as token-invariant.Source: §4.1 · p. 76
  • Alignment tuning typically reduces Branching Factor (BF) by 2–5× overall and up to an order of magnitude at initial positions, leaving fewer branches for decoding to prune.The introduction links this reduction to aligned models’ insensitivity to decoding parameters.Source: §4.1 · p. 77
  • BF typically declines during generation, and Chain-of-Thought (CoT) shifts answer generation into later, lower-BF regions, producing more deterministic outcomes.A prefix-substitution intervention separates task-independent autoregressive narrowing from alignment’s effect on BF level and steepness; unexpected input can raise BF on demand.Source: §4.1 · p. 77
  • A short aligned-style prefix rapidly lowers a base model’s BF, suggesting alignment steers generation toward latent low-entropy subspaces rather than reshaping the model’s manifold.The result comes from the introduction’s nudging experiments.Source: §4.1 · pp. 77–78

4.2 Background

  • The realized sequence-level entropy is an unbiased estimator of the marginal entropy of the full generative distribution.The background notes the practical importance of this proxy because computing the full conditional entropy requires marginalizing over exponentially many prefix trajectories.Source: §4.2 · pp. 79–80
  • For variable-length generation, the framework pads early-terminated sequences with repeated special tokens while preserving their probability, allowing all generations to be treated as length N.The repeated tokens indicate termination, for example with EOS.Source: §4.2 · p. 80

4.3 Measuring LLM Branching Factor

  • BF maps an irregular effective tree to an equivalent balanced tree and measures average plausible next-token choices with a normalized, token-invariant metric.The balanced-tree branching factor is derived by equating its leaf count at depth N with the effective set size.Source: §4.3 · p. 81
  • Because total entropy is intractable, the estimator computes exact realized entropy for short sequences and uses NLL for long ones, avoiding naive Monte Carlo’s finite-sample underestimation.The NLL proxy is motivated by the convergence between length-averaged NLL and realized entropy for long horizons.Source: §4.3 · pp. 81, 83
  • As output length increases, NLL–realized-entropy deviation vanishes, and estimator standard deviation diminishes rapidly within the first 10 tokens.The figure reports these trends for sampled Llama-3-8B-Instruct outputs across multiple datasets.Source: §4.3 · p. 82, Figure 4.2 · p. 84

4.4 Benchmarking and Attributing Branch Factors

  • The benchmark covers Llama-2 and Llama-3 base and aligned models, comparing BF across reasoning, controlled, news, creative, and random-string generation tasks.The task set includes MMLU, COGNAC, BBCLATEST-NEWS, CREATIVE STORYGEN, and a synthetic RANDOM STRINGS task.Source: §4.4 · p. 84
  • The experiments vary prompt complexity, alignment tuning, model size, and model generation, with sampling set to p=0.9 and T=1.0.Prompt complexity controls input informativeness; alignment status distinguishes Instruct from Base models.Source: §4.4 · pp. 84–85

4.4.1 BF Dynamic in Generation Process

  • Base-model BF is often 2–5 times aligned-model BF, and BF generally declines during generation as an aggregate tendency, not a token-wise law.The decline also appears for random strings, where no semantic structure is available.Source: §4.4.1 · p. 86
  • At matched output lengths, reasoning-oriented models have lower BF than direct-answer aligned models, including at the earliest positions.This comparison indicates that their lower BF is not merely due to longer generations.Source: Figure 4.4 · p. 87, §4.4.1 · p. 87

4.4.2 Why Does BF Decrease? Disentangling Self-Narrowing from Alignment

  • Replacing a model’s own prefix with external random tokens surges BF, after which autoregression lowers it again; alignment sets the absolute level and recollapse speed.The intervention separates self-conditioning from alignment effects.Source: Figure 4.5 · p. 88, §4.4.2 · p. 88
  • Adversarial or random-noise environment feedback raises BF relative to expected-progress feedback, with a smaller effect in more heavily aligned models.The authors describe this as an empirical pattern and do not claim a mechanistic explanation.Source: Figure 4.6 · p. 89, §4.4.2 · p. 89

4.4.3 Pareto Analysis of BF

  • 80%: Across tasks, alignment tuning consistently crosses or approaches this cumulative-impact threshold for BF reduction, surpassing other factors by a large margin.The analysis ranks alignment against model size, generation, and prompt complexity.Source: Figure 4.7 · p. 90, §4.4.3 · p. 91
  • For richer-input tasks, prompt complexity and model size are secondary factors; more context can increase BF, potentially when processing complex structures such as negated constraints.The prompt-complexity effect is not uniformly a reduction in BF.Source: §4.4.3 · p. 91
  • For open-ended Cognac and Story Generation tasks, generation G plays a more dominant role, particularly in improvements from Llama-2 to Llama-3.The authors suggest this shift likely reflects training on larger, more diverse datasets.Source: §4.4.3 · p. 91

Commitment

  • BF declines during generation and is lower in aligned models, providing a probabilistic account of their decoding insensitivity, reduced majority-vote variance, and performance cost from late branching.The authors describe this concentration of probability mass as a commitment to a specific reasoning trajectory.Source: Commitment · p. 92

4.5.2 Reduced Variance in Majority Voting

  • Low BF predicts smaller Majority@K variance, with long-CoT DeepSeek-R1-Distill models showing the smallest variance as they generate longer outputs and achieve the lowest global BF.The authors argue this stability is tied to a narrowed generation manifold rather than being merely an artifact of specific decoding hyperparameters.Source: Table 4.2 · p. 93, §4.5.2 · p. 94

4.5.3 Risks of Mid-Generation Forking

  • Resampling at later, lower-BF positions sharply reduces performance, suggesting that aligned models commit to reasoning trajectories and become more error-prone when forced to deviate.The experiment resamples DeepSeek-Distilled Llama-8B continuations at the 25th and 200th output tokens; the authors recommend branching early while BF remains high.Source: §4.5.3 · p. 95, Figure 4.8 · p. 95

4.6 How does Alignment Tuning Impact BF?

  • Nudging results support the hypothesis that a small number of stylistic tokens can steer a base model into the low-BF regime characteristic of aligned models.Most nudging occurs early in generation, and BF provides a per-step view of where a prefix induces concentration that sample-level metrics cannot resolve.Source: §4.6 · pp. 96–97

Which Training Stage Reduces BF Most?

  • At 7B, SFT reduces BF 42.3% versus DPO’s 20.5%; at 13B, DPO reduces it 25.6% versus SFT’s 14.9%, showing stage effects vary with scale and recipe.The OLMo-2 comparison is preliminary because intermediate checkpoints are not generally available and post-training schedules may interleave stages.Source: Figure 4.10 · p. 97, Which Training Stage Reduces BF Most? · p. 97
  • On RLHFlow/OpenRLHF, SFT lowers BF 4.05 →2.35; DPO and PPO further lower it to 2.17 and 2.19, while Iterative DPO avoids part of the shrinkage.The study treats this pipeline as a controlled case, not a universal algorithm ranking.Source: Figure 4.11 · p. 98, Which Training Stage Reduces BF Most? · p. 98
  • The authors suggest Iterative DPO’s online loop refreshes the reference policy and gathers preferences from updated outputs, potentially exposing underrepresented solution regions.They present this as a plausible explanation for preserving distributional breadth relative to fixed offline preference data.Source: Which Training Stage Reduces BF Most? · p. 98

4.7 Related Works

  • BF quantifies probability concentration across tasks without strong assumptions about output categories and identifies a shrinking distributional landscape during token-by-token generation.The framework is presented as a mechanism underlying symptoms such as aligned models’ reduced sensitivity to sampling hyperparameters.Source: §4.7 · pp. 99–100
  • Traditional lexical-diversity metrics are sensitive to vocabulary size and output length, and the paper reports that lexical diversity poorly correlates with BF and fails to measure concentration robustly.Source: §4.7 · p. 100

4.8 Discussion

  • In diversity-valued settings, low-BF models leave few viable alternatives for beam search, and off-path trajectories are often lower quality, so remedies should target training rather than decoding alone.The discussion notes that simply adjusting decoding parameters such as temperature is unlikely to restore diversity without severely degrading quality.Source: §4.8 · p. 100
  • Low BF may be beneficial on tasks with unique correct answers, while the paper’s primary contribution is diagnostic and its discussion focuses mainly on settings where output diversity is valued.Source: §4.8 · p. 101
  • BF’s downward trend over generation is a robust aggregate tendency, not a monotonic token-wise law, as conditioning on a growing prefix often makes the next-token distribution more concentrated.Source: §4.8 · p. 101
  • Unexpected context can raise BF temporarily, but autoregressive self-conditioning narrows it again; alignment separately lowers BF and steepens early narrowing.The discussion identifies injecting unexpected content as a local diversity intervention, not a lasting cure.Source: §4.8 · pp. 101–102
  • Alignment tuning reduces BF and homogenizes outputs, potentially reinforcing social biases and limiting creativity and novel ideas.The authors argue that quantifying probability concentration is a first step toward alignment techniques that preserve diversity and robustness.Source: §4.8 · p. 102

4.9 Limitations

  • BF is a scalar average that can obscure distinct generation-tree structures, so finer topology analysis requires positional or higher-order measures.Two example distributions have BF = 2 despite qualitatively different branching trees.Source: §4.9 · pp. 102–103
  • The hybrid estimator uses exact entropy for short sequences and an NLL approximation for longer ones, while finite-sample entropy estimates undercount rare high-entropy tails.Estimator variance reaches practical levels within roughly 20 tokens; absolute BF magnitudes are lower bounds, while base/aligned ratios are more robust to undersampling bias.Source: §4.9 · p. 103
  • Raw BF depends on tokenizer granularity, making direct cross-family comparisons inappropriate; within-family base/aligned ratios avoid this dependence.The paired models share a vocabulary, so their ratio is tokenizer-invariant.Source: §4.9 · pp. 103–104
  • Token-level BF is distinct from semantic uncertainty, but its sequence-level concentration is associated with reduced lexical and semantic diversity in aligned models.The per-prompt mapping from BF to semantic diversity remains an open question.Source: §4.9 · p. 104

Chapter Overview

  • The chapter evaluates self-reflection under a stricter setting that excludes external feedback.The overview frames this as a test of reflective capability without external cues.Source: Chapter Overview · p. 105

5.1 Introduction

  • In a single-round ChatGPT QA case study, self-reflection improved TruthfulQA (Lin et al., 2022) but reduced HotpotQA (Yang et al., 2018) performance.Its effectiveness depended on initial-answer confidence and human-rated difficulty: it was often harmful when initial answers were reliably correct, but beneficial on harder questions, and it reduced majority-voting tendencies.Source: §5.1 · p. 106
  • The study aims to help users decide when to use self-reflection.Source: §5.1 · p. 106

5.2 Self-Reflection Prompting

  • Single-Round Self-Reflection Verification (SR2V) excludes external feedback and prompt optimization, disallows iterative prompting, and uses exploration, reflection on each candidate, and revision.Candidates are sampled independently, then their response-reflection pairs are combined as input for the revised output.Source: §5.2 · p. 107

5.3 Preliminary Study: Does Self-Reflection Prompting Work Under SR2V?

  • On TruthfulQA, self-reflection outperforms exploration-only and standard prompting, but on HotpotQA its accuracy is about 4% lower than both; LLaMA-2 and Mixtral show similar results.HotpotQA uses manual assessment because exact match can mark semantically correct answers wrong.Source: §5.3 · pp. 108–109, Table 5.1 · p. 108

5.4 Why Self-Reflection May Not Work?

  • Reflection’s effects interact with question difficulty and response accuracy: it is mostly harmful when easy-question candidates are already correct, while harder questions retain benefits to higher accuracy.For Medium questions, reflection helps when most or all candidates are wrong and is often harmful when at least half are correct, except in the 75% response-accuracy bin.Source: §5.4 · p. 110

5.5 Error Analysis via Artificial Response

  • With artificial response pairings, reflection’s benefit-to-harm turning point occurs around 50% response accuracy for Hard, 30% for Medium, and 20% for Easy questions.Benefits are concentrated at low response accuracy; the study identifies confidence in initial accuracy as a major contributor, with benefits persisting to higher accuracy on harder questions.Source: §5.5 · pp. 111–112
  • The TruthfulQA analysis suggests self-reflection may work better when good initial responses are less frequent, though the dataset does not support exact quantification.The passages present this as speculation based on the results.Source: §5.5 · p. 112

5.6 Effects on Majority Voting

  • Without self-reflection, outputs align strongly with majority voting when candidate accuracy is high or low; self-reflection significantly reduces that tendency across RA levels.The analysis uses K = 10, including artificially generated responses; at higher RA, reduced majority voting can have a harmful effect.Source: §5.6 · pp. 112–113

5.7 Discussion

  • Guidelines recommend self-reflection at low RA, or at intermediate RA when questions are difficult; without RA estimates, response consistency serves as a proxy.Highly consistent responses suggest little benefit; low consistency, especially on harder questions, may indicate benefit.Source: §5.7 · pp. 113–114

5.8 Conclusion

  • Self-reflection mainly benefits cases with unreliable initial answers, with more persistent gains on harder questions, and reduces majority-voting behavior.The conclusion reports mixed results from a stringent single-round, multi-response evaluation.Source: §5.8 · p. 114

Limitations

  • The evaluation uses one ChatGPT snapshot and two datasets; artificial-response results are a rough estimate based on ten samples, and broader model and dataset evaluation remains future work.The authors report verifying their conclusions on LLaMA-2 and Mixtral, while noting that results may vary across model versions and systems.Source: Limitations · pp. 114–115
  • RA depends on access to ground-truth answers, and an initial GPT-4 attempt to estimate it was unsatisfactory.This limits RA’s practical utility and is identified as a topic for future research.Source: Limitations · p. 115

Chapter Overview

  • The chapter proposes an amortized model that predicts input-feature Shapley values without additional model evaluations, achieving speedup while maintaining stability.The chapter frames this as a response to the high evaluation cost of computing Shapley values for large pretrained models.Source: Chapter Overview · p. 116

6.1 Introduction

  • Sampling-based Shapley explanations vary substantially across random seeds and require more than 2,000 perturbation samples for stability; KernelSHAP took about 183 seconds per Yelp-Polarity instance.This sensitivity and cost hinder reliable interpretation of model predictions.Source: §6.1 · pp. 117–118
  • On Yelp-Polarity with an A100, the model cuts computation from about 3.47s per instance to less than 50ms, 60 times faster than baseline methods.With as few as 5000 training instances, it achieves high estimation accuracy.Source: §6.1 · p. 118
  • The amortized estimator produces stable estimates across random seeds and can be paired with SVS for instance-specific inference-time adaptation when more computation is available.Source: §6.1 · p. 119
  • The authors assess explanation functionality through feature-selection and domain-calibration case studies using estimated explanation scores.Source: §6.1 · p. 119

6.2 Related Works

  • Changing random seeds can yield weakly correlated Shapley estimates unless many perturbation samples are used, which incurs high computational cost.Source: §6.2 · p. 120
  • Compared with CXPlain (Schwab and Karlen, 2019) and FastSHAP, the method assumes no causal relation between features and outputs and targets text's discrete-token features.Source: §6.2 · p. 120

6.3 Background

  • For text classification, local explanations assign token-level contributions, while KernelSHAP uses linear regression to estimate Shapley scores for all tokens at once.More perturbation samples require more model evaluations.Source: §6.3 · pp. 121–122
  • The background identifies the number of model evaluations as the main performance bottleneck, with single-instance evaluation potentially costly for large Transformer models.Source: §6.3 · p. 123

6.4 Stability of Local Explanation

  • 0.16 Spearman correlation across 25–200 perturbation samples indicates low ranking stability; exceeding 0.40 generally requires thousands of evaluations and roughly 33.40 seconds per instance.Increasing the sample count improves stability but can raise runtime from one second to two minutes per instance.Source: Table 6.1 · p. 124, Table 6.2 · p. 124, §6.4 · p. 125
  • Low MSE does not ensure reliable explanations: score MSE only weakly correlates with ranking stability, and explanations can still produce distinct rankings.For Yelp-Polarity, the reported correlation is R = −0.41 with p < 0.05, which the passage describes as not significant.Source: §6.4 · p. 125
  • Longer inputs degrade ranking stability from 20 tokens onward, and the same trend appears across Yelp-Polarity and MNLI.The section notes that this makes stable explanations challenging for many NLP tasks with sentences longer than 20 tokens.Source: §6.4 · p. 125, Figure 6.3 · p. 126
  • Uniformly sampling n-grams as input length grows reduces the chance of sampling some n-grams, increasing variance in observed model outputs.The proposed explanation for instability is that important tokens or n-grams may be missed in perturbation samples.Source: §6.4 · p. 126

6.5 Amortized Inference for Shapley Values

  • An amortized model predicts explanation scores from an input without perturbation-sample model evaluations for each test example.It is trained on precomputed reliable reference scores, then makes a single inference on each new input.Source: §6.5 · pp. 126–127

6.5.1 Better Fit via Local Adaption

  • Amortized explanations can initialize SVS scores to speed evaluation versus random initialization, while applying SVS afterward can improve amortized explanations by identifying missed tokens.Additional SVS sampling can identify important low-frequency tokens not captured by the amortized method.Source: §6.5.1 · p. 127

6.6 Experiments

  • Experiments use Yelp-Polarity and MNLI validation data, compare against an adapted FastSHAP baseline, and evaluate explanation quality in two downstream applications.The authors report that adapting FastSHAP to text is non-trivial: memory constraints limit perturbation sampling to 32 per instance, making its KS-32 reference scores unstable.Source: §6.6 · p. 128

6.6.1 Shapley Values Approximation

  • Spearman correlation reaches 0.61 > 0.60 on Yelp-Polarity and 0.42 > 0.40 on MNLI; inference takes 50 milliseconds, 40–60 times faster than KS-200 and 400–600 times faster than KS-2000.The amortized model outperforms KS-200 and FastSHAP, but not the more time-consuming KS-2000/8000; SVS-25 is listed as an upper bound. MSE is reported for reference because it correlates poorly with ranking stability.Source: Table 6.3 · p. 129, §6.6.1 · p. 129

6.6.2 Sensitivity Analysis

  • Across training runs using the same data, output correlations are 0.77 on MNLI and 0.76 on Yelp-Polarity; stability improves with more training samples.The sensitivity study reports a standard deviation below 1e-2 and stability even with 10% of the data.Source: §6.6.2 · p. 131, Table 6.4 · p. 131

6.6.3 Local Adaption

  • On Yelp-Polarity and MNLI, local adaptation improves the amortized model’s approximation, while using it to initialize SVS improves sample efficiency across datasets.The comparison of SVS-X and Adapt-X supports the sample-efficiency finding.Source: §6.6.3 · p. 131, Table 6.5 · p. 132

6.6.4 Domain Transferability

  • The amortized model achieves approximately 0.50 Spearman correlation with ground-truth SV (SVS-25) at 0.017 seconds per instance, versus KS-100’s 0.46 at 1.6 seconds; KS-200 correlates slightly better but takes significantly longer.Source: §6.6.4 · p. 132

6.7 Evaluating the Quality of Explanation

  • In feature-selection evaluations, the amortized model is more faithful than KS-200 but less faithful than KS-2000/8000 and SVS-25, yielding a better efficiency-faithfulness trade-off.Faithfulness is assessed by masking highly ranked tokens and measuring the resulting model-output or accuracy changes.Source: §6.7 · p. 133
  • The amortized model’s explanations yielded better calibration accuracy on out-of-domain data than KS-2000.The comparison is reported for calibration of a fine-tuned MNLI model to MRPC.Source: §6.7 · p. 134, Table 6.6 · p. 134

6.8 Discussion

  • The discussion identifies unstable explanation scores on long inputs and proposes an amortized model trained on precomputed reference scores for efficient estimation.The approach is motivated by different instances sharing similarly important features.Source: §6.8 · p. 135

Limitations

  • The authors state that the approach can extend to black-box post-hoc explainers such as LIME, although experiments covered BERT-based models; they also describe it as applicable to other LLMs.They report that training can use model outputs and thousands of examples.Source: Limitations · p. 135
  • Exact Shapley-value computation is computationally prohibitive for lengthy LLM inputs because it requires exponentially many perturbation evaluations, so training uses SVS-25 as an approximation.The passage describes SVS-25 as a reliable approximation.Source: Limitations · pp. 135–136

Chapter Overview

  • AI Realtor separates factual grounding from personalization, generating persuasive copy that is factually rigorous.Its architecture enforces grounding before style.Source: Chapter Overview · p. 138

7.1 Introduction

  • The introduction describes a three-module agent: grounding identifies credible selling points, personalization tailors content to preferences, and marketing maintains factual consistency.The paper also reports a real-world evaluation using a Zillow dataset and an experimental house-search website.Source: §7.1 · p. 139
  • AI Realtor achieved a 70% win rate over human experts while maintaining comparable factual accuracy.The evaluation used a real-estate marketing testbed and recruited potential home buyers to assess generated content.Source: §7.1 · p. 139

7.2 A Benchmark for Grounded Persuasion

  • The study presents real-estate marketing as a controlled testbed for persuasion, where high-stakes decisions call for compelling language grounded in facts.Persuasiveness is subjective and depends on individual preferences and contexts.Source: §7.2 · p. 140
  • The persuasion benchmark uses 50k+ real-world listings, immersive interaction, dynamic preference elicitation, pairwise buyer choices, Elo scores, and metadata-based factuality checks.Source: §7.2 · p. 141

7.3 An Economic Scaffolding of Copywriting

  • The economic scaffold maps factual attributes to abstract signaling features, then uses selected features and buyer preferences in prompts to steer grounded copy without claiming optimal signaling.The objective is a design abstraction, not a claim that the deployed model solves a specified optimization problem.Source: §7.3 · pp. 142–144
  • AI Realtor composes real-estate descriptions from multiple levels of marketing information through grounding, preference elicitation, and factual feature-selection modules.Source: §7.3 · pp. 144–145

7.4.1 Grounding Module: Predicting Credible Features for Marketing

  • The Grounding Module must predict high-level marketing features from raw attributes because the dataset lacks feature annotations, while natural language is open-ended.Source: §7.4.1 · p. 145
  • LLM-assisted schema construction with human validation yielded a compact feature representation, with around 60% model-human agreement comparable to human-human agreement.Most disagreements involved subjective features such as aesthetics and style; directly prompting LLMs produced redundant or incomplete feature sets.Source: §7.4.1 · pp. 145–146
  • 69.39% accuracy and 67.43% F1 were achieved by the feature-mapping model on a random 4:1 train-test split in a noisy multi-label setting.The strongest baseline achieved approximately 59% F1; subjective features such as style and aesthetics remain ambiguous.Source: §7.4.1 · p. 146

7.4.2 Personalization Module: Aligning with Preferences

  • The personalization module elicits and rates users’ valued features, then combines individual ratings with population-level feature intensity to select features for generation.The resulting interpretable preference signal shifts which grounded features the agent emphasizes.Source: §7.4.2 · p. 147

7.4.3 Marketing Module: Capturing Surprisal via RAG

  • The marketing module uses RAG to provide localized feature information, grounding persuasive generation in factual evidence and problem contexts.In a participant’s example, a description stood out by noting the rarity of ample storage and built-in cabinetry in similarly priced listings.Source: §7.4.3 · pp. 147–148

7.5.1 Evaluation by Human Feedback

  • AI Realtor receives higher human preference ratings than human experts and other model variants, measured by Elo ratings and win rates.The evaluation compares model outputs using pairwise human feedback.Source: §7.5.1 · pp. 148–149, Figure 7.3 · p. 149

7.5.2 Evaluation through AI Feedback

  • AI-simulated buyer feedback achieved 61.6% accuracy across users, with performance above 50% for 79.2% of users.The authors characterize it as potentially useful for preliminary diagnostics, but its accuracy is too low for reliable replacement of human evaluation.Source: Figure 7.4 · p. 151, §7.5.2 · p. 152
  • AI feedback showed high and increasing user-wise accuracy variance as more examples were provided, underscoring difficulty in dynamically predicting preferences.The authors note that the increase is expected with fewer data points, but still highlights the challenge of preference simulation.Source: §7.5.2 · p. 152
  • AI-simulation errors included length bias, confident choices despite tie comments, and selections reflecting emergent or contradictory preferences.The manual analysis excluded 56.1% of error cases that lacked clearly explainable patterns.Source: §7.5.2 · p. 153
  • Other simulation errors involved correct predictions emerging only after sufficient in-context samples and apparently random predictions when buyer information was insufficient.The authors suggest collecting more selections per buyer or improving preference elicitation as possible mitigations.Source: §7.5.2 · p. 153

7.5.3 Hallucination Checks

  • GPT-4o's model ranking aligned closely with human evaluations for hard attributes but diverged for soft attributes, exposing verification challenges for loosely matched facts.Figure 7.5 reports the faithfulness-check comparison; the text identifies divergence on Xsoft as a challenge in verifying loosely matched attributes.Source: Figure 7.5 · p. 154, §7.5.3 · p. 155
  • Models tended to describe factual attributes accurately when mentioned, but human realtors and SFT showed more hallucination, often through vague descriptions of hard attributes.The authors note that whether vague attribute descriptions count as hallucination is debatable, although some buyers complained about this language.Source: §7.5.3 · p. 154
  • Human and GPT-4o evaluations find AI Realtor faithful on Xhard and comparable on Xsoft, suggesting minimal hallucination risk; human evaluators judge its descriptions as trustworthy as humans.The supplied passages do not provide the comparison target or numerical results.Source: §7.5.3 · p. 155

7.5.4 Qualitative Analysis

  • Users favor grounded, market-relevant surprisal, but confusing statistical claims expose a trade-off between precision and clarity.In one case, a “top 2% for amenities” claim confused a user.Source: §7.5.4 · pp. 155–156
  • AI Realtor addresses omissions and poor formatting in human-written descriptions by providing comprehensive outputs in a professional, fact-grounded tone.Users objected to a human description missing size or location and another written entirely in capitals.Source: §7.5.4 · p. 156
  • Users differ in preferred description length and style, supporting the need for the Personalization Module to adapt content to user profiles.Some prefer concise descriptions or plain amenity lists, while others favor detail or evocative language.Source: §7.5.4 · p. 157

Adaptive Tone across Market Segments

  • AI Realtor shifts from security and efficiency for a $110k listing to luxury and openness for a $1.8M listing, tailoring language to buyer needs across market segments.The passages characterize this as moving beyond simple template-filling to semantic adaptation.Source: Adaptive Tone across Market Segments · p. 158

7.6 Related Work

  • In contrast with related persuasion and marketing studies, this work develops a full agentic framework for automated marketing and receives higher human preference ratings than a supervised fine-tuning baseline.The framework spans learning domain-expert knowledge and crafting localized features.Source: §7.6 · p. 159

7.7 Discussion

  • The paper presents a persuasive-language framework integrating economic signaling concepts into agentic LLM design and reports that it outperforms professional human-written listings on a controlled real-estate benchmark.Its design prioritizes factual grounding, mapping raw attributes to a compact set of high-level features.Source: §7.7 · p. 159
  • Evaluation relies on high-quality human feedback; automated LLM evaluators show promise for factuality but remain unreliable for nuanced qualities such as persuasiveness.The authors call for more sophisticated evaluation benchmarks.Source: §7.7 · pp. 159–160
  • Empirical validation is concentrated in Chicago, and persuasion’s cultural and economic sensitivity leaves stability across regions and demographics unverified.Generalization to less-structured inputs or domains with different persuasive norms, such as brand marketing or legal arguments, also remains an important challenge.Source: §7.7 · p. 160
  • Future work includes domain-specific compliance constraints, broader persuasion theories as controllable modules, larger datasets, new copywriting domains, and real-world A/B testing.The discussion also identifies balancing factuality and expressiveness as a direction for further work.Source: §7.7 · p. 160

7.8 Conclusion

  • Human evaluations found AI Realtor more persuasive than professional realtors and baseline LLMs while maintaining high factual fidelity.Users valued its balance of evocative language and concrete details, with tone adapted across market segments.Source: §7.8 · p. 161

7.9 Ethics Statement

  • The ethics statement frames grounded persuasion around verifiable facts to mitigate misinformation and manipulation, reporting factual accuracy comparable to or exceeding human experts.It presents this high-stakes real-estate work as a foundation for research on ethical guardrails for strategic language models.Source: §7.9 · p. 161

Chapter Overview

  • The chapter presents BACO as an inference-time framework using uncertainty and content-based routing to improve diversity and quality with a controllable trade-off.It dynamically combines a base LLM with its aligned counterpart.Source: Chapter Overview · p. 163

8.1 Introduction

  • BACO combines a base model’s diversity with its aligned counterpart’s quality through token-level routing in a single decoding pass, without fine-tuning or prompting.The framework aims to jointly optimize diversity and quality, surpassing either model alone through collaboration.Source: §8.1 · p. 165
  • BACO addresses the diversity-quality trade-off by collaborating between base and aligned models, using lightweight dynamic token routing to jointly optimize both qualities.The routing operates in one decoding pass and requires neither fine-tuning nor prompting.Source: §8.1 · p. 165
  • A fixed diversity–quality operating point is unsuitable because tasks and user preferences vary; BACO is designed to adjust generation along this spectrum on demand.The preliminary framing treats the goal as approaching the Pareto frontier, where improving diversity requires sacrificing quality and vice versa.Source: §8.1 · p. 165, §8.2 · p. 169
  • The evaluation spans instruction following, dialogue, and creative writing, comparing 11 diversity metrics with 2 quality metrics and including long-form and human evaluations.The chapter also cross-checks quality on mathematical reasoning and verifiable instruction-following tasks.Source: §8.1 · p. 166
  • BACO’s best router achieves a 21.3% improvement over the strongest inference-time baselines, with larger gains on semantic diversity metrics and consistent results across tasks, metrics, and human evaluations.The experiments report a new state-of-the-art in optimizing the diversity–quality trade-off at inference time.Source: §8.1 · p. 166
  • BACO combines a base model and its aligned counterpart through inference-time token-level collaboration and lightweight routing strategies to produce diverse, high-quality outputs.An adjustable routing threshold tunes the models’ contributions during decoding; complementary routing strategies add control across diversity dimensions.Source: §8.1 · pp. 165, 167

8.2 Preliminary

  • 3.15x more diverse outputs came from the base model, while the aligned model had 5.95x higher quality in the preliminary experiment.Diversity was measured by semantic-equivalent output clusters and quality by average reward per output.Source: Table 8.1 · p. 168, §8.2 · p. 169

8.3 BACO: Base-Aligned Collaboration for Diversity and Quality

  • Prior collaboration approaches use a small fraction of aligned-model tokens to recover base-model task performance comparable to the aligned model, but primarily focus on improving base-model quality.The passage attributes this approach to Fei et al. (2025).Source: §8.3 · p. 169
  • BACO routes generation between a base model for diversity and its aligned counterpart for quality, switching only at word boundaries to accommodate differing tokenizations.A lightweight router selects which model generates the next token.Source: §8.3 · p. 170

8.3.1 Routing Strategy Design

  • Routing strategies use a threshold γ to shift the diversity-quality balance and draw on token-distribution uncertainty or predicted-token semantic roles.Higher γ favors the base model and diversity, while lower γ favors the aligned model and quality; logit-based strategies use probability or entropy, while content-based strategies route by token role.Source: §8.3.1 · p. 171
  • Combining logit-based and content-based routing signals yields the strongest performance among the strategies described.For example, BACO-P-FC prioritizes function-word routing before probability-based decisions.Source: §8.3.1 · p. 172

8.4 Experimental Setup

  • Experiments cover instruction following, dialogue, and creative writing, comparing two open-weight base/aligned model pairs against inference-time baselines.The datasets are NoveltyBench, WildChat, and Narrative-Discourse; model pairs are Llama-3-8B/Llama-3-8B-Instruct and Olmo2-7B/Olmo2-7B-Instruct.Source: §8.4 · pp. 172–173

8.4.1 Automatic Evaluations

  • Automatic evaluation uses 11 established diversity metrics and 2 quality metrics to assess the diversity-quality trade-off.Figure 8.2 illustrates Coverage, the area under a method’s trade-off curve, and Dominance, its proportion of the global Pareto frontier.Source: §8.4.1 · p. 174, Figure 8.2 · p. 174

8.5.1 Overall Performance

  • Across datasets and metrics, BACO improves Coverage by 0.142 and achieves 32.7% Dominance; Dominance reaches 40.5% in diversity, with BACO outperforming baselines across all semantic and most lexical spaces.Source: Table 8.2 · p. 175, §8.5.1 · pp. 177–178

8.4.1 Automatic Evaluations

  • Coverage measures trade-off area and Dominance measures contribution to the global Pareto frontier; both are averaged across 11 × 2 spaces, separating lexical and semantic diversity.Diversity metrics are paired with quality metrics to produce method-level indicators.Source: §8.4.1 · pp. 176–177

8.5 Results

  • The results section compares BACO with inference-time baselines, analyzes routing strategies and model-pairing ablations, and evaluates long-form diversity and human judgments.Source: §8.5 · p. 177

8.5.1 Overall Performance

  • Across IFEval and GSM8K, BACO achieves higher diversity at matched quality or accuracy than the aligned baseline; on GSM8K, base–aligned collaboration maintains accuracy while improving diversity.Naive temperature scaling degrades GSM8K accuracy without delivering meaningful diversity.Source: §8.5.1 · p. 179, Figure 8.4 · p. 179
  • 0.681 vs. 0.320 Coverage and 72.8% vs. 27.2% Dominance show BACO-P-PUNC outperforming the aligned baseline on MT-Bench across diversity and quality metrics.Source: §8.5.1 · p. 179
  • 15% of BACO generations fall below 0.6 similarity, versus 8% for Min-p and 5% for Top-p, with BACO’s threshold selected to match Min-p quality.The distribution is flatter, with more generations in lower-similarity bins.Source: Figure 8.5 · p. 180, §8.5.1 · p. 180

8.5.2 Router Performance Comparison

  • The -RAND router’s aggregate gains concentrate on lexical metrics, while its semantic performance collapses, including 0% Dominance on Semantic Entropy.The passage relates this pattern to random or nonsensical text.Source: §8.5.2 · p. 180
  • Despite extensive prompt engineering, -JUDGE underperforms simpler heuristic routers while incurring substantially higher computational cost.The prompt-based router is inspired by multi-agent systems (Talebirad and Nadiri, 2023).Source: §8.5.2 · p. 181
  • Single-strategy -P and -FC trade weaker aggregate performance for semantic-diversity gains and both outperform -RAND on Cosine Dissimilarity and Semantic Entropy.Unguided switching injects surface-level randomness rather than meaningful semantic diversity.Source: §8.5.2 · pp. 180–181
  • Combining complementary routing strategies, especially -P-PUNC, gives BACO its strongest controllability, with high Coverage and Dominance across lexical and semantic spaces.The contribution is the router-robust collaboration framework, not any single router.Source: §8.5.2 · pp. 181–182
  • Human judges consistently prefer BACO’s best variant, -P-PUNC, to the aligned-model baseline across all evaluated aspects on NoveltyBench and WildChat.The reported preference encompasses perceived diversity, quality, and creativity.Source: §8.5.2 · p. 182, Table 8.5 · p. 182

8.5.3 Base-Aligned vs. Aligned-Aligned Collaboration

  • On NoveltyBench, base-aligned collaboration significantly outperforms aligned-aligned collaboration in both Coverage and Dominance, supporting the need for model complementarity.The comparison uses the same -P-PUNC router; the AACO table reports little improvement in the diversity–quality trade-off, particularly semantic diversity.Source: Table 8.4 · p. 181, §8.5.3 · pp. 182–183

8.5.4 Long-Form Diversity Evaluation

  • Long-form creative writing is evaluated with a discourse-level framework that captures plot structure and affective dynamics rather than surface lexical or single-vector semantic variation.The approach follows Tian et al. (2024).Source: §8.5.4 · p. 183

Appendix G.8.1). … SITTEST: TESTING SITUATIONAL UNDERSTANDING IN CHATGPT

These parts frame Grounded Alignment as a response to weak grounding in context and generation, and cover control approaches including BACO and Exploratory Annealed Decoding for exploration in reinforcement learning. They also describe an OUD self-disclosure project, from annotation and model evaluation to classification results, error analysis, and limitations.

Appendix G.8.1).

  • Figure 8.6 shows BACO achieves substantially higher turning-point and arousal diversity than the aligned baseline at comparable quality, extending to long-form generation.Example outputs are provided.Source: Appendix G.8.1 · p. 183

8.5.5 Human Evaluation

  • Despite near-identical automatic quality scores (aligned: 5.93; BACO: 5.85), humans rated BACO higher on NoveltyBench (4.04 vs. 2.83) and WildChat (3.83 vs. 3.44).BACO also won on diversity across format and content (Fleiss’ κ=0.268 overall); agreement was Pearson r=0.816 and ICC(2, k)=0.907.Source: §8.5.5 · p. 183
  • BACO’s outputs were judged most creative for 79.6% of NoveltyBench prompts and 61.8% of WildChat prompts (Fleiss’ κ=0.485).The authors suggest the improved diversity–quality trade-off yields outputs people find more creative, while leaving deeper investigation for future work.Source: §8.5.5 · p. 184

8.6.1 Contribution Distribution and Switching Frequency

  • Across three tasks, base-model contribution and switching frequency decrease over generation; dynamic, position-aware thresholds are proposed as a mitigation.The pattern may be less suitable for list-structured tasks that benefit from uniform exploration over time.Source: §8.6.1 · p. 184

8.6.2 Failure Mode: Inherent Early Stopping

  • More diversity-oriented routing can cause emergent premature end-of-sequence termination, risking truncated continuations but also guarding against repetition or off-topic text.Because truncations are detectable by length, a simple restart strategy offers a mitigation.Source: §8.6.2 · pp. 184–185

8.6.3 Future Work

  • BACO is framed as a preliminary “breadth thinking” mode that explores a wide, validated space of perspectives, with controllable user-facing diversity beyond temperature scaling.The proposed interface could let users shift along the diversity–quality spectrum.Source: §8.6.3 · p. 185

Chapter Overview

  • EAD implements an explore-at-the-beginning, exploit-at-the-end strategy by annealing sampling temperature from high to low.The chapter overview grounds the approach in the idea that early tokens shape a sequence’s semantic direction.Source: Chapter Overview · p. 186

9.1 Introduction

  • EAD improves sample efficiency over temperature sampling across GRPO, DAPO, and EntropyMech on small and larger models, and tuned schedules further enhance test-time generation quality.Its strategy explores early, when tokens shape semantic direction, and reduces exploration later to support coherent completions close to the target policy.Source: §9.1 · p. 188

9.2 Preliminary

  • Learned reward models can reward plausible but incorrect solutions, whereas RLVR uses verifiable rule-based feedback, such as binary correctness, to optimize the policy (DeepSeek-AI et al., 2025).A large pass@1-to-pass@k gap suggests some prompts have essentially zero underlying accuracy.Source: §9.2 · pp. 189–190

9.3 Sequential Exploration: Explore Early, Exploit Late

  • Because expected entropy tends to fall as context grows, effective exploration is most beneficial early, while fixed temperatures trade diversity against quality.The entropy trend is empirically illustrated for Llama-3-8B-Instruct on MMLU; high temperatures can yield nonsensical tokens and hallucinations, while low temperatures limit novel solutions.Source: §9.3 · pp. 191–192, Figure 9.1 · p. 192
  • Early-branched trajectories consistently outperform later-branched ones on MMLU Maj@k, supporting early exploration and late exploitation.The forking experiment compares branches from early and late generation steps.Source: Figure 9.2 · p. 193, §9.3 · p. 193

9.4 A Method for Sequential Exploration

  • EAD operationalizes “explore early, exploit late” with an annealed temperature schedule that starts above standard temperature (τ > 1).For RLVR, its decay rate is also global-step-aware.Source: §9.4 · p. 193
  • EAD starts each rollout above temperature 1 and progressively lowers token temperature; its decay rate adjusts with training step, while truncated importance sampling mitigates off-policy instability.The schedule’s decay rate controls how long exploration remains high; global-step-aware decay addresses increasing response lengths and helps avoid low-temperature repetition, off-topic drift, and verbosity. As a plug-and-play change, EAD adds negligible computational overhead and is compatible with RLVR pipelines and DAPO or GRPO.Source: §9.4 · pp. 194–195

9.5.1 Experimental Setup

9.5.2 EAD Improves RLVR Training

  • EAD (w/o TIS) consistently outperforms fixed-temperature baselines in Pass@16 on Llama and Qwen, while TIS is essential for stable gains on Worst@16.All reported Pass@16 and Worst@16 comparisons are significant; the figure notes that EAD’s gain diminishes over time and importance sampling can help sustain training.Source: §9.5.2 · p. 196, Figure 9.4 · p. 197
  • On Qwen-2.5-Math-7B, EAD’s performance gains remain significant, and its entropy can rise rather than decrease monotonically as training progresses.The entropy dynamics indicate a gradual transition out of local optima without external intervention such as tree search.Source: §9.5.2 · pp. 197–198, Figure 9.5 · p. 198
  • EAD achieves strong results with 4 or 8 rollouts, and still outperforms fixed-temperature baselines with 16 rollouts despite diminished relative gains.The optimal 8-rollout learning rates are 10^-6 for Llama-3.2-1B-Instruct and 2 × 10^-6 for Qwen-2.5-Math-1.5B.Source: §9.5.2 · pp. 198–199
  • EAD improves RLVR training “by a clear margin,” according to the excerpt.Source: §9.5.2 · p. 199

9.5.3 EAD is Compatible with Various RL Algorithms

  • EAD consistently outperforms fixed-temperature sampling with both GRPO and EntropyMech, supporting its use across different RL algorithms.The advantage diminishes when rollout count is scaled to 16, but EAD still beats traditional same-temperature sampling.Source: §9.5.3 · pp. 199–200, Figure 9.8 · p. 200

9.5.4 EAD Improves Inference-Time Scaling

  • Without fine-tuning, EAD improves majority-vote performance over fixed-temperature sampling for most sample counts, indicating it can discover higher-quality solutions at inference time.The evaluation used off-the-shelf Qwen-2.5 models and Majority@N.Source: Figure 9.9 · p. 201, §9.5.4 · p. 201

9.5.5 Ablation Study

  • Reverse annealing fails to outperform standard temperature sampling, supporting the need for EAD’s decreasing temperature schedule.Source: §9.5.5 · pp. 201–202, Figure 9.10 · p. 202
  • EAD is robust to a wide range of decay caps when the cap permits smooth annealing; a very small cap causes a performance drop as response lengths increase.A large initial decay can preserve schedule smoothness with α = 0, but training improves slowly because exploration is overly conservative.Source: §9.5.5 · p. 202

9.5.6 Stress Test: DAPO-17K Recipe for AIME24

  • On AIME24, EAD consistently outperforms standard temperature sampling under the DAPO recipe with α ∈ {5, 50}.The stress test used Qwen-3-8B-Base and capped response length at 8,192 tokens to accommodate GPU memory constraints.Source: §9.5.6 · pp. 202–203, Figure 9.12 · p. 204

9.6 Related Work

  • The paper presents a systematic analysis of sampling temperature and a purely sampling-level annealed schedule that explores before stabilizing answers, enabling new solutions and more stable training.Source: §9.6 · p. 205
  • The study introduces intra-sequence annealed temperature scheduling, varying temperature within a single sequence to enable nuanced exploration control and improve RLVR performance.Source: §9.6 · p. 206

9.7 Discussion

  • EAD begins generation at a high sampling temperature and gradually cools, promoting broad early exploration and precise completions while improving diversity and controlling variance.The discussion presents EAD as useful for RL training and as a test-time method for improving single- or multi-sample decoding quality without additional computation cost.Source: §9.7 · p. 206
  • The discussion identifies limited compute as preventing a systematic scaling study, leaves combinations with other exploration-promoting RLVR methods for future work, and uses a uniform rather than prompt-adaptive temperature schedule.Adaptive scheduling is described as difficult because prompt distributions may shift during iterative training, while tracking prompt-specific statistics would add overhead and complexity.Source: §9.7 · pp. 206–207

Chapter Overview

  • The OUD project uses explicit rationale spans to ground phase predictions, reporting gains in detecting subtle stages such as misuse and recovery.The annotation framework covers Medical Use, Misuse, Addiction, Recovery, and Relapse, with a Not Using category for posts not discussing the author’s OUD experience.Source: Chapter Overview · p. 208, §10.1 · p. 209

10.1 Introduction

  • The OUD study proposes a research-grounded annotation scheme and an expert- and novice-annotated dataset with class labels and highlighted text explanations.The scheme is designed for both expert and novice annotation, and the study evaluates zero-shot, few-shot, and supervised models.Source: §10.1 · p. 209
  • Explanations improved OUD self-disclosure detection, and smaller models fine-tuned on novice-annotated data with explanations outperformed zero- and few-shot large models, including GPT-4.Ablations indicate that explanation quality matters; analyses also examine label uncertainty and errors in cases of human agreement or disagreement.Source: §10.1 · p. 210

10.2.1 Data Collection and Annotation

  • The study collects posts from r/Opiates, r/OpiatesRecovery, and r/drugs to address the lack of large-scale public datasets for opioid-usage categories.Source: §10.2.1 · pp. 210–211
  • They retained 455 expert-labeled posts (13 few-shot, 442 test) and 2,086 novice-majority posts (1,936 train, 150 test; Fleiss’ κ = 0.47).The novice dataset began with 2,250 posts labeled by three qualified annotators; posts without a majority vote were excluded.Source: §10.2.1 · p. 213
  • Annotators selected the minimum span justifying each label; spans overlapped significantly in 63% of cases, while the longest explanation was selected in the other 37%.When overlapping explanations were very short, the longest explanation matching the majority label was chosen.Source: §10.2.1 · pp. 213–214

10.2.2 Disagreements in Annotation

  • Expert–novice disagreements were most frequent for Addiction–Recovery (22.35%) and Not Using–Misuse (19.35%), followed by Addiction–Misuse (12.90%).Medical Use–Misuse accounted for 10.75% and Recovery–Relapse for 8.60% of disagreements.Source: §10.2.2 · p. 214

10.3 Modeling Strategies

  • The study compared zero-shot, few-shot, and supervised OUD prediction, each with and without span-level explanations in the input.For zero-shot evaluation, it prompted instruction-tuned T0pp and used greedy search to generate labels.Source: §10.3 · p. 215
  • Few-shot prompts used 13 expert-annotated examples, while supervised training used novice annotations and evaluated on expert- and novice-annotated test sets.The supervised data comprised 1,936 training examples, 442 expert-annotated test examples, and 150 novice-annotated test examples.Source: §10.3 · pp. 216–217

10.4 Experiments

  • Small supervised models outperformed few-shot methods including GPT-4; T5-11B exceeded T5-3B by 8.4 accuracy points and 9.3 F1 points on average.The scaling pattern did not hold across model families: DeBERTa-v3-large outperformed T5-3B with and without explanations.Source: §10.4 · p. 218
  • T5-11B achieved 71.4% accuracy without explanations and 76.6% with explanations on expert data, making it the best overall model.The expert-set random and majority baselines were 17% and 35%, respectively.Source: §10.4 · p. 218

10.5 The Role of Explanations

  • Gold explanations produced the best results; leakage was 5.6% on expert data and 8% on novice data, so most explanations did not reveal the label.Both gold and silver explanations outperformed the random-sentence baseline.Source: §10.5 · pp. 219–220, Table 10.8 · p. 219

10.6 Error Analysis

  • Model errors clustered in “Not Using–Misuse,” “Recovery–Addiction,” and “Not Using–Addiction,” with confusion directions occurring asymmetrically.Both DeBERTa and T5-11B struggled with confusion on the expert evaluation dataset.Source: Table 10.9 · p. 220, §10.6 · p. 221
  • Adding explanations mostly reduced confusion between Not Using and Recovery, the two dominant labels and top-two most difficult categories in expert-annotated data; misclassification patterns also surfaced asymmetrically.Source: §10.6 · p. 221

10.7 Related Research

  • The study contributes a substance-use-grounded annotation scheme and explanation-infused models for identifying post-level self-disclosures.The scheme covers Medical Use, Misuse, Addiction, Recovery, Relapse, and Not Using, using expert and novice annotators.Source: §10.7 · p. 223
  • Unlike prior explanation work focused on faithfulness, this study examines extractive span-level explanations as supervision and their relationship to inference-time results.The passage relates the latter finding to Sun et al. (2022).Source: §10.7 · p. 223

10.8 Conclusions

  • The study defines a Reddit-post task for detecting self-disclosures across phases of opioid use and reports accurate explanation-infused models for use, misuse, addiction, recovery, and relapse.The authors state that explanations provide insight into annotator disagreement and model prediction errors, while annotation uncertainty affects performance.Source: §10.8 · pp. 224–225

Limitations

  • Reddit anonymity prevents characterizing the study population’s demographics or geographic extent, and the study identifies self-disclosures at the message level.Source: Limitations · p. 225
  • The thesis notes its opioid-related work lacks a global user-level temporal view and proposes studying Reddit users’ posts over time and improving prediction explanations.Its closing remarks characterize LLM understanding as fragile beneath fluent output and present grounding-focused tools and interventions as a constructive path forward.Source: Limitations · p. 225, §11.4 · p. 230

Ethical Considerations

  • Because Reddit data skew toward young, white American men, conclusions about OUD-stage explanations may not generalize to broader groups, especially people with OUD; the models should be used with human oversight.The thesis describes human reviewers assessing predicted labels and explanations.Source: Ethical Considerations · p. 226

11.1.1 Part I: Situational Grounding Evaluation

  • SitTest found ChatGPT fails to maintain coherent environment states over time, with non-persistent in-context memory and susceptibility to hallucinated updates.ReCode found code-generation models rely on surface heuristics rather than deep syntactic understanding and degrade under semantic-preserving perturbations.Source: §11.1.1 · p. 227

11.1.2 Part II: Generative Grounding Evaluation

  • Alignment tuning reduces the branching factor by 2–5 overall and up to an order of magnitude at generation start; Hindsight found self-reflection can reinforce hallucinations without external feedback.The thesis characterizes this as premature stylistic collapse and a lack of generative grounding.Source: §11.1.2 · p. 228

11.2 Key Contributions

  • The thesis presents diagnostic frameworks, Branching Factor, and practical grounding interventions as contributions, including efficient stable Shapley Value estimation for model-behavior analysis.It frames the overall goal as agents grounded in reality, not merely aligned to preferences.Source: §11.1 · p. 227, §11.1.2 · p. 228, §11.2 · p. 229

11.1.3 Part III: Dynamic Control for Human Alignment

  • AI Realtor shows retrieved context should be enforced before personalization, while BACo uses token-level routing to separate base-model exploration from aligned-model structural guidance.The thesis reports that this collaboration improves diversity and quality and provides controllable generation.Source: §11.1.3 · p. 228
  • Annealed Sampling improves reinforcement-learning sample efficiency by synchronizing exploration with probability-distribution cooling, while Addiction Support uses model-generated rationalization as a communication interface.The thesis says explicit reasoning assistance improves performance and trust in high-stakes health domains.Source: §11.1.3 · pp. 228–229

11.3 Future Directions

  • Future directions include persistent memory for state coherence across extended interactions and techniques that preserve output diversity while maintaining safety and helpfulness.The thesis also proposes multimodal grounding and broader use of grounding separated from personalization.Source: §11.3 · pp. 229–230

A.1 Distractor on Action … B.1.4 Natural Transformations on Code Format

This appendix section reports robustness and model-format comparisons for action and state tracking, alongside prompt handling and natural transformations of docstrings, function names, code syntax, and code format. It also describes implementation choices intended to preserve naturalness or semantic equivalence and reports a substantial State-EM gap between Vicuna-13B and ChatGPT under the stated setting.

A.1 Distractor on Action

  • Adding action distractors leaves State-EM less affected than counter-intuitive instructions, while Step-EM degrades somewhat but inconsistently in both natural-language and synthetic-language settings.The authors interpret this as evidence that ChatGPT can track interactions and select useful information at each step.Source: §A.1 · p. 231, Table A.1 · p. 231
  • Under two-shot prompting with action distractors, synthetic language yields better Step-EM than natural language, but the advantage diminishes as more in-context samples are provided.The authors suggest synthetic language may initially help distinguish useful information from distractors, while additional examples help the model learn that extraction directly.Source: §A.1 · p. 232

A.2 Partial Usage of Synthetic Languages

  • Applying synthetic language only to functors or arguments produces results very similar to using it throughout.The authors omit the detailed results from the appendix discussion to save space.Source: §A.2 · p. 232

A.3 Interaction with OpenAI API and Prompt Format

  • Traditional one-round prompts and faked multi-round prompts have usually less than a 5% performance difference, but the latter costs more tokens and can face response rejection, so the paper reports traditional-format results.The comparison covers Step-EM and State-EM in fully natural-language and fully synthetic-language settings.Source: §A.3 · p. 234

A.4 Post-Processing to Extract Answers from OpenAI API Responses

  • Because ChatGPT may not follow the requested output format, the authors use lightweight regular-expression postprocessing to extract logical statements and their functors and arguments.The described cleanup also removes leading and trailing spaces and newline characters.Source: §A.4 · p. 235

A.5 Comparison with other models

  • With five in-context samples and no counter-intuitive instructions, Vicuna-13B reaches only 32% State-EM in the synthetic-language environment, versus 94% for ChatGPT.The authors report that Vicuna’s outputs were often incomplete and hard to parse.Source: §A.5 · pp. 235–236

B.1.1 Natural Transformations on Docstrings

  • Docstring perturbations include German back-translation, low-rate character changes, case changes, and inflection variants constrained to the same part-of-speech tag.The reported settings include 0.05 randomness for character perturbation and a default 0.35 probability for changing characters to uppercase.Source: §B.1.1 · pp. 237–238
  • Docstring perturbations include typo-like character and whitespace changes, synonym operations, and tense conversions.Adjacent-character swaps use probability 0.05; synonym insertion excludes punctuation and stopwords and uses probability 0.35. Past-tense conversion is deterministic.Source: §B.1.1 · pp. 238–239

B.1.2 Natural Transformations on Function Names

  • Function-name transformations modify references consistently and include case-style changes and inflection constrained to the same part-of-speech tag.CamelCase converts to snake-case or vice versa; inflection changes noun number or verb tense while preserving the Penn Treebank POS tag.Source: §B.1.2 · pp. 239–240

B.1.3 Natural Transformations on Code Syntax

  • Code-syntax transformations modify code content in the prompt, including insertion of useless code, random selection of a for- or while-loop, and operand swapping in a binary logical operation.Comparisons to the nominal baseline use the same block of code.Source: §B.1.3 · pp. 240–241

B.1.4 Natural Transformations on Code Format

  • Code-format transformations alter indentation, line breaks, or documentation-comment style, with several transformations specified as deterministic.Examples include swapping spaces and tabs, splitting the longest line, converting Python docstrings to comments, and inserting empty lines at selected positions.Source: §B.1.4 · pp. 243–244

B.2 Limitations … C.9 Results on LLaMA-2-7b-chat

This appendix documents ReCode robustness evaluations, annotation and transformation checks, prompting procedures, and API-calling examples, alongside evaluations of robustness metrics and self-reflection. It reports LLaMA-2-7b-chat’s differing self-reflection effects across TruthfulQA and HotpotQA, its limits in assessing its comprehension, and related evaluation constraints.

B.2 Limitations

  • ReCode is limited to Python function-completion datasets based on HumanEval and MBPP, but its transformations could extend to other languages and code-related tasks.The benchmark evaluates robustness rather than mitigating its lack; the authors suggest using its perturbed data for training augmentation and future robust-training research.Source: §B.2 · p. 245

B.3 Failure Case Study under Perturbations

  • DeadCode insertion can mislead predictions, especially when completion is required immediately afterward; CodeGen-16B-mono may output only a newline after a meaningless loop.The passage also notes that an inserted return statement may mislead the model.Source: §B.3 · pp. 247–248
  • Newline insertion can cause wrong if-else predictions, while docstring typos can cause misunderstanding and incorrect completions.The newline perturbation is especially effective when the required code is complicated.Source: §B.3 · p. 248

B.4.1 Details for Human Evaluation

  • Human evaluation reached moderate agreement: Fleiss’ kappa averaged 0.52 overall and 0.36 for semantic and naturalness measurements on perturbed samples.The reported 95% bootstrap confidence intervals were [0.515, 0.528] and [0.358, 0.364]; annotators’ preferences, especially for naturalness, were not fully consistent.Source: §B.4.1 · p. 249

B.4.2 Sentence Transformers for Docstring/Function Names Similarity

  • Average similarity was 0.93 and 0.92 for docstring perturbations, and 0.80 and 0.81 for function-name perturbations on HumanEval and MBPP, respectively.Some function-name perturbations scored relatively low because short names lack sentence context, making minor word changes affect cosine similarity substantially.Source: §B.4.2 · p. 251

B.4.3 CodeBLEU Scores for Code Similarity

  • 77% of transformations had over 0.9 CodeBLEU syntax scores and 89% had over 0.9 dataflow scores, supporting semantic preservation overall.The authors caution that CodeBLEU does not perfectly measure naturalness or semantic preservation; transformations that alter syntax or dataflow can score lower.Source: §B.4.3 · p. 255

B.5.1 Fine-grained Robustness Evaluation

  • The fine-grained robustness evaluation reports results for each perturbation type across CodeGen, InCoder, and GPT-J models using greedy sampling.For randomized perturbations, five perturbed datasets were generated; non-random perturbations were evaluated on one dataset.Source: §B.5.1 · pp. 255–256

B.5.2 Additional Results for Different k

  • Robust Drop stays stable across different k values.The passage provides no further numerical results in this excerpt.Source: §B.5.2 · p. 256

B.5.3 Additional Results for Large Sampling n

  • Larger n can benefit the three robustness metrics without significant differences, while evaluation cost increases linearly with n.The evaluation uses CodeGen-16B-mono on HumanEval with n = 100, top-p 0.95, and temperature 0.2.Source: §B.5.3 · p. 257, Table B.12 · p. 262

B.5.2 Additional Results for Different k

C.2 Artificial Response Generation

  • Artificial responses and reflections are regenerated from minimally edited, manually checked HotpotQA contexts to simulate imperfect context comprehension, rather than editing outputs directly.The authors avoid direct output edits so results reflect the model’s natural response distribution.Source: §C.2 · p. 263

C.3 Conditional Prompting Results

  • Conditional prompting shows no significant difference from parallel prompting, so the evaluation uses parallel prompting to avoid conditional prompting’s implicit bias, as Huang et al. (2023a) point out.Source: §C.3 · p. 265

C.4 Evaluation details for TruthfulQA

  • TruthfulQA evaluation compares responses with preferred and undesired references using Rouge-1 and BLEURT similarity.Source: §C.4 · pp. 265–266

C.5.5 HotpotQA: Standard Prompt

C.5.7 HotpotQA: Response Without Reflection

  • HotpotQA responses are prompted for short answers without reflection across repeated response-and-question calls.The supplied snippets show successive response_1 through response_4 prompts, but do not report answer-quality results.Source: §C.5.7 · pp. 271–272

C.5.8 HotpotQA: Response With Reflection

  • HotpotQA reflection prompts ask the model to critique its previous response, avoid unnecessary apologies, and refer back to the original context before answering again.The prompt sequence repeats this critique-and-answer cycle across responses.Source: §C.5.8 · p. 273

C.5.9 HotpotQA: Fake Evidence Generation

  • Fake supporting-fact generation requests 10 versions, each changing only one relevant sentence to contain false information while leaving other sentences unchanged.Each generated version is to retain the real supporting facts’ sentence count.Source: §C.5.9 · p. 275

C.6 Illustration of API Calling Processes

  • The API-calling illustration covers SR2V, conditional prompting, and the Exploration-Only Baseline.The section is described as a simple example of their API-calling process.Source: §C.6 · p. 277

C.6.1 SR2V API Calling Process

  • The SR2V illustration shows an initial call with instructions and context, followed by a call with an added reflection instruction and a subsequent third call.The provided excerpt identifies the first, second, and third calls but does not specify the third call’s contents.Source: §C.6.1 · p. 277

Comprehension

  • Across 20 examples from all-correct and all-incorrect response sets, models assigned high comprehension scores, typically 4 or 5, regardless of correctness.The prompt asked models to rate understanding from −5 to 5; reliable self-assessment remains an open challenge.Source: Comprehension · p. 282

C.9 Results on LLaMA-2-7b-chat

  • On LLaMA-2-7b-chat, self-reflection improves TruthfulQA performance but worsens HotpotQA performance, matching the reported ChatGPT trends (Touvron et al., 2023b).The authors report these findings in Table C.2 and state that they support generalizing their conclusions across models.Source: §C.9 · p. 283
  • Self-reflection is more beneficial when initial responses are incorrect and question difficulty is higher.The LLaMA-2-7b-chat error-analysis trend closely follows the reported ChatGPT results.Source: §C.9 · p. 283
  • The authors did not replicate the 10-response artificial experiments on LLaMA-2-7b-chat because its 4096-token context limit was shorter than the task required.They report replicating the experiment with Mixtral models, which have larger token limits.Source: §C.9 · p. 283

C.10 Results on Mixtral-8x7B-v0.1 … ESTIMATION

On Mixtral-8x7B-v0.1, self-reflection prompting may help TruthfulQA but harms HotpotQA, and its effects vary little with question difficulty or model comprehension. Additional analyses find benefits only in some artificial-response conditions and little change to majority-voting behavior.

C.10 Results on Mixtral-8x7B-v0.1

  • On HotpotQA, self-reflection prompting fails to improve performance across question difficulty levels and model-comprehension levels.The analysis reports a pattern somewhat different from the preceding comparison.Source: Figure C.4 · p. 284, §C.10 · p. 285
  • Self-reflection may improve TruthfulQA performance but harms HotpotQA on Mixtral-8x7B-v0.1.The reported trend is similar to ChatGPT for the preliminary TruthfulQA and HotpotQA study.Source: §C.10 · pp. 283, 285, Table C.3 · p. 285
  • The authors hypothesize that Mixtral is less sensitive to reflection instructions or less able to understand them, making reflection a distractor.This is offered as an explanation for the divergent performance patterns, not as an established mechanism.Source: §C.10 · p. 285
  • In artificial-response experiments, self-reflection helps on easy questions under 0% RA, but is harmful in most cases.The passage notes that self-reflection is not always harmful to performance.Source: §C.10 · p. 285, Figure C.5 · p. 286
  • Compared with ChatGPT, adding self-reflection has minimal impact on Mixtral’s majority-voting trends, suggesting its default behavior is difficult to change this way.The authors say this is consistent with the hypothesis that Mixtral lacks sensitivity or competence in self-reflection.Source: §C.10 · pp. 285–286, Figure C.6 · p. 286
  • The authors speculate that additional training may be needed to unlock Mixtral-8x7B-v0.1’s potential for self-reflection prompting.This proposed need follows from the observed limits in sensitivity or competence.Source: §C.10 · p. 286

D.1 Adaption for FastSHAP Baseline … E.6 Full Task-wise BF Evaluation on Different Prompt Complexity

The appendix documents implementation constraints and evaluation setups, then shows that branching factor (BF) varies with alignment, generated-token position, task, and prompt complexity rather than following one consistent response to more context.

D.1 Adaption for FastSHAP Baseline

  • FastSHAP’s GPU memory demands limited runs to batch size 1 and 32 perturbations per instance, equivalent to approximating unreliable KS-32; convergence took three days on one A100.Despite hyperparameter search, optimization remained unstable.Source: §D.1 · p. 287

D.2 Scientific Artifacts License

  • MNLI is released under ONAC’s license, Yelp-Polarity and SST-2 lack detailed licenses, and the model checkpoints are MIT-licensed.The dataset markers are MNLI (Williams et al., 2018), Yelp-Polarity (Zhang et al., 2015), SST-2, and checkpoints from textattack.Source: §D.2 · p. 287

D.3 Training Details

  • MNLI and Yelp-Polarity were split 8:1:1 for training, validation, and testing; amortized models used Adam at 5e-5 for up to 10 epochs with early stopping.Hyperparameters were tuned on the validation set and checkpoints selected by early stopping.Source: §D.3 · p. 288

E.1 Case Study Implementation Details; E.2 Dataset-Specific Processing

  • The case study sampled 200 MMLU-STEM examples and averaged performance over 64 trials; prompt complexity was operationalized differently across tasks, including in-context sample counts and added plot words.MMLU prompt complexity C is the number of in-context samples, while Creative Story Generation uses C × 25 words in the plot.Source: §E.1 · p. 289, §E.2 · pp. 289, 291

E.3 Entropy Underestimation via Monte Carlo Sampling

  • Monte Carlo entropy estimates rise with sample count, showing that small samples systematically underestimate true entropy by missing the distribution’s long tail.More samples mitigate the bias but incur significant computational cost; the passage contrasts this with using a typical sequence’s negative log-likelihood.Source: §E.3 · pp. 291–292, Figure E.3 · p. 293

E.5 Curious Case of Prompt Complexity

  • On Cognac, greater prompt complexity can increase BF, potentially due to negation or complex linguistic structures, while higher complexity generally lowers BF on News Generation.The reported Cognac increase is described as one instance of unexpected content raising BF.Source: Figure E.4 · p. 293, §E.5 · p. 294

E.4 Full BF Output Dynamics Investigation

  • The base models’ average BF is approximately 12 versus approximately 1.2 for aligned models, and BF often declines smoothly as more output tokens are generated.The full Llama-2 and Llama-3 dynamics are shown in Figures E.1 and E.2.Source: Figure E.1 · p. 290, Figure E.2 · p. 292, §E.4 · p. 293

E.6 Full Task-wise BF Evaluation on Different Prompt Complexity

  • Across models and tasks, prompt complexity modulates BF in highly inconsistent ways, with no clear monotonic pattern as context increases.The passage reports this full task-wise evaluation in Figure E.5.Source: §E.6 · p. 294

E.7 Generalization to Additional Tasks and Models … F.3 Proof of Temperature-Entropy Relationship

This section extends and analyzes branching factor (BF) across tasks and models, examining its relation to information density and sample diversity, the effects of unexpected context and autoregressive self-conditioning, and whether contamination explains BF patterns. It also reports MMLU nudging results and presents training details and theoretical analyses for sampling-based reinforcement learning, including convergence and the temperature–entropy relationship.

E.7 Generalization to Additional Tasks and Models

  • Across XSUM, Aya, Qwen3-4B, and OLMo-2, findings generalize: alignment lowers BF, but OLMo-2’s reduction is less pronounced than Llama’s.The additional-task results include summarization on XSUM and multilingual tasks on Aya; OLMo-2 is compared across Base and DPO stages on Creative StoryGen and MMLU at 7B and 13B.Source: §E.7 · p. 296, Figure E.6 · p. 297, Figure E.8 · p. 298

E.8 Proof of LLM Log-Likelihood Convergence

  • The proof establishes convergence in probability between length-averaged NLL and realized entropy, using bounded transformer log-probabilities and Chebyshev’s inequality.The variance of the summed deviation is the sum of its variances because the cross-terms are zero.Source: §E.8 · pp. 300–301

E.9 Full Nudging Experiment Results

  • On MMLU, nudging significantly lowers base-model BF, though less quickly than on Just-Eval-Instruct.The full MMLU nudging results are shown in Figure E.10.Source: §E.9 · p. 301, Figure E.10 · p. 301

E.10 BF and Information Density

  • BF, defined as exponentiated entropy rate, formalizes its link to information density and supports a unified analysis of alignment training, decoding dynamics, and output variability across modern LLMs.Source: §E.10 · p. 302

E.11 Discussion: BF and Cross-Sample Diversity Measures

  • BF estimates the underlying distribution’s entropy rate, not sample-pool diversity; their weak empirical correlation makes BF and cross-sample diversity complementary, not interchangeable.Source: §E.11 · pp. 302–303

Self-Narrowing from Alignment

  • Autoregressive self-narrowing is a robust aggregate tendency across base and aligned models, often concentrating next-token distributions even on meaningless inputs; local increases can occur.Source: Self-Narrowing from Alignment · p. 304
  • Alignment lowers absolute BF and often accelerates early narrowing, but it governs the level and rate rather than solely explaining decreasing BF.The RANDOM STRINGS result is consistent with self-narrowing even without semantic structure; the AEP result should not be read as proving monotone token-wise BF decrease.Source: Self-Narrowing from Alignment · p. 304

A controlled intervention: substituting external random tokens for the model’s own

  • Replacing a model-generated prefix with random tokens causally restores high BF, which generally decays again during continued autoregressive generation.The intervention separates the self-conditioning tendency from alignment: alignment changes the absolute BF scale and often accelerates re-collapse, but not the qualitative dynamics.Source: A controlled intervention:… · p. 305
  • Adversarial and random-noise environment feedback increase BF relative to matched control, with the largest effect in less-aligned models and effects sometimes near zero in heavily aligned models.Together with negation and random-token substitution, these findings indicate that unexpected context can raise BF, whereas continued self-conditioning tends to lower it; the claim is behavioral, not a mechanistic explanation or proof of monotonic decrease.Source: A controlled intervention:… · pp. 307–308

E.13 Discussion: Diversity and BF Correlation

  • BF and Distinct-N show no consistent correlation: depending on model and task, their relationship can be strongly positive, strongly negative, or absent.The analysis includes an example of no relationship for Llama-3-70B-Instruct on Cognac.Source: §E.13 · pp. 308–309
  • BF and Distinct-N show no consistent correlation, because BF characterizes the full output distribution while sample-based diversity can miss differences between distributions and is sensitive to generation length (Liu et al., 2022).The discussion treats probability concentration measured by BF as a more fundamental lens on alignment, linking BF reduction to a shortened generative horizon, lower sensitivity to decoding, and Chain-of-Thought shifting information toward low-BF regions.Source: §E.13 · p. 309

E.14 Confounder Investigation: Data Contamination

  • Figure E.13 shows no consistent BF–Min-K% correlation across models and tasks, so contamination cannot fully explain BF patterns.Some Llama models show significant negative correlations, others positive correlations, and correlations are notably weaker for other models.Source: §E.14 · p. 310, Figure E.13 · p. 312

F.1 Minimal-RL Training Details

  • The experiments follow the Minimal-RL recipe to fairly compare rollout sampling strategies, with hyperparameters listed in Table F.1.Source: §F.1 · p. 313, Table F.1 · p. 313

F.2.2 Truncated Importance Sampling Ratio Correction

  • EAD truncates the importance-sampling likelihood ratio with an upper bound to limit extreme-ratio harms and improve training stability.Source: §F.2.2 · p. 316

F.3 Proof of Temperature-Entropy Relationship

  • The proof establishes that softmax entropy is non-decreasing in temperature τ > 0, since entropy is non-increasing in inverse temperature β and its derivative is non-positive.The derivative’s sign follows from the non-negative variance of the logits.Source: §F.3 · pp. 317, 319

F.4 Increasing Length in RL Training … G.1 Related Work

EAD learns to lengthen reasoning selectively on harder problems, while the appendix frames BACO as an inference-time approach to the diversity–quality trade-off. The paper also notes limits in collaboration cost, task scope, and checkpoint coverage.

F.4 Increasing Length in RL Training

  • EAD selectively lengthens reasoning on harder problems after initially shortening responses, boosting final performance compared with temperature sampling.Both methods first shift from complex code-based solutions to direct mathematical reasoning; temperature-sampled length then stabilizes, while EAD increases reasoning length for harder problems. EAD is shown with DAPO rollouts.Source: §F.4 · p. 319, Figure F.2 · p. 320

Limitations

  • Two-model collaboration adds per-token FLOPs, and the prototype is not yet fully speed-engineered.The paper identifies speculative decoding, LoRA-based aligned models, and KV-cache sharing as possible mitigations.Source: Limitations · p. 321
  • The study focuses on inference-time improvement and canonical base and aligned checkpoints, leaving longer-horizon agent and code-generation extensions and RL-rollout diversity out of scope.Intermediate alignment checkpoints, where available, are described as a promising direction for additional Pareto gains.Source: Limitations · p. 321

G.1 Related Work

  • The passage says alignment techniques such as RLHF enhance instruction-following and reasoning performance but systematically curtail output diversity, with aligned models described as less diverse than their base counterparts.Source: §G.1 · p. 321
  • Alignment-related diversity loss is associated with more homogeneous formats, diminished creativity, restricted reasoning strategies, and reduced diversity and creativity in human interactions.The cited work also describes losses in linguistic idiosyncrasies (Chakrabarty et al., 2025), data synthesis, deep research, social simulation, and gaming, and effects on how people ideate, create, and engage.Source: §G.1 · p. 322

BASE-ALIGNED MODEL COLLABORATION

  • BACO is an inference-time collaborative decoding framework that combines base and aligned models to control the diversity–quality trade-off.The paper positions it as offering more explicit and reliable control over that trade-off.Source: §G.1 · pp. 322–323

G.1 Related Work

  • Diversity-promotion methods include training-time objective changes and inference-time decoding or prompting, but training requires substantial retraining compute and inference methods can be costly or reduce quality.Training examples include diversity-aware DPO and generative flow networks; inference examples include diverse beam search and paraphrasing, while high compute and quality degradation complicate a stable diversity–quality trade-off.Source: §G.1 · pp. 322–323

APPENDIX G

  • BACO advances token-level collaboration, whose prior approaches mainly target quality or efficiency, by focusing on the diversity–quality trade-off.Related work groups multi-model collaboration into weight-level approaches, token-level exchanges, and multi-agent discussion or debate.Source: §G.1 · pp. 323–324

G.2 BACO Framework Implementation and Router Details … G.5 Experiment Setup

This section details BACO’s decoding and routing implementation, including switching and termination rules and the strategies evaluated. It also specifies the experiment setup, datasets, inference and decoding configuration, and quality–diversity metrics, and reports prototype runtime comparisons and qualitative differences among strategy families.

G.2 BACO Framework Implementation and Router Details

  • BACO enforces same-model generation within semantic units to avoid malformed words or punctuation, and its worst-case cost is about 2× FLOPs per token regardless of group size.The router’s ordered strategy cascade terminates with a definite model decision because its last strategy always fires.Source: §G.2 · p. 324
  • BACO starts with the aligned model, then evaluates router strategies in order at word boundaries; the first strategy that fires selects the model for generation.A router can contain one strategy or combine multiple strategies, and the aligned model controls whether an end-of-sequence token is accepted.Source: §G.2 · pp. 324–325
  • BACO’s prototype is slower than single-model baselines but comparable to or faster than several diversity baselines, while outperforming them on the diversity–quality trade-off.The implementation is not yet engineered for speed, and the passage notes that listed optimizations could further reduce wall-clock cost.Source: §G.2 · pp. 325–326, Table G.1 · p. 326

G.2.1 Additional Rules in Router

  • The router begins with an aligned-model token, which improves trajectory quality and reduces degenerate completions; it accepts termination only when aligned-model top-1 predicts EOS.This rule addresses premature termination by the aligned model and repetition or verbose listing by the base model after model switches.Source: §G.2.1 · p. 326

G.2.2 All Routing Strategies and Notations

  • The evaluated routers include random, probability- and entropy-based, content-word, punctuation, external-judge, and multi-condition strategies for choosing between base and aligned models.The judge checks whether continuation can diverge and whether the sampled base continuation is acceptable; combined routers apply conditions such as content-word or punctuation rules alongside probability thresholds.Source: §G.2.2 · pp. 327–328
  • Aligned-model logits metrics are less distinctive, making -P-A and -H-A less effective; -FC and -PUNC underperform their content-feature motivation, with on-tier performance and PUNC lighter than FC.FC’s part-of-speech parsing adds computational cost, so some experiments use PUNC as the representative strategy.Source: §G.2.2 · p. 328
  • The -H-PUNC router applies -PUNC and then -P, routing to the base model for non-punctuation/formatting tokens or entropy above γ, and otherwise to the aligned model.The threshold is γ ∈ [0, +∞).Source: §G.2.2 · p. 329

G.3 Dataset Details

  • Evaluation combines NoveltyBench’s multiple-valid-answer prompts, WildChat’s open-ended conversations, and Narrative-Discourse’s long-form fictional film-synopsis continuations.NoveltyBench prompts are simple and often short, so the study adds more complex datasets; Narrative-Discourse annotations support evaluation of long-form structural diversity.Source: §G.3 · pp. 329–330

G.4.1 Diversity Metrics

  • For each prompt, the diversity evaluation samples n = 10 outputs and applies a broad set of automated metrics.NoveltyBench spans randomness, underspecified factual knowledge, creative writing, and subjective queries.Source: §G.3 · p. 329, §G.4.1 · p. 330

Lexical Metrics

  • Lexical diversity uses Distinct-n and length- and vocabulary-normalized EAD-n, while lower Self-BLEU and Self-ROUGE-L scores indicate greater diversity.Self-BLEU averages pairwise BLEU scores, and Self-ROUGE-L averages pairwise ROUGE-L scores.Source: Lexical Metrics · pp. 330–331

Semantic Metrics

  • Semantic diversity is assessed with embedding cosine dissimilarity, Vendi Score, NLI Diversity, NoveltyBench Distinct Score, and Semantic Entropy.These metrics capture pairwise embedding distance, effective distinct modes, contradiction, functional equivalence classes, and diversity across entailment-based semantic clusters, respectively.Source: Semantic Metrics · pp. 331–333

G.4.2 Aggregate Metrics over Quality–Diversity Spaces

  • Quality–diversity spaces normalize metric pairs to a unit square anchored by base and aligned models at temperature 1.0, discarding points outside the feasible region.Control parameters such as temperature or routing threshold trace the operating points within each space.Source: §G.4.2 · pp. 333–334
  • Coverage summarizes area under a method’s normalized quality–diversity curve, while Dominance measures its share of the global Pareto frontier and complements Coverage’s limitation on optimality.Coverage is computed over the normalized unit-square domain; Dominance combines quality- and diversity-axis comparisons.Source: §G.4.2 · pp. 334–335
  • Pareto-point responsibility intervals are split at midpoints between adjacent frontier points, and summed coverage is normalized by the total frontier span.Coverage is computed separately along the diversity and quality axes; both normalized values lie in [0, 1].Source: §G.4.2 · p. 335
  • A single Dominance score is the harmonic mean of the diversity- and quality-axis coverage values, with holistic summaries averaging over all measurement spaces.The averaging yields metric-agnostic summaries.Source: §G.4.2 · p. 335

G.5 Experiment Setup

  • For group-level diversity, each prompt produces n = 10 outputs, generally sampled at temperature 1.0 with top-p 0.9 and no top-k truncation, following Zhang et al. (2025c).Exceptions are limited to baselines that inherently require alternative decoding strategies.Source: §G.5 · p. 336
  • Diverse Beam Search uses beams = 20, beam_groups = 10, and diversity_penalty = 1.0, following Vijayakumar et al. (2016).These settings follow the original paper.Source: §G.5 · p. 336

G.6 Validation on Verifiable Benchmarks … H.6.1 Automatic Evaluation

This section presents evaluations of BACO’s diversity, quality, and creativity across verifiable and open-ended tasks, using benchmark results, metrics, examples, and human judgments. It also describes AI Realtor’s grounded real-estate marketing pipeline and its automatic and human evaluations of feature selection, personalization, and generated property descriptions.

G.6 Validation on Verifiable Benchmarks

  • On IFEval, BACO achieves higher diversity than the aligned baseline at matched quality; on GSM8K, it maintains high accuracy, including 90% Pass@10, with substantially higher diversity across operating points.The reported trends on both verifiable benchmarks mirror those on open-ended generation tasks.Source: §G.6 · p. 336, Figure G.1 · p. 339, Figure G.2 · p. 341

G.7.1 Instruction Following on NoveltyBench

  • On NoveltyBench, BACO improves Coverage by 0.274 overall (0.222 lexical, 0.291 semantic), dominates 39.9% of the frontier, and outperforms baselines on all metrics except lexical Dominance.The reported Coverage gains are 0.274 overall, 0.222 lexical, and 0.291 semantic; BACO dominates 31.0% of the lexical and 48.8% of the semantic frontier.Source: §G.7.1 · p. 337, Table G.2 · p. 337

G.7.2 Dialogue on WildChat

  • On WildChat, BACO-P-PUNC improves Coverage by 29.8% over the aligned baseline, dominates 30.8% of the frontier, and dominates 48.5% of the semantic frontier.The results are presented for naturally complex prompts with longer outputs on average than NoveltyBench.Source: §G.7.2 · p. 338, Table G.3 · p. 338

G.7.3 Creative Writing on Narrative-Discourse

  • On Narrative-Discourse, BACO achieves 13.5% higher Coverage and dominates 20.3% of the overall diversity–quality frontier.The task tests structurally diverse, long-term coherent narratives emphasizing sustained creativity and narrative arc.Source: §G.7.3 · pp. 338–339, Table G.4 · p. 339

G.7.4 Extensional Tasks Results

  • The extensional-task materials provide full IFEval and GSM8K trade-off results and NoveltyBench comparisons across methods; NoveltyBench Dominance values in the table’s top and bottom halves are not directly comparable.BACO All has the best Cov. and Dom. sum across routers in the NoveltyBench table.Source: §G.7.4 · pp. 339–340, Table G.5 · p. 340

G.8.1 Creative Writing Task Setup

  • The creative-writing task continues a story from a prefix ending at its first turning point, measuring discourse-level variation through turning-point and arousal diversity.Arousal diversity compares smoothed curves of sampled sentence-level arousal scores using KL divergence.Source: §G.8.1 · p. 342

G.8.2 Narrative-Discourse Evaluation Metrics

  • Narrative-Discourse evaluation measures turning-point diversity across outputs and arousal diversity from LLM-judged sentence scores interpolated into smooth trajectories.The turning-point measure uses K = 5 annotated turning points.Source: §G.8.2 · p. 343

G.9 Additional Analysis Material

  • Additional analyses report BACO-P-PUNC’s per-token contribution and switching frequencies across three tasks, and illustrate an early-stopping failure in which a router switch leads to incoherent continuation.Further examples show outputs changing as routing threshold γ traverses the diversity–quality spectrum.Source: §G.9 · p. 344, Table G.8 · p. 345
  • Lower routing thresholds increase aligned-model routing and output length, while higher thresholds favor base-model outputs that are more often incomplete and lower quality.The example uses -P as router and greedy decoding for clarity; the trend also holds with other sampling methods and routers.Source: Table G.9 · p. 347

G.9.1 Extended Future Work

  • Speculative decoding could reduce BACO’s worst-case two-pass decoding overhead, while LoRA-tuned aligned models could cut storage costs and preserve the diversity–quality contrast.The proposed LoRA arrangement replaces N× model storage with one base model plus N× adapter size.Source: §G.9.1 · p. 347
  • Intermediate or partially aligned checkpoints may offer better diversity–quality operating points, and studying alignment dynamics could identify stronger collaborator pairs.The discussion presents broader exploration of the training trajectory as a way to extend the diversity–quality frontier beyond base and fully aligned checkpoints.Source: §G.9.1 · pp. 347–348

G.10.1 Setup; G.10.2 Annotators.

  • Human evaluation sampled 20 prompts per dataset and three outputs per method from Novelty-Bench and WildChat, using four LLM-informed student annotators and shorter-output strata.BACO and the aligned baseline were tuned for comparable automatic quality scores; prompts were stratified by average response length while maintaining comparable quality and diversity scores.Source: §G.10.1 · pp. 348–349, §G.10.2 · p. 349

G.10.3 Instructions.

  • Annotators rated each response’s quality on a 1–5 scale, then compared response groups for holistic, stylistic, and core-content diversity.The quality criteria prioritize fluency, relevance, and substance.Source: §G.10.3 · pp. 350–352
  • Diversity judgments ask whether responses differ in core ideas, angles, topics, perspectives, or examples, while creativity evaluation selects the most creative response overall.The instructions emphasize meaningful variation in core substance.Source: §G.10.3 · pp. 352–353

G.10.5 Examples.

  • The email examples compare an aligned model with BACo outputs, with annotations marking one group as more diverse and an output as most creative.Source: §G.10.5 · pp. 354, 357

G.10.5 Examples

  • BACO demonstrates better human-evaluated quality, diversity, and creativity, with configurations matched to automatic quality metric results.Table G.11 presents examples of individual output quality, group-level diversity, and creativity results.Source: Table G.11 · p. 361
  • The judge prompt assesses whether context supports multiple valid next words using a prompt, response, Yes/No decision, and rationale; No answers require two completions.The rationales explain why a next word is flexible or predictable from the supplied context.Source: §G.10.5 · p. 361
  • Examples treat story openings as flexible, while narrative coherence or names specified in the prompt can make a continuation more predictable.A story beginning permits varied settings or openings; in other cases, Bob is expected after Abe, or the prompt's character names constrain the next word.Source: §G.10.5 · pp. 362–363

G.10.5 Examples.

  • The examples’ judge rationales treat factual questions as permitting little next-word diversity, while fictional prompts permit multiple valid continuations.For a simulated six-sided die roll, any integer from 1 to 6 is considered valid.Source: §G.10.5 · pp. 364–365

G.11 Qualitative Examples

  • Semantic Entropy was 1.3554 and Perplexity was 17.4937 for the unaligned response, versus 0.5714 and 1.4832 for the aligned response.Source: §G.11 · pp. 365, 367
  • BACO (Llama-3-8B and Llama-3-8B-Instruct) has Semantic Entropy: 2.6046 and Perplexity: 3.6892 in a qualitative example.Source: §G.11 · p. 369
  • On NoveltyBench, BACO shows the best diversity in terms of semantic entropy among the base, aligned, and BACO models.The supplied table caption does not provide numerical cell values.Source: Table G.13 · p. 371
  • For the aligned Llama-3-8B-Instruct example, Semantic Entropy is 0.0000 and Perplexity is 1.8952.Source: §G.11 · p. 374
  • Generated descriptions portray Soylent Bugs as a staple insect-based food in a dystopian future, with attention to its taste, nutrition, production, and contested consumption.Source: §G.11 · pp. 376–377
  • BACo (Llama-3-8B and Llama-3-8B-Instruct) reports Semantic Entropy: 1.7247 and Perplexity: 8.5552.Source: §G.11 · p. 378
  • Other examples present Soylent Bugs in promotional product copy and explain its name as a continuation of the Soylent Green theme.Source: §G.11 · pp. 379–380
  • BACO shows the best diversity-quality trade-off in semantic entropy and perplexity among the compared WildChat outputs.Table G.14 compares generated outputs from the base model, aligned model, and BACO, showing three randomly sampled outputs out of ten for each.Source: Table G.14 · p. 382
  • Semantic entropy is 2.0639 and perplexity is 7.0864 for the aligned Llama-3-8B-Instruct model.Source: §G.11 · p. 383
  • Quasi refuses the king’s offer to serve as his personal torturer, choosing to break free from his miserable existence.Source: §G.11 · p. 386
  • Semantic Entropy was 2.0956 and Perplexity was 5.8706 for Llama-3-8B and Llama-3-8B-Instruct.Source: §G.11 · p. 387
  • In another account, Quasi becomes a monk and eventually a high-ranking pope, succeeding Pope Cornelius and working toward peace between kingdoms.Source: §G.11 · p. 389
  • BACO shows the best diversity-quality trade-off in semantic entropy and perplexity among the compared Narrative-Discourse outputs.The table compares outputs from the base model, aligned model, and BACO, showing three of ten sampled outputs.Source: §G.11 · p. 391, Table G.15

H.1.1 User Feedback on Generated Descriptions with Surprisal Features

  • The user chose Description B because its evocative, sensory language created a stronger emotional connection and more vivid picture of the condo’s lifestyle.The feedback highlights its aspirational tone, emphasis on space and luxury, and more vivid sense of place.Source: §H.1.1 · pp. 394–395
  • The authors acknowledge that generating appropriate language for surprising features can be challenging and may confuse users.They introduce a second example as one that confuses the user.Source: §H.1.1 · p. 395
  • In one case, a user questioned what a generated description’s “top 2% of amenities” claim meant.The example’s user comment concerns the meaning of the claim, not whether it was accurate.Source: §H.1.1 · p. 396

H.1.2 The Failing Cases of Human-Written Descriptions

  • Human-written descriptions could omit important facts that model-generated descriptions included, although a user also said a generated description lacked size and location details.The examples show that either description type could be disfavored for missing information.Source: §H.1.2 · pp. 396–397
  • A user preferred a human-written listing over an all-caps alternative because the capitalization felt like being yelled at.The case analysis notes that human-drafted descriptions can look unpleasant.Source: §H.1.2 · p. 398

H.1.3 The Dichotomy of User Preferences on Writing Styles

  • Users differed on description length: some preferred concise, direct writing, while others wanted more property details.The paper presents these as subjective preferences rather than a single preferred style.Source: §H.1.3 · p. 398
  • User reactions show varied preferences: some favor concise copy, while others prefer longer, more descriptive prose or plain descriptions listing amenities.Comments describe concise copy as getting to the point faster, longer descriptions as useful when interested, and plain copy as better at listing amenities.Source: §H.1.3 · pp. 400, 402, 403
  • The examples suggest no single writing style suits every user; future work could tailor descriptions to users’ preferred styles to improve persuasiveness.Source: §H.1.3 · p. 404

H.1.4 The Diversity of Writing Styles On Different Listings

  • AI Realtor uses linguistically diverse listing styles based on property features, enabling it to tailor descriptions to different listings.Source: §H.1.4 · p. 404
  • Low-end listings emphasize “Safety & Survival,” while high-end listings emphasize “Display & Views.”The contrast is described in terms of security, enclosure, and reassurance for low-end listings versus openness, visual richness, and mastery over the environment for high-end listings.Source: §H.1.4 · p. 404

H.2.1 Survey Screening Interface; H.2.4 Feature Annotation Interface

  • Screening uses quiz questions on an example listing to verify participants’ home-search experience, and a separate web interface supports feature annotation.The screening quiz requires correct responses; the feature-annotation interface is shown in Figure H.4.Source: §H.2.1 · p. 406, §H.2.4 · p. 409

H.2.2 Preference Elicitation Interface

  • The preference survey gathers buyer profiles and ratings across five categories, filters by price and bedrooms, then elicits ratings and reasoning for five relevant listings.An LLM also narrows likely preferred features for participants to rate for importance.Source: §H.2.2 · p. 407

H.2.3 Human Evaluation Interface

  • Human evaluation uses A/B comparisons of two model descriptions for one listing and an interleaved test where buyers select listings based on descriptions.After choices, participants rate preferences on a 1–5 scale and provide rationales.Source: §H.2.3 · p. 408

H.3.1 GPT-4o-Polished Human Control

  • Against GPT-4o-polished human descriptions, AI Realtor recorded 254 wins, 51 losses, and 31 ties (83.3% win rate excluding ties), with Elo 1168 versus 832.The 83.3% win rate had a 95% Wilson confidence interval of [78.7%, 87.0%]; the evaluation collected 336 pairwise comparisons.Source: §H.3.1 · p. 410, Figure H.4 · p. 410

H.3.2 Elo Uncertainty and Convergence

  • The full pipeline is strongest overall and clearly separated from vanilla LLM and human baselines, while major-system rankings remain stable across vote-pool subsamples.By the 75% subsample, absolute Elo estimates were within approximately ±30 points; the authors do not claim every adjacent module addition yields a significant gain.Source: §H.3.2 · p. 410

H.3.3 Grounding Module Evaluation

  • Grounding accuracy is around 78% for concrete structural, layout, and room-configuration features, versus around 61% for subjective ambiance and lifestyle appeal.The grounding module is evaluated as multi-label binary classification with each feature predicted independently; the table reports feature-extraction and grounding-model performance.Source: Table H.1 · p. 411, §H.3.3 · p. 411

H.3.4 Annotation Reliability

  • Three graduate-student annotators showed substantial agreement on hard attributes (κ = 0.74) and moderate agreement on soft attributes (κ = 0.58).The annotators had NLP evaluation experience and checked hard and soft factual attributes.Source: §H.3.4 · p. 411
  • Annotators achieved κ = 0.81 on feature-category assignments, while GPT-4o-generated grounding labels received κ = 0.74 agreement on a stratified 200-listing sample.Models and humans agreed less on ambiguous or subjective features, at around 60%, roughly matching agreement among human annotators.Source: §H.3.4 · p. 411, §H.4.1 · p. 414

H.3.5 Hyperparameter Selection

  • The feature-existence threshold was selected by held-out F1 grid search, while the authors did not exhaustively test Elo sensitivity because each setting required another human evaluation.Personalization used c = 0.01 and r0 = 2, with the top 10 features passed to generation; surprisal-based marketing features used β = 30%.Source: §H.3.5 · p. 412

H.4.1 Grounding Module: Predicting Marketable Features

  • LLMs induced a hierarchical feature schema from unlabeled descriptions, enabling attribute-feature mapping with minimal human supervision.Direct LLM prompting produced redundant features and missed important ones; small-batch prompting with Claude-3.5-Sonnet yielded a more structured schema, which was manually reviewed and refined.Source: §H.4.1 · pp. 412–413
  • The mapping achieved 69.39% testing accuracy and 67.43% F1, which the authors considered reasonably high because signaling is stochastic and deterministic predictions may not exactly match human-written features.Source: §H.4.1 · p. 415
  • The grounding module deterministically selects predicted marketable features above threshold α = 1/2 rather than randomly sampling signals.The stated rationale is that generated marketing content is a tiny portion of the corpus and the objective prioritizes rigorousness.Source: §H.4.1 · p. 415

H.4.2 Personalization Module: Aligning with Preferences

  • The personalization module elicits structured buyer preferences and uses them to select marketable features for the LLM to emphasize in marketing copy.Users rate feature importance on a 1–5 scale and provide rationales; the implementation adjusts population-level feature scores using those ratings.Source: §H.4.2 · p. 416

H.4.3 Marketing Module: Capturing Surprisal via RAG

  • The marketing module retrieves comparable listings and supplies localized features ranked in the top 30% percentile to guide persuasive language generation.Comparisons use listings from proximal locations or listings with similar features.Source: §H.4.3 · pp. 417–418

H.5.1 Dataset raw attribute schema

  • The evaluation dataset contains around 50,000 Zillow listings collected in April 2024 from the 30 most populous U.S. cities, with 95 columns of listing features.Non-residential listings and listings missing crucial data were excluded.Source: §H.5.1 · p. 418

H.5.2 Final Feature Schema

  • The final feature schema organizes vocabulary for property spaces and interiors, amenities, outdoor areas, location, neighborhood characteristics, transportation access, and walkability.Its terms include examples such as bedrooms, countertops, pools, proximity to schools, and walk score.Source: §H.5.2 · pp. 419–421
  • The final feature schema groups real-estate terms across property characteristics, distinctive interior elements, financial and legal aspects, and home measurements.Examples include property age and renovation terms, interior features, fees and rental potential, and lot size or square footage.Source: §H.5.2 · pp. 423–424

H.5.3 Diversity of The Real Estate Market in Chicago

  • Chicago has the highest home-type entropy and strongest price dispersion among the major cities examined, indicating a particularly heterogeneous market.Home-type entropy captures the mix of housing types, while price dispersion is measured by percentile ratios.Source: §H.5.3 · p. 424

H.6 Hallucination Experiment Details

  • Hallucination experiments include both automatic and human evaluation.The section introduces implementation details for these two evaluation modes.Source: §H.6 · p. 425

H.6.1 Automatic Evaluation

  • Automatic evaluation uses GPT-4o fine-grained fact-checking, following FActScore (Min et al., 2023), to assess hard attributes and soft home insights or address information.Price, living area, bedrooms, and bathrooms are treated as hard attributes; home insights and address are soft attributes.Source: §H.6.1 · p. 425
  • Faithfulhard scores mentioned attributes by whether their extracted values match listing information, while Faithfulsoft extracts soft attributes, judges contextual matches, and sums the scores.For home insights, a significant subset of true insights must be identified; for addresses, a subset such as city or state can count as a match when mentioned.Source: §H.6.1 · pp. 426–427

H.6.2 Human Evaluation … I.2 Heatmap for Worker-Expert labels over the Qualification Test

This section reports human evaluation of AI Realtor and specifies its prompts and retriever fields. It also documents the artifact release and LLM use, expert annotation guidance, and a heatmap comparing worker and expert qualification-test labels.

H.6.2 Human Evaluation

  • AI Realtor has the highest faithfulness on Xhard, while human-written descriptions score lowest in credibility; on Xsoft and credibility, AI Realtor is comparable to humans.The authors interpret the subjective-evaluation results as evidence that AI Realtor does not rely on hallucination or deception to persuade users.Source: §H.6.2 · p. 429

H.7.1 Keyword Extraction Prompt

  • The keyword-extraction prompt asks for attractive words or phrases from a house description, separated by commas.Source: §H.7.1 · p. 429

H.7.2 Keyword Extraction Normalization Prompt

  • The normalization prompt removes quantifiers, numbers, and adjectives, returning only a precise noun phrase or keyword without explanation.Source: §H.7.2 · pp. 429–430

H.7.3 Schema Induction Prompt

  • The schema-induction prompt processes 1k+ keywords in batches of 100, assigning them to existing or informative new categories and retaining original keywords at leaf nodes.Source: §H.7.3 · pp. 430–431

H.7.4 Feature Extraction Based on Description Prompt

  • Feature extraction asks whether a feature appears in the description, using example keywords to explain it and requiring a YES-or-NO response.Source: §H.7.4 · p. 433

H.7.5 Persuasive Language Generation with Personalized Features

  • Personalized generation supplies listing attributes and features alongside client preferences, then asks for a concise persuasive description emphasizing matching features.Source: §H.7.5 · pp. 433–434

H.7.6 Persuasive Language Generation with Localized Feature Prompt

  • Localized generation frames features through comparisons with similar listings, Chicago, and the user’s neighborhood, and asks the model to highlight preference-matching features.Source: §H.7.6 · p. 434

H.7.7 User Simulation Prompt

H.8 Artifact Evaluation

  • The authors report open-sourcing code and part of the anonymized data.Source: §H.8 · p. 436

H.9 Usage of Large Language Models

  • LLMs were used to polish writing, retrieve literature, and assist with code writing and debugging; the authors say they monitored for plagiarism and fabricated facts.The listed tools include Gemini 2.5 Pro and ChatGPT 4/5 for writing, Gemini 2.5 Pro Deep Research for literature retrieval, and Claude 3.5 Sonnet for code assistance.Source: §H.9 · pp. 436–437

I.1 Annotation Guideline

  • Expert guidelines define six stages of the OUD continuum and provide expert-selected example posts to help annotators apply the definitions.The full guideline is too large to include in the paper, and experts also drafted FAQs to clarify initial annotation trials.Source: §I.1 · p. 442, Table I.2 · p. 443
  • Posts without evidence that the poster used opioids, and posts about non-opioid drugs, are labeled “not using”; misuse is not limited to prescription opioids.Infrequent or synthetic/semi-synthetic opioid use without signs of addiction is labeled “misuse.”Source: §I.1 · pp. 442–443
  • Relapse is defined as opioid use after an attempt to quit, so use of other drugs during recovery does not count as relapse.Source: §I.1 · p. 443

I.2 Heatmap for Worker-Expert labels over the Qualification Test

  • A heatmap summarizes differences between workers’ and experts’ annotations on the qualification test.The supplied passage does not report the differences’ values or direction.Source: §I.2 · p. 444

I.3 In-Context Samples in Few-Shot Learning Settings … I.7 Scientific Artifacts

The appendix documents few-shot sample selection, GPT-output post-processing, supervised fine-tuning, class-wise performance reporting, and the scientific artifacts used.

I.3 In-Context Samples in Few-Shot Learning Settings

  • The few-shot prompts used 13 in-context examples selected as representative after expert discussion, with category distribution based on held-out preliminary experiments.Table I.4 presents the examples for each Opioid Usage category.Source: §I.3 · p. 444, Table I.4 · p. 444

I.4 Post-processing needed for processing GPT-3 outputs

  • GPT-3 outputs were post-processed by truncating at newlines, extracting the term between numbered responses, and manually correcting morphological variants and typos.Source: §I.4 · pp. 444, 446
  • Applying the same post-processing to GPT-4 produced no significant changes, which the authors suggest may reflect better instruction-following than GPT-3.Source: §I.4 · p. 446

I.5 Fully Supervised Fine-Tuning Details

  • DeBERTa-v3-large (He et al., 2021) was fine-tuned for 10 epochs using Transformers, a 2e−5 learning rate, and AdamW.Source: §I.5 · p. 446
  • T5 fine-tuning used the 3B and 11B models, 100 held-out validation examples, batch sizes of 1024 and 512, and a 1e-4 learning rate with AdamW.The 3B model was trained for 20 epochs and the 11B model for eight, with four A100 GPUs and DeepSpeed integration for the 11B model. Source sequences were truncated to 1024 tokens and target length was below the default 128 tokens.Source: §I.5 · p. 446

I.6 Class-Wise Performance Decomposition

  • Class-wise performance is decomposed across expert- and novice-annotated test sets because OUD categories differ and their importance varies by application.The appendix points to Table I.3 for the decomposition and Table 10.7 for average performance with and without explanations.Source: §I.6 · pp. 446–447

I.7 Scientific Artifacts

  • The artifact list includes cleantext1 (v1.1.4) for raw social-media text preprocessing and Transformers (Wolf et al., 2020; v4.35.0) for model training and inference.The listed licenses are MIT for cleantext and Apache-2.0 for Transformers.Source: §I.7 · p. 447
  • OpenAI-python3 (v1.0.0) was used to prompt GPT-series models, and the authors plan to release their codebase and dataset under an MIT license.The listed OpenAI-python3 license is Apache-2.0.Source: §I.7 · p. 447

Related papers

Loading 2608.29610v1…