PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Surface Alignment: Grounding the Dynamics of Situational Understanding and Generative Control in LLMs

Chenghao Yang

arXiv:2608.29610v1cs.CL

TL;DR

Surface alignment can produce fluent, safe, and stylistically consistent behavior without reliable grounding. This thesis formulates Grounded Alignment, evaluates situational and generative grounding, and develops dynamic controls; its applications span grounded copywriting, model collaboration, verifiable reinforcement learning, and addiction support. The studies report brittle state tracking, surface-heuristic dependence, premature stylistic collapse, and limits of self-reflection, motivating agents grounded in context and generation.

  • Problem

    The thesis addresses limited grounding beneath observable aligned behavior, including unreliable situation tracking and generation understanding.

  • Method

    The thesis analyzes input-side situational grounding and output-side generative grounding, then introduces dynamic-control mechanisms and applications.

  • Results

    The studies find failures in state tracking, reliance on surface heuristics, premature stylistic collapse, and weak understanding of model-generated outputs.

  • Takeaways & Limitations

    The work offers grounded interaction mechanisms including context engineering, base-aligned collaboration, annealed sampling, and rationale-grounded communication for high-stakes domains.

  • Takeaways & Limitations

    The situational-grounding evaluation uses a controlled synthetic environment, limiting the scope of what it directly establishes.

Abstract

from arXiv · show

The current alignment tuning paradigm for Large Language Models (LLMs) prioritizes surface-level behaviors -- fluency, safety, and tonal consistency. While effective for casual chat, this thesis argues that such surface alignment masks a lack of grounding, creating models that are stylistically confident but situationally brittle. We propose a framework of Grounded Alignment, analyzing how models process context (Input) and structure generation (Output), then aligning these grounded behaviors to human needs. First, we evaluate failures in Situational Grounding. SitTest shows that despite large context windows, state-of-the-art models struggle to maintain a consistent "mental model" of a changing environment. ReCode further shows that models rely on surface heuristics rather than deep syntactic dependencies: they "read" extensive histories without truly "understanding" the evolving situation. Second, we evaluate Generative Grounding. We introduce the Branching Factor (BF) to map LLM generation, finding that standard alignment tuning constricts this landscape into premature stylistic collapse. Hindsight further shows that models often fail to understand their own generations. Finally, we propose Dynamic Control for grounded interaction. AI Realtor demonstrates context engineering to compensate for poor situational grounding. Base-Aligned Model Collaboration decouples exploration from stylistic constraints. We also present Annealed Sampling for verifiable reinforcement learning and apply these ideas to Addiction Support, where model-generated rationalization offers a communication interface for high-stakes domains. Collectively, this work moves beyond surface alignment toward agents anchored in both context and generation.

7 AI REALTOR: TOWARDS GROUNDED PERSUASIVE LANGUAGE GENERATION FOR AUTOMATED COPYWRITING

This section presents AI Realtor as a grounded persuasive-language-generation framework for automated real-estate copywriting.

  • 7.4 The Agentic Implementation of AI Realtor: AI Realtor combines grounding, personalization, and marketing modules for automated real-estate copywriting.The listed components include credible-feature prediction, preference alignment, and retrieval-augmented surprisal capture.
  • 7.2 A Benchmark for Grounded Persuasion: The work evaluates AI Realtor through human and AI feedback on grounded persuasion.The chapter outline identifies a real-estate benchmark and separate human-feedback and AI-feedback evaluations.

8 OPTIMIZING DIVERSITY AND QUALITY THROUGH BASE-ALIGNED MODEL COLLABORATION

This section introduces Base-Aligned Model Collaboration as a framework for improving diversity and quality through routing between base and aligned models.

  • 8.3–8.6: The chapter includes routing-strategy design, experimental setup, contribution distribution, switching frequency, and an inherent early-stopping failure mode.

9 ANNEALED SAMPLING FOR VERIFIABLE REINFORCEMENT LEARNING

The thesis examines failures of situational and generative grounding, then proposes dynamic-control mechanisms spanning context engineering, model collaboration, annealed sampling, and rationale-grounded support.

  • Situational Grounding: ReCode shows that models rely on surface heuristics rather than resolving deep syntactic dependencies, especially when surface cues are removed.Variable naming conventions and docstrings are cited as examples of cues whose removal exposes fragility.
  • Dynamic Control: Dynamic Control introduces context engineering, Base-Aligned Model Collaboration, and Annealed Sampling to address grounding and exploration constraints.The applications include AI Realtor, verifiable reinforcement learning, and rationale-grounded Addiction Support.
  • Generative Grounding: Branching Factor maps the generation landscape, while alignment tuning constricts it into premature stylistic collapse.The thesis characterizes aligned models as entering low-entropy trajectories from the first token and pruning valid semantic space.
  • Generative Grounding: Hindsight shows that models often lack a deep semantic grasp of their own generations, so unsupported self-reflection can reinforce hallucinations.
  • Dynamic Control: Rationale-grounded Addiction Support uses transparent model reasoning to improve detection of subtle Opioid Use Disorder stages and build trust.
  • Situational Grounding: SitTest finds that models lose coherent environment states across time despite simple tasks and access to full dialogue histories.Performance degrades as interactions increase, with state loss, propagated errors, and hallucinated updates.

Changes

The paper finds that language models struggle to maintain coherent situational states over evolving interactions and remain vulnerable to semantically preserving prompt perturbations. It introduces synthetic evaluation environments and ReCode to diagnose these limitations, while noting important scope boundaries.

  • State tracking over steps: Step-EM falls much faster than State-EM as the number of interaction steps increases, showing difficulty characterizing the complete environment state.Under normal instructions, State-EM reaches 90%, while Step-EM can be up to 70% lower in 2-shot and 40–50% lower in 5-shot settings.
  • Logical inconsistency: Counterintuitive instructions reduce Step-EM to 10–20% in most settings, compared with more than 20% or 30% under normal instructions.The pattern suggests reliance on standard language-to-logic mappings rather than fully understanding the instructions.
  • Instruction and language effects: Synthetic language sometimes improves performance by helping models detach from unhelpful pretraining biases and rely more robustly on in-context information.The effect is described as variable rather than uniformly beneficial.
  • Fine-grained error analysis: State-tracking errors accumulate through both propagated errors and failures to retain previously correct, unchanged states.Correct Update and especially Maintain Correctness cases decrease over time, alongside increases in Dirty Read transitions.
  • Overall findings: The evaluation concludes that ChatGPT lacks robust situational state tracking despite access to the complete dialogue history.The proposed synthetic environment is intended to diagnose limitations relevant to multi-round interaction, instruction following, and alignment research.
  • ReCode: InCoder-6B and CodeGen-16B-mono succeed on nominal prompts but fail after meaning-preserving docstring paraphrasing or function-name perturbations.These failures indicate sensitivity to surface form despite preserved prompt meaning.
  • ReCode: ReCode tests code-generation robustness with more than 30 natural transformations, including changes to docstrings, names, syntax, and formatting.Human and quantitative similarity evaluations indicate that most perturbations preserve semantic meaning and naturalness.

Appendix G.8.1).

The section presents Base-Aligned Collaboration (BACO) and Exploratory Annealed Decoding (EAD) as strategies for separating exploration from stylistic constraints and balancing exploration with exploitation. BACO improves human-perceived diversity and quality at comparable automatic quality, while EAD improves RL exploration and sample efficiency across settings.

  • BACO: Base-Aligned Collaboration for Diversity and Quality: BACO achieves substantially higher turning-point and arousal diversity than the aligned baseline at comparable quality in long-form generation.Its outputs are judged most creative in 79.6% of NoveltyBench prompts and 61.8% of WildChat prompts.
  • BACO: Base-Aligned Collaboration for Diversity and Quality: BACO receives higher human quality ratings than the aligned baseline despite near-identical automatic quality scores.Human ratings are 4.04 vs. 2.83 on NoveltyBench and 3.83 vs. 3.44 on WildChat, while automatic scores are 5.93 and 5.85.
  • BACO: Base-Aligned Collaboration for Diversity and Quality: BACO produces human-perceivable diversity across both format and content dimensions, with significant large diversity win rates.The overall diversity evaluation reports Fleiss’ κ=0.268.
  • BACO: Base-Aligned Collaboration for Diversity and Quality: BACO’s base-model contribution and switching frequency are high early in generation and decrease over time across three tasks.The resulting early stopping is emergent and can be detected by length, enabling a simple restart mitigation with marginal cost.
  • Exploratory Annealed Decoding: EAD dynamically anneals temperature to explore early and exploit late, aligning sampling with the generation process’s changing entropy.Early branching consistently outperforms late branching, and EAD is designed as a plug-and-play RLVR enhancement.
  • Exploratory Annealed Decoding: EAD improves sample efficiency and outperforms fixed-temperature baselines across RLVR algorithms, models, rollout settings, and test-time inference.The gains remain significant on larger models, while truncated importance sampling is essential for stable Worst@16 gains.
  • Exploratory Annealed Decoding: EAD’s benefits depend on a sufficiently smooth decreasing temperature schedule rather than reverse annealing or an effectively fixed schedule.Very small dmax causes performance drops, large initial d0 makes exploration overly conservative, and reverse annealing fails to outperform standard sampling.
  • Limitations: The study identifies scaling behavior as incompletely explored because of limited computational resources.The authors retain current settings with reference to available resources and identify broader scaling as future work.

A.5 Comparison with other models

This appendix compares models and documents ReCode’s perturbation procedures for robustness evaluation in code generation. The comparisons show substantial situational-understanding gaps for open-source models, while the benchmark applies semantics-preserving transformations to code prompts.

  • A.5 Comparison with other models: Vicuna-13B outputs are often hard to parse and incomplete, and its best observed State-EM is only 32% even with five in-context samples.The result is reported in the SL environment without counter-intuitive instructions.
  • A.5 Comparison with other models: ChatGPT achieves 94% situational-understanding performance, leaving a reported gap relative to open-source models.The comparison explicitly characterizes the open-source capability gap against ChatGPT.
  • ReCode perturbations: ReCode constructs perturbed HumanEval and MBPP prompts by modifying docstrings, function names, whitespace, and code while preserving the same baseline code block for comparison.The benchmark uses transformations including synonym changes, case changes, inflection, loop conversion, operand swapping, and dead-code insertion.
  • ReCode perturbations: Docstring perturbations include paraphrasing, character typos, case changes, inflectional variation, synonym operations, tense changes, and whitespace edits.Several transformations are deterministic, while others use controlled random probabilities to balance naturalness and perturbation success.
  • ReCode perturbations: Function-name perturbations include character swaps, inflection, synonym substitution, case changes, and deterministic conversion between camel-case and snake-case.References to the modified function name in the prompt are also changed to maintain consistency.
  • ReCode perturbations: Code transformations include dead-code insertion, equivalent for/while conversion, and operand swapping with operator adjustment for semantic equivalence.Dead-code blocks may be zero-iteration loops or always-false conditions.

B.2 Limitations

The appendix documents limitations and validation details for ReCode and related evaluations. ReCode is restricted to Python function-completion settings and evaluates robustness without itself providing a mitigation strategy.

  • ReCode limitations: ReCode evaluates robustness on Python HumanEval and MBPP function-completion tasks, limiting coverage of other languages and code-completion use cases.The authors state that the transformations are generalizable and could be extended to other languages and code-related datasets.
  • ReCode limitations: ReCode is designed for robustness evaluation and cannot itself mitigate a model’s lack of robustness.The benchmark can generate a comprehensive collection of perturbed data for future robust-training strategy design.
  • Failure cases: Dead-code insertion and newline perturbations can mislead code-generation models, especially when completion is required immediately after the perturbation.CodeGen-16B-mono may output only a newline after inserted meaningless code or produce wrong completions after an empty newline.
  • Failure cases: ButterFingers docstring perturbations can cause large performance drops because introduced typos may lead models to misunderstand the target docstrings.The appendix illustrates this failure mode with CodeGen-16B-mono.
  • Evaluation limitations: Human annotation agreement is moderate, with average Fleiss’ κ of 0.52 overall, 0.36 for semantic similarity, and 0.36 for naturalness measurements.Bootstrap 95% confidence intervals are reported as [0.515, 0.528] and [0.358, 0.364] for the corresponding measurements.
  • Evaluation limitations: Sentence-transformer similarity is less reliable for isolated function names because they lack complete sentence context.The appendix notes that minor word changes can produce large cosine-similarity changes, such as a 0.21 score for a case-changed name.
  • Self-reflection results: The self-reflection findings generalize across ChatGPT and LLaMA-2-7b-chat, but self-reflection improves TruthfulQA while worsening HotpotQA.Its benefit is greater when initial responses are incorrect and questions are more difficult.

C.10 Results on Mixtral-8x7B-v0.1

On Mixtral-8x7B-v0.1, self-reflection prompting generally fails to improve performance and can harm it, unlike some gains observed for ChatGPT.

  • On TruthfulQA and HotpotQA, self-reflection improves TruthfulQA performance but harms HotpotQA performance compared with ChatGPT.
  • Across HotpotQA difficulty levels and comprehension conditions, self-reflection prompting fails to improve performance.
  • Artificial-response experiments show that self-reflection prompting is not always harmful.
  • Self-reflection improves easy-question performance under 0% response accuracy, but is harmful in most cases.
  • Majority voting changes minimally with self-reflection, suggesting Mixtral is comparatively difficult to steer through self-reflection prompts.
  • The authors speculate that Mixtral lacks sensitivity or competence in self-reflection and may need additional training to unlock its potential.

D.1 Adaption for FastSHAP Baseline

The appendix documents implementation choices for the FastSHAP adaptation and BF studies, including constrained compute, dataset preprocessing, and entropy-estimation experiments.

  • The amortized model uses a pretrained BERT encoder, but FastSHAP requires two GPU-hosted language models and batch size 1 with 32 perturbation samples.
  • This setup teaches the amortized model to approximate KS-32, described as an unreliable interpretation method.
  • FastSHAP optimization is unstable despite extensive hyperparameter search, with a learning rate of 1e-6 selected for training.
  • Monte Carlo entropy estimates increase with sample count and underestimate true entropy at small sample sizes because they miss the distribution’s long tail.
  • For Llama-2 and Llama-3, base models average approximately BF 12 versus approximately BF 1.2 for aligned models, with BF often declining during generation.
  • Prompt complexity affects BF inconsistently across tasks: it can increase BF in Cognac but generally lowers BF in News Generation.

E.7 Generalization to Additional Tasks and Models

Additional tasks and models support the paper’s BF findings while separating autoregressive self-narrowing from alignment effects and distinguishing BF from surface-diversity measures.

  • BF conclusions remain robust on XSUM, AYA, and Qwen3-4B evaluations.
  • OLMo-2 shows alignment-related BF reduction across Creative StoryGen and MMLU, but less pronounced than in Llama models.
  • BF measures the entropy rate of the underlying conditional distribution, not similarity between sampled outputs.
  • BF and surface-level diversity correlate only weakly, so the measures are complementary rather than competing.
  • Autoregressive self-conditioning generally narrows BF over time, while unexpected context can reopen the consideration set and raise BF.
  • Alignment lowers the absolute BF band and accelerates re-collapse without changing the qualitative dynamics.
  • Across models and tasks, BF has no consistent correlation with Distinct-N or Min-K%.

F.4 Increasing Length in RL Training

EAD selectively lengthens reasoning for harder problems during RL training, while the proposed diversity-quality collaboration framework remains subject to computational and scope limitations.

  • F.4 Increasing Length in RL Training: EAD incentivizes longer, more effective reasoning chains for difficult problems, especially in 7B models.
  • F.4 Increasing Length in RL Training: Unlike temperature sampling, EAD later increases response length selectively for harder problems and boosts final performance.
  • Limitations: The collaboration framework incurs additional per-token FLOPs relative to single-model inference and is not yet perfectly speed-engineered.
  • Limitations: The study focuses on inference-time improvement and leaves learned routers, longer-horizon tasks, and intermediate alignment checkpoints for future work.
  • G. Optimizing Diversity and Quality Through Base-Aligned Model Collaboration: BACO combines base-model diversity with aligned-model quality through inference-time token-level routing.
  • G. Optimizing Diversity and Quality Through Base-Aligned Model Collaboration: BACO targets a stable diversity-quality trade-off, whereas existing inference-time methods can require multiple decoding passes or degrade quality.

G.2 BACO Framework Implementation and Router Details

BACO combines base and aligned models through token-level routing, with safeguards for model switching, termination, and tokenization. Its prototype incurs roughly 2× per-token FLOPs but can outperform or match several inference-time diversity baselines in runtime and diversity–quality trade-offs.

  • Tokenization alignment: BACO keeps tokens forming one semantic unit with the same model to prevent broken punctuation and malformed words caused by tokenization mismatches.The constraint applies particularly around punctuation, special tokens, and rare words.
  • Framework and routing: BACO performs inference-time token-level collaboration by dynamically routing generation between base and aligned models.The framework uses an ordered cascade of routing strategies, with the first firing strategy determining the active model.
  • Cost and runtime: BACO requires two forward passes per decoding step at worst, producing approximately 2× FLOPs overhead per token independent of desired group size n.This contrasts with approaches requiring n sequential passes or at least n expanded beams.
  • Decoding procedure: The decoding algorithm starts with the aligned model, routes at word boundaries, samples from the selected model, and enforces aligned-controlled termination.If the selected model emits EOS before the aligned model’s top prediction does, BACO resamples while excluding EOS.
  • Cost and runtime: The current prototype is slower than single-model baselines but comparable to or faster than several diversity-promoting baselines while substantially outperforming them on diversity–quality.The implementation is not yet engineered for speed, and caching, overlapping computation, speculative decoding, LoRA, and KV-cache sharing are identified as complementary optimizations.
  • Router strategies: The router supports probability-, entropy-, content-, punctuation-, random-, judge-, and multi-condition routing strategies.Strategies may route to the base model when specified thresholds or linguistic conditions are met, otherwise retaining the aligned model.

G.6 Validation on Verifiable Benchmarks

BACO’s diversity–quality improvements extend from open-ended generation to verifiable instruction-following and mathematical-reasoning benchmarks. It maintains high accuracy while achieving higher diversity, and its advantage persists across additional tasks, model families, and longer prompts.

  • IFEval: At matched quality levels on IFEval, BACO achieves consistently higher diversity than the aligned model.IFEval uses automatically verifiable instruction-following constraints and evaluates accuracy alongside diversity.
  • GSM8K: On GSM8K, BACO maintains high accuracy, including 90% Pass@10, while achieving substantially higher diversity across operating points.The benchmark measures Acc and Pass@10 alongside diversity over sampled solution outputs.
  • Cross-benchmark validation: The qualitative trends on IFEval and GSM8K mirror open-ended generation results, indicating that BACO’s gains extend beyond open-ended evaluation metrics.These benchmarks were used to test whether the improvements were artifacts of open-ended evaluation.
  • NoveltyBench: On NoveltyBench, BACO outperforms all baselines on every metric except lexical Dominance and improves overall Coverage by 0.274.It dominates 39.9% of the overall diversity–quality frontier, including 48.8% of the semantic frontier.
  • WildChat: On WildChat, BACO-P-PUNC improves Coverage by 29.8% over the aligned baseline and dominates 30.8% of the frontier.Its strongest advantage is semantic diversity, where it dominates 48.5% of the frontier, and the pattern persists for longer prompts.
  • Narrative-Discourse: BACO achieves 13.5% higher Coverage and dominates 20.3% of the overall diversity–quality frontier on Narrative-Discourse.This evaluation targets sustained creativity and narrative arc beyond instruction following and dialogue.

H.6.2 Human Evaluation

The human evaluation examines credibility, grounding of hard and soft attributes, and user preferences for generated real-estate descriptions. Results indicate strong performance on concrete attributes, comparable performance on subjective judgments, and sensitivity to users’ preferred writing length and style.

  • Evaluation design: Human raters evaluate description credibility on a 1–5 scale and assess whether hard and soft attributes are reflected when mentioned.Credibility captures whether a persuasive description feels trustworthy, while factuality is defined by grounding claims in the provided attribute set.
  • Credibility and subjective judgment: AI Realtor is comparable to humans on evaluations requiring subjective judgment, while human-written descriptions score lowest in credibility.The reported results include AI Realtor’s performance on soft attributes and credibility evaluations.

I.3 In-Context Samples in Few-Shot Learning Settings

The few-shot setting uses thirteen representative in-context examples for each opioid-usage category, with category proportions informed by expert discussion and held-out experiments. GPT-3 outputs require normalization before evaluation, while the same processing produces no significant changes for GPT-4.

  • I.3 In-Context Samples in Few-Shot Learning Settings: Thirteen in-context examples are used for each opioid-usage category in the few-shot learning setting.The examples were selected as representative samples after discussions with the experts who provided annotations.
  • I.3 In-Context Samples in Few-Shot Learning Settings: Category distributions are determined using preliminary experiments on held-out data.
  • I.4 Post-processing needed for processing GPT-3 outputs: GPT-3 outputs cannot generally be treated as exact matches because they may contain formatting deviations and typos.Post-processing removes content after newlines, extracts answers from numbered responses, restores morphological forms, and corrects typos.
  • I.4 Post-processing needed for processing GPT-3 outputs: Applying the same post-processing to GPT-4 produces no significant changes, which may indicate stronger instruction-following than GPT-3.
  • Fully Supervised Fine-Tuning Details: Fully supervised models are fine-tuned with model-specific implementations, sequence-length limits, validation examples, and optimizer settings.The reported setup includes DeBERTa-v3-large and two T5 variants, with source lengths capped at 1024 and target lengths below 128 tokens.
Loading 2608.29610v1…