PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Value-Action Architecture for Value-Driven Large Language Model Agents

TianZe Zhang, Sirui Sun, Yuhang Xie, Xin Zhang, Zhiqiang Wu, Guojie Song

arXiv:2604.05939v1cs.AIcs.HC

TL;DR

Existing LLM agents can be behaviorally rigid and fail to capture human heterogeneity. This paper introduces CVA, which uses value verification to guide behavior; experiments report reduced polarization and stronger alignment with human distributions while maintaining interpretability.

  • Problem

    Existing LLM-based agents often exhibit behavioral rigidity and stereotyping, while psychological prompting lacks intrinsic mechanisms to reproduce human behavioral heterogeneity.

    Source: §1 · p. 1

  • Method

    CVA uses a Value Verifier to align generated actions with context-activated human values, decoupling value alignment from action generation.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    CVA mitigates polarization and achieves superior alignment with ground-truth human distributions while maintaining high interpretability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CVA achieves superior fidelity in replicating individual behaviors and recovering group psychological indicators, with advantages in interpretability.

    Source: §5 · p. 9

    CVABench currently covers approximately 15,000 users across three primary domains.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise in simulating human behavior, yet existing agents often exhibit behavioral rigidity, a flaw frequently masked by the self-referential bias of current "LLM-as-a-judge" evaluations. By evaluating against empirical ground truth, we reveal a counter-intuitive phenomenon: increasing the intensity of prompt-driven reasoning does not enhance fidelity but rather exacerbates value polarization, collapsing population diversity. To address this, we propose the Context-Value-Action (CVA) architecture, grounded in the Stimulus-Organism-Response (S-O-R) model and Schwartz's Theory of Basic Human Values. Unlike methods relying on self-verification, CVA decouples action generation from cognitive reasoning via a novel Value Verifier trained on authentic human data to explicitly model dynamic value activation. Experiments on CVABench, which comprises over 1.1 million real-world interaction traces, demonstrate that CVA significantly outperforms baselines. Our approach effectively mitigates polarization while offering superior behavioral fidelity and interpretability.

1 Introduction

The introduction identifies behavioral rigidity and biased evaluation as problems in LLM-based human simulation, then presents CVA and CVABench as a value-grounded architecture and empirical testbed.

Introduction

  • LLM-based agents often exhibit rigidity and stereotyping, while psychological prompts can amplify model biases and LLM-as-a-judge evaluation can reward polarized behavior.The judge may share the agent’s pretraining biases, masking the gap from authentic human dynamics.Source: §1 · p. 1
  • Grounded in the S-O-R model and Schwartz’s Theory of Basic Human Values, CVA uses a human-data-trained Value Verifier to model context-dependent value activation and assess action alignment.The framework treats behavior as dynamic: context can activate one value and suppress another.Source: §1 · p. 2
  • CVABench provides an empirical testbed and diagnostic framework grounded in over one million real-world interactions and data from over 15,000 human participants.It supports objective assessment of behavioral rigidity and value polarization against real-world ground truth.Source: §1 · pp. 2–3
  • CVA mitigates polarization from intensified prompt-driven reasoning and achieves superior alignment with ground-truth human distributions while maintaining high interpretability.The analysis finds that explicit reasoning can produce caricatured behavior rather than nuanced simulation.Source: §1 · pp. 2–3
  • CVA decouples action generation from cognitive reasoning, using SFT and DPO to produce diverse candidate actions and a Value-Driven Verifier to align reasoning with human cognitive structures.The design targets behavioral fidelity and interpretability.Source: §1 · pp. 2–3

2 Proposed Approach

The section frames behavior as context-dependent value activation leading to action, then introduces CVA’s calibration-and-verification architecture and CVABench’s empirical evaluation system.

2.1 Psychological Foundations

  • The S-O-R formulation treats context as environmental and historical stimuli, value as a dynamic 10-dimension activation vector, and action as behavioral output modeled by P(A|C, V).The value vector represents values triggered by context, rather than static personality traits.Source: §2.1 · p. 3

2.2 Motivation: Analysis of Existing Paradigms

  • Prompt-driven reasoning can caricature authentic values, and increasing reasoning intensity exacerbates polarization and collapses population variance, motivating verifier-guided value alignment.The section contrasts this distortion with modeling actions from context and authentic value activation.Source: §2.2 · p. 3

2.3 The CVA Architecture

  • CVA follows a “Generate-then-Verify” principle with two stages: Value-Action Mapping Calibration and Value-Driven Reasoning.The architecture is designed to address polarization by separating calibration from inference-time value-driven selection.Source: §2.3 · p. 3

2.3.1 Value-Action Mapping Calibration

  • Value-Action Mapping Calibration uses supervised fine-tuning on CVABench trajectories and preference optimization to favor nuanced value consistency over caricatures.This two-stage pipeline aims to reconstruct authentic value-behavior mappings and suppress distorted reasoning pathways.Source: §2.3.1 · pp. 3–4

2.3.2 Value-Driven Reasoning

  • CVA’s Value-Driven Verifier scores generated candidate actions against the target values and selects the highest-scoring action, using an independent discriminator trained on authentic context-value-action triplets.The inference protocol generates candidates conditioned on context and value profile, scores their alignment, and selects the optimal candidate.Source: §2.3.2 · p. 4
  • CVA’s verifier attention makes the value dimensions informing an action transparent, and behavioral fidelity peaks at a limited computational budget rather than improving monotonically.The authors describe the verifier as explicitly modeling value-based action selection and note that exhaustive evaluation does not yield monotonically higher fidelity.Source: §2.3.2 · pp. 4–5

2.4 Evaluation System: CVABench

  • CVABench aggregates over 1.1 million authentic interaction traces from 15,571 users across three behavioral domains.It is introduced as a benchmark for training and rigorous validation of the CVA architecture.Source: §2.4 · p. 5
  • CVABench evaluates individual behavioral consistency with prediction accuracy and MSE, and population-level value distribution with variance and polarization using GPV (Ye et al., 2025).GPV infers value profiles from generated behavioral logs rather than self-reported questionnaires.Source: §2.4 · p. 5

3 Experiments

Across CVABench experiments, prompt-driven psychological inference sharpens and polarizes simulated value distributions, while CVA is evaluated for behavioral fidelity, component effects, and interpretability.

3.1 Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents

  • Stronger prompt-driven psychological inference shifts simulated value means toward extremes and lowers variance, sharpening distributions and collapsing population diversity.The observed pattern appears across baseline simulations of the Stimulation value dimension; Figure 3 visualizes the polarization and solidification.Source: Figure 3 · p. 5, §3.1 · p. 6

3.2 Benchmarking CVA Model Performance

  • CVA significantly enhances behavioral fidelity across all domains and mitigates the rigidity and value polarization found in group simulations.Baselines do not consistently improve accuracy with in-context learning or explicit prompt-driven reasoning; complex prompt-driven psychological modeling can underperform straightforward data-driven role-playing.Source: Table 3 · p. 6, §3.2 · pp. 6–7

3.3 Ablation Study on CVA Model

  • The full CVA architecture achieves the highest performance across ablation metrics, while inference gains diminish beyond four reasoning rounds.The ablation incrementally adds SFT, DPO, and Verifier Reasoning; Figure 4 reports sentiment accuracy trends across inference intensities.Source: Table 4 · p. 7, §3.3 · p. 7, Figure 4 · p. 7

3.4 CVA Model Interpretability Case Study

  • Verifier embeddings align with Schwartz’s circular value structure, while cross-attention weights reveal context-dependent value activations at the word level.The only noted structural deviation is in Achievement; word-level analyses filter stop words and extraneous noise.Source: §3.4 · p. 7

4 Related Works

The related work motivates automated value measurement and contrasts prompt-driven with trained human-like agents; the paper adopts GPV and reports evidence on polarization and CVA-based training.

4.1 Automated psychometric assessment for LLM Agents

  • GPV provides automated, non-reactive value measurement that is reported as reliable, accurate, scalable, and economical compared with traditional questionnaires (Ye et al., 2025).The passage says its measurements are unaffected by reactivity bias and that GPV has been extensively tested with real human subjects.Source: §4.1 · p. 8
  • The authors corroborate reported LLM-agent opinion polarization and quantitatively measure it in behavioral simulation tasks (Li et al., 2025; Xie et al., 2025).Source: §4.1 · p. 8

4.2 LLM Role-Play Agents and LLM Human-Like Agents

5 Conclusion

The conclusion presents CVA as a response to polarization in prompt-driven agents, reporting superior behavioral fidelity, group psychological indicator recovery, and interpretability on CVABench.

5 Conclusion

  • Prior prompt-driven methods guided by specific psychological traits can polarize psychological metrics and produce extreme behaviors that diverge from realistic human decision-making.Source: §5 · pp. 8–9
  • On CVABench, CVA achieves superior fidelity in replicating individual behaviors and recovering group psychological indicators, with advantages in interpretability.The conclusion also characterizes its efficacy and robustness as exceptional.Source: §5 · p. 9
  • CVA mitigates polarization by reducing base-model psychological biases and introducing a value-driven verifier.Source: §5 · p. 9

6 Limitations

The authors identify limitations in benchmark breadth, value measurement, baseline scale, and safe behavioral simulation, while describing safeguards and directions for future work.

Scale and Domain Coverage

  • CVABench covers approximately 15,000 users across three domains, motivating expansion to broader contexts such as consumer behavior and cultural preferences.Source: §6 · p. 9

Value Measurement and Bias

  • GPV offers greater stability and construct validity than human self-reports and other LLM-based tools, but measuring complex values remains difficult and bias-prone (Ye et al., 2025).Source: §6 · p. 9
  • Ground-truth supervision and a verifier that aligns values with heterogeneous behaviors help avoid a self-reinforcing evaluator-generator loop, even if value measurements contain noise or bias.The verifier is not the sole arbiter of quality, and the authors say the approach captures value-action correlations without trapping the model in a self-referential loop.Source: §6 · p. 9
  • Future work calls for more sophisticated value modeling and large-scale human-subject experiments to examine the study’s HCI implications.Source: §6 · p. 9

Baseline Comparisons

  • Computational demands for large-scale simulations constrained the number of baselines, and the authors are optimizing evaluation to enable broader comparisons.Source: §6 · p. 9

Ethical Considerations and Potential Risks

  • CVABench removes PII and keeps users disjoint across its three behavioral domains, preventing cross-domain reconstruction of a comprehensive digital persona.Source: §6 · p. 9
  • Authentic internet training data risks toxic or biased outputs; extreme hate speech is filtered, but some human imperfections remain and outputs do not represent the authors’ values.AI assistants were used only for grammatical polishing and formatting; scientific claims and intellectual content are original.Source: §6 · p. 9

A Lead-in Case Study … 10 end

A preliminary case study illustrates behavioral rigidity in GPT-4o, while the implementation sections describe role-play and prompt-driven value-reasoning baselines and their experimental settings.

A Lead-in Case Study

  • 0.93 was GPT-4o’s probability of biased decision-making at Hedonism 0.6; gym choice was nearly absolute at ≤0.5 with Self-Direction fixed at 0.9.The evaluation covered 100 trials.Source: Figure 7 · p. 12, A Lead-in Case Study · p. 12
  • The case study is preliminary and illustrative; subsequent larger-scale experiments are described as systematically examining behavioral rigidity and polarization.Source: A Lead-in Case Study · p. 12

B.1 Prompt-Driven Role-Play Agent

  • The role-play baseline uses working and long-term memory of a user’s behavior, retrieving relevant history while excluding explicit value information from its decision input.Long-term memory is constructed heuristically from available user memories to fit the model’s context window.Source: §B.1 · pp. 12–13
  • In the illustrative role-play example, the agent chose the gym after a long workday and justified exercise as restoring control and relieving stress.The prompt specified Self-direction 0.9 and Hedonism 0.4.Source: §B.1 · p. 13

B.2 Prompt-Driven Reasoning Agents

  • Prompt-driven value reasoning generates candidate actions, evaluates their value satisfaction, and selects the action with the highest value consistency.The baseline uses a generate-then-evaluate pipeline, with a reasoner scoring candidates against the agent’s values and context.Source: §B.2 · pp. 13–14

10 end

  • Greater reasoning strength broadens and deepens exploration of candidate actions.The paper describes reasoning strength as modulating the search space.Source: §10 · p. 14
  • The reasoning experiments set candidate count K = 3 and varied reasoning rounds T across {0, 1, 2, 4, 8}; the zero-strength variant still uses value preferences.The zero-strength baseline uses the initial action and is explicitly value-conditioned, unlike the vanilla role-play baseline.Source: §10 · p. 14

B.3 Training Required Agents

The required-agent baseline is trained on CVA Bench data with a one-epoch SFT-then-DPO pipeline, using ground-truth human behaviors as SFT targets and similarity-based preference construction for DPO.

B.3 Training Required Agents

  • The baseline uses a two-stage pipeline of Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO), with each stage trained for one epoch.Training uses the CVA Bench dataset.Source: §B.3 · p. 14
  • SFT conditions on role-playing system prompts and targets ground-truth human behaviors, optimizing with standard Cross-Entropy loss.Source: §B.3 · p. 14
  • For DPO, the authors sample K = 10 actions from the SFT-tuned model at temperature 0.8, score candidate similarity to ground truth, and construct chosen/rejected pairs.Source: §B.3 · p. 14
  • The DPO objective combines sigmoid DPO loss, BCO pair loss, and SFT regularization as Ltotal = 1.0 · LDPO + 0.2 · LBCO + 1.2 · LSFT.Source: §B.3 · pp. 14–15

C Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details · D CVABench Metrics and Detailed Results

As prompt-driven agents reason through more iterations, their value-alignment scores become less diverse and more polarized, exposing limits in prompt-only modeling of human values. The appendix reports group-level results across all ten value dimensions.

C Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details

  • Relying only on LLM instruction-following and in-context learning for prompt-driven inner-state modeling lacks individual-level precision and exhibits group-level psychological bias.Source: C Psychological Bias of Group… · p. 15
  • The appendix presents group-level results for all ten value dimensions, supplementing the single representative dimension shown in the main paper due to space constraints.Source: C Psychological Bias of Group… · p. 15
  • As iterations increase across k ∈ {0, 1, 2, 4, 8}, alignment-score variance decreases while scores shift toward −1 and 1, reducing diversity and making positions more binary.Source: C Psychological Bias of Group… · p. 15
  • In the alignment scores, 1 indicates strong support for a value and high consistency, while −1 indicates strong opposition and high inconsistency.Source: C Psychological Bias of Group… · p. 15
  • Table 5 reports a monotonic decrease in standard deviation across dimensions and average preferences deviating from ground truth toward extremes; Figure 8 shows distributions narrowing into rigid modes.Source: C Psychological Bias of Group… · p. 15
  • Repeated explicit reasoning pushes the model toward simplified, exaggerated positions, failing to capture the subtle and moderate value alignment of complex human behavior.Source: C Psychological Bias of Group… · p. 15

D.1 Metrics Explanation … E.1 Word-to-Value Projection Analysis

The appendix defines CVABench’s behavioral and linguistic evaluation metrics, reports that CVA agents often achieve lower linguistic Wasserstein Distance than prompt-driven agents, and analyzes how words map to Schwartz value activations.

D.1 Metrics Explanation

  • Social Media Review evaluation uses Accuracy for rating and sentiment predictions, plus 1-Wasserstein Distance between generated and ground-truth Type-Token Ratio distributions.Accuracy compares predicted and ground-truth rating or sentiment labels.Source: §D.1 · p. 15

Panel B: Mean Analysis (Absolute Mean Difference relative to GT)

  • Increasing value-inference strength is detrimental in the reported analysis, simultaneously reducing population variance toward rigidity and producing polarized mean deviations from ground truth.Table 5 analyzes strengths from value_infer_0 to 8; its Average column distinguishes insufficient guidance from the dominant overfitting trend.Source: Table 5 · p. 17

D.2 Detailed Results

  • CVA agents frequently achieve smaller Wasserstein Distance values than prompt-driven human-like agents, while adding reasoning rounds does not necessarily improve human alignment.The comparison covers linguistic indicators across Media Review and Online Conversation; lower WD indicates better alignment with human linguistic patterns.Source: Table 6 · p. 18, §D.2 · p. 19

Panel B: Mean Analysis (Absolute Mean Difference relative to GT)

  • Conversation Discourse uses Attitude Accuracy and TTR-distribution Wasserstein Distance, while mobility uses Category Accuracy for POIs and MSE for stay duration.The mobility duration prediction is measured in minutes.Source: Panel B: Mean Analysis (Absolute Mean… · p. 18
  • Value Distribution Variance measures simulated-versus-human variance deviation: excessive variance indicates unstable behavior, collapse indicates rigidity or polarization, and values closer to 0 indicate better alignment.The measure assesses value-alignment-score diversity across the three domains.Source: Panel B: Mean Analysis (Absolute Mean… · pp. 18–19

D.2 Detailed Results

  • The seven linguistic metrics compare review and sentence lengths, Type-Token Ratio, and noun, adverb, adjective, and verb frequency distributions using Wasserstein Distance.These metrics extend the linguistic analysis in the main text.Source: §D.2 · p. 19

E.1 Word-to-Value Projection Analysis

  • The word-to-value projection estimates each word’s value activation using TF-IDF-weighted cross-attention across contexts, then row-normalizes scores to show each word’s relative value preference.The resulting relevance score is interpreted as expected activation given a word’s presence; normalization helps identify lexical markers specific to value dimensions.Source: §E.1 · pp. 19, 21
  • The analysis visualizes lexical associations with a row-normalized heatmap and value-specific word clouds, which distinguish relative preference from absolute relevance.Heatmap values of 1.0 mark a word’s strongest relative value association; word-cloud size reflects unnormalized relevance.Source: Figure 10 · p. 20, §E.1 · p. 21, Figure 11 · p. 21

E.2 Quantitative Analysis of the Circular Embedding Structure · F CVA Architecture Training Details · F.1 Generator Training: Value-Conditional De-biasing

The verifier recovers the theoretical circular structure of human values, while its embeddings retain specific semantic entanglements. The generator is trained with value-conditioned context-action inputs in both SFT and DPO.

E.2 Quantitative Analysis of the Circular Embedding Structure

  • Circular Inversion Score measures preserved pairwise relationships by comparing PCA-derived angular value order with the theoretical Schwartz circumplex order.It finds the minimum rotational inversion distance and normalizes the score, with 1.0 indicating a perfect reconstruction.Source: §E.2 · p. 22
  • The trained verifier demonstrated a significant capability to recover the latent circular structure of human values, with CIS 0.75; the metric’s perfect-fit reference is 1.00.Source: §E.2 · p. 22, Table 8 · p. 22
  • Universalism–Benevolence and Tradition–Conformity adjacency was largely preserved, capturing coarse-grained relationships between value clusters.These adjacencies correspond to the Self-Transcendence and Conservation sectors, respectively.Source: §E.2 · p. 23
  • Achievement shifted from Self-Enhancement toward Self-Direction, while Hedonism’s adjacency to Tradition suggests difficulty disentangling gratification from restraint.The passage attributes the latter entanglement possibly to data sparsity in opposing pairings.Source: §E.2 · p. 23

F.1 Generator Training: Value-Conditional De-biasing

  • The generator’s SFT learns Pθ(A|C, V) rather than the baseline Pθ(A|C), and DPO also uses joint context and value inputs to align generation with value profiles.Explicit value conditioning is intended to mitigate base-LLM psychological biases and decouple generation from prior biases.Source: §F.1 · p. 23

F.2 Verifier Architecture and Training

The Value-Guided Verifier models human preference selection by scoring action consistency with context and value profiles. It is trained to rank actions that better align with human ground-truth behavior higher.

Architecture

  • Context-guided cross-attention refines the value embedding, which is combined with the action embedding to predict a value-consistency score.The multiencoder processes the action and context with a text encoder and the value profile with a value encoder.Source: §F.2 · p. 23

Training Objective

  • Five candidate actions sampled per context from the DPO-tuned model at temperature 0.8 are ranked by linguistic similarity to ground-truth behavior.Preferred and rejected actions are denoted Aw and Al, respectively.Source: §F.2 · p. 23
  • The pairwise ranking loss trains the verifier to give higher consistency scores to preferred actions that better align with human ground truth under the value profile.The objective maximizes the score margin between preferred and rejected actions.Source: §F.2 · pp. 23–24

Related papers

Loading 2604.05939v1…