PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-Value-Action Architecture for Value-Driven Large Language Model Agents

TianZe Zhang, Sirui Sun, Yuhang Xie, Xin Zhang, Zhiqiang Wu, Guojie Song

arXiv:2604.05939v1cs.AIcs.HC

TL;DR

Existing LLM-based agents can be behaviorally rigid, while self-referential LLM-as-a-judge evaluations may fail to expose this mismatch with authentic human behavior. The paper proposes CVA, which decouples action generation from cognitive reasoning through a Value Verifier modeling context-dependent value activation. On CVABench, CVA mitigates value polarization and achieves superior behavioral fidelity, group-indicator recovery, and interpretability, while the benchmark remains limited in scale and domain coverage.

  • Problem

    Existing agents exhibit behavioral rigidity and stereotyped outputs, while prompt-driven reasoning and self-referential evaluation do not reliably capture heterogeneous human behavior against empirical ground truth.

  • Method

    CVA grounds behavior in the S-O-R model and Schwartz’s values theory, decouples action generation from reasoning, and uses a Value Verifier trained on authentic human data to model dynamic value activation.

  • Results

    CVA mitigates value polarization and achieves superior fidelity in replicating individual behavior and recovering group psychological indicators while maintaining high interpretability.

  • Takeaways & Limitations

    Evaluating against authentic human distributions, rather than relying solely on self-referential judgments, supports more behaviorally faithful and interpretable human-like agents.

  • Takeaways & Limitations

    CVABench currently covers approximately 15,000 users across three primary domains, limiting validation of generalizability to broader contexts.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown promise in simulating human behavior, yet existing agents often exhibit behavioral rigidity, a flaw frequently masked by the self-referential bias of current "LLM-as-a-judge" evaluations. By evaluating against empirical ground truth, we reveal a counter-intuitive phenomenon: increasing the intensity of prompt-driven reasoning does not enhance fidelity but rather exacerbates value polarization, collapsing population diversity. To address this, we propose the Context-Value-Action (CVA) architecture, grounded in the Stimulus-Organism-Response (S-O-R) model and Schwartz's Theory of Basic Human Values. Unlike methods relying on self-verification, CVA decouples action generation from cognitive reasoning via a novel Value Verifier trained on authentic human data to explicitly model dynamic value activation. Experiments on CVABench, which comprises over 1.1 million real-world interaction traces, demonstrate that CVA significantly outperforms baselines. Our approach effectively mitigates polarization while offering superior behavioral fidelity and interpretability.

1 Introduction

Existing LLM-based agents often produce rigid, stereotyped behavior, while psychological prompting and self-referential evaluation can amplify model biases rather than reflect authentic human diversity. The CVA architecture addresses this gap by separating action generation from cognitive reasoning, modeling context-dependent value activation, and evaluating behavior against empirical human interactions.

  • Existing LLM-based agents frequently exhibit behavioral rigidity and stereotyping despite the need to capture human complexity, diversity, and stochasticity.
  • Psychological prompting can amplify latent model biases, producing caricatured outputs instead of heterogeneous and nuanced human behavior.
  • LLM-as-a-judge evaluations can mask these problems through a self-referential loop that endorses polarized or stereotypical behavior instead of comparing against empirical ground truth.
  • CVA models behavior as context-dependent value activation, using historical context to identify activated values that guide action selection.
  • CVA decouples action generation from cognitive reasoning and introduces a Value Verifier trained on authentic human behavioral data to assess alignment between actions and activated values.
  • CVABench uses over one million authentic interactions and empirical data from more than 15,000 human participants to diagnose behavioral rigidity and value polarization against real-world ground truth.

2 Proposed Approach

The proposed approach models behavior through context-dependent value activation and separates value alignment from action generation. It combines CVA’s verifier-guided architecture with CVABench’s empirical evaluation of individual fidelity and population-level diversity.

  • Psychological Foundations: CVA models behavior as conditional action generation P(A|C, V), where context activates a dynamic 10-dimensional value vector.Context aggregates situational and historical stimuli, while action is the final behavioral output.
  • Limitations of Existing Paradigms: Prompt-driven reasoning can distort authentic values into caricatures, increasing polarization and collapsing population variance as reasoning intensity rises.This motivates separating value alignment from prompt-driven reasoning.
  • CVA Architecture: CVA uses a Generate-then-Verify pipeline with Value-Action Mapping Calibration followed by Value-Driven Reasoning.Calibration reconstructs authentic value-behavior mappings through SFT on CVABench trajectories and DPO preference refinement.
  • CVA Architecture: The independent Value-Driven Verifier scores N context-conditioned candidate actions against target values and selects the highest-consistency action.The verifier is trained on authentic context-value-action triplets rather than relying on the generative LLM itself.
  • Evaluation and Interpretability: CVA combines valid psychological reasoning with decision transparency and evaluates population diversity alongside target-user behavioral consistency.GPV infers value profiles from generated behavioral logs for group-level alignment assessment.
  • CVABench Evaluation: CVABench aggregates over 1.1 million authentic interaction traces from 15,571 users across social-media reviews, conversational discourse, and spatio-temporal mobility.The benchmark evaluates individual behavioral consistency using accuracy or MSE and group-level value distributions using variance and polarization.

3 Experiments

Experiments on CVABench diagnose polarization and rigidity in prompt-driven agents, then evaluate CVA’s behavioral fidelity, interpretability, and sensitivity to inference depth. Across population-level and agent-level analyses, CVA mitigates these effects while additional reasoning eventually yields diminishing returns.

  • Experimental Setup: CVABench evaluates simulated population-level value distributions against empirical human distributions using standardized psychological assessments.The benchmark also supports domain-specific behavioral-fidelity metrics and comparisons across multiple agent baselines.
  • Psychological Bias: Increasing prompt-driven psychological inference shifts simulated means toward -1 or +1 and reduces variance, sharpening value distributions into rigid characterizations.The observed collapse is described as polarization and rigidity rather than increased fidelity to human diversity.
  • Main Results: CVA significantly enhances behavioral fidelity across domains and mitigates the behavioral rigidity and value polarization observed in group-level simulations.Existing baselines do not consistently improve simulation accuracy with ICL or explicit prompt-driven reasoning, and complex reasoning can be counterproductive.
  • Ablation Study: The full CVA architecture achieves the highest performance across all ablation metrics after incrementally adding SFT, DPO, and Verifier Reasoning.The ablation isolates contributions from the training and verifier components.
  • Inference Intensity: Inference depth initially improves Value-Guided Inference, but gains saturate beyond four reasoning rounds and additional steps provide diminishing returns.Figure 4 reports sentiment top 1/2/4 accuracy across four discrete inference intensities.
  • Interpretability: Value-Guided Verifier embeddings align closely with Schwartz’s theoretical circular structure, while cross-attention exposes words associated with context-dependent value activation.The analysis reports only minor deviations in the Achievement dimension and uses word-level contributions to interpret value preferences.

4 Related Works

Related work develops prompt-driven psychological agents and automated psychological measurement, but the paper argues that prompt-driven methods do not faithfully reproduce human behavioral heterogeneity. It positions value-based measurement and empirical evaluation as alternatives for studying polarization and behavioral fidelity.

  • Psychological Measurement: The paper uses automated natural-person value measurement through GPV to characterize cognitive structures in human-like agents.GPV is presented as non-reactive and less affected by reactivity bias than traditional questionnaire methods.
  • Evaluation: Prior work has reported opinion polarization when LLMs serve as agents, while this study adds quantitative measurements in behavioral simulation tasks.The paper frames its evaluation as an empirical contribution to that line of research.
  • Human-Like Agents: Human-like agent research spans virtual avatars, social simulacra, embodied VLA systems, and task-oriented assistants.Across these settings, faithfully capturing human behavioral complexity, diversity, and stochasticity remains a stated requirement.
  • Prompt-Driven Methods: Prompt-driven role-play methods are widely used to model psychological decision-making, but the paper argues they can polarize psychological metrics and produce extreme behaviors.The related-work discussion contrasts prompt-driven, training-free approaches with research exploring training-based alternatives.

5 Conclusion

The paper identifies polarization from prompt-driven psychological guidance as a source of unrealistic behavior and proposes CVA to address it. In CVABench evaluations, CVA reports stronger individual and group-level behavioral fidelity with interpretability advantages.

  • Conclusion: Prompt-driven methods guided by psychological traits can polarize psychological metrics and produce behaviors that diverge from realistic human decision-making logic.This motivates the paper’s value-driven alternative.
  • Conclusion: CVA mitigates polarization through base-model bias mitigation and a value-driven verifier.The architecture is evaluated within CVABench rather than through self-referential judgment alone.
  • Conclusion: CVA demonstrates superior fidelity in replicating individual behaviors and recovering group psychological indicators while offering interpretability advantages.The conclusion characterizes these results as evidence of efficacy and robustness within the benchmark.

6 Limitations

The paper’s evidence is bounded by CVABench’s scale and domain coverage, measurement challenges and possible GPV bias, limited baseline comparisons, and risks inherited from authentic internet data.

  • Scale and Domain Coverage: CVABench covers approximately 15,000 users across three primary domains, limiting current evidence about generalizability to broader contexts.The authors identify consumer behavior and cultural consumption as planned expansion areas.
  • Value Measurement and Bias: Value measurement remains challenging and potentially biased, although GPV is used for stability and construct-validity advantages.The authors explicitly acknowledge that GPV may still contain encoded biases.
  • Value Measurement and Bias: The verifier aligns value representations with heterogeneous behaviors while ground truth remains the main model’s ultimate supervision signal.This design avoids making the verifier the sole arbiter of quality, even when value measurement contains noise.
  • Baseline Comparisons: Large-scale simulation costs constrained the number of baseline comparisons.The authors plan evaluation-pipeline optimization to enable broader comparisons.
  • Ethical Considerations and Potential Risks: Training on authentic internet data creates an inherent risk of generating toxic or biased content present in the source distribution.Extreme hate speech is filtered, but some human imperfections are retained for behavioral fidelity.

A Lead-in Case Study

A GPT-4o role-play case study exposed behavioral rigidity: despite varying Hedonism while fixing Self-Direction high, the agent repeatedly selected the gym. The section then formalizes role-play and prompt-driven value reasoning as context-to-action pipelines using memory, candidate generation, and value-based selection.

  • Lead-in Case Study: Figure 7 reports near-absolute gym-choice probability for Hedonism ≤0.5 and a 0.93 bias at 0.6 with Self-Direction fixed at 0.9.The result illustrates behavioral rigidity under persona-based prompting.
  • Lead-in Case Study: Across 100 trials, GPT-4o frequently chose the gym even when Hedonism was set to 0.6, and this preference became nearly absolute at 0.5 or lower.The case varied Hedonism from 0.2 to 0.6 while fixing Self-Direction at 0.9.
  • Role-Playing Baseline: The role-playing baseline transforms working and retrieved long-term memory into actions through a mapping function R(Cw, Cl).Long-term memory is constructed heuristically from all available user memories, Cl = f(Cm).
  • CVABench Templates: CVABench supplies unified templates for conversation, mobility, and social-media simulations, while identical prompts support fair comparison across agents.The RolePlay baseline does not use the value-preference information during reasoning, unlike other baselines including CVAgent.
  • Prompt-Driven Value Reasoning: Prompt-driven value reasoning generates an initial action, iteratively adds candidate actions, evaluates value alignment, and selects the highest-scoring action.Its inputs include context, global memory, value preference, persona, and reasoning rounds.

B.3 Training Required Agents

The required-agent baseline uses a two-stage training pipeline: supervised fine-tuning on human behavior followed by direct preference optimization. Candidate preferences are constructed by comparing sampled actions with ground-truth responses.

  • Training Pipeline: The baseline is trained with Supervised Fine-Tuning followed by Direct Preference Optimization, with both stages running for one epoch.The pipeline uses the CVA Bench training dataset.
  • Supervised Fine-Tuning: SFT maps role-playing prompts x to ground-truth human behaviors ygt using standard cross-entropy loss.The prompts provide the context and the human behavior is the target response.
  • Direct Preference Optimization: DPO samples K = 10 candidate actions from the SFT model at temperature 0.8 and selects preferred and rejected samples using linguistic similarity to ground truth.The preference dataset consists of (x, yw, yl) triples.
  • Direct Preference Optimization: The DPO objective combines sigmoid DPO loss, BCO pair loss, and SFT regularization as Ltotal = 1.0 · LDPO + 0.2 · LBCO + 1.2 · LSFT.The combined objective preserves the stated weighting of all three terms.

C Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details

The group-simulation experiments show that stronger prompt-driven value reasoning narrows output variance and pushes value preferences toward extremes. These changes increase polarization and reduce fidelity to diverse human population distributions.

  • Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details: The baseline analysis finds that relying solely on instruction-following and in-context learning is inadequate for individual-level precision and group-level psychological indicators.The analysis covers all ten value dimensions, with additional data supplied beyond the representative main-text dimension.
  • Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details: Increasing reasoning iterations substantially decreases alignment-score variance, making responses more predictable and less diverse.The iterative procedure progressively narrows the range of agent outputs.
  • Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details: Alignment scores move toward −1 and 1 as reasoning increases, replacing neutral positions with increasingly binary value stances.A score of 1 denotes strong support and −1 strong opposition.
  • Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details: Table 5 shows monotonic standard-deviation decreases and mean preferences shifting toward extreme bounds relative to ground truth.Figure 8 visualizes the distributions changing from diverse spreads into narrow, rigid modes.
  • Psychological Bias of Group Simulation in Prompt-Driven Humanoid Agents Experiment Details: Repeated explicit reasoning pushes agents toward simplified and exaggerated value positions rather than the subtle, moderate alignments found in real human behavior.The passage identifies this as a critical failure of prompt-driven value reasoning.

D.1 Metrics Explanation

The evaluation framework measures predictive accuracy, linguistic distributional fidelity, mobility timing, and population-level value diversity across simulation domains. Results indicate that CVA agents frequently achieve smaller Wasserstein distances to human ground-truth distributions than prompt-driven agents.

  • Social Media Review: CVABench evaluates social-media ratings and sentiment with Accuracy and compares generated and real lexical-richness distributions using Type-Token Ratio and Wasserstein Distance.The Wasserstein formulation compares joint distributions whose marginals are the generated and real distributions.
  • Population Diversity: Table 5 reports standard deviation relative to ground truth across ten value dimensions, split into Self-Direction–Power and Security–Universalism panels.The table analyzes varying value-inference strengths from value_infer_0 to 8.
  • Linguistic Fidelity: Table 6 evaluates review and conversation outputs with Wasserstein Distance, where lower values indicate better alignment with human linguistic patterns.The seven metrics cover sentence structure, TTR, and noun, adverb, adjective, and verb frequencies.
  • Conversation Discourse: Conversation Discourse uses Attitude Accuracy for user stances and Wasserstein Distance for differences between generated and authentic comment TTR distributions.The task evaluates stance replication across subreddits.
  • Spatio-Temporal Mobility: Spatio-Temporal Mobility predicts the next POI category and stay duration using Category Accuracy for location and MSE for temporal precision.Predicted and actual stay durations are measured in minutes.
  • Population Diversity: Value Distribution Variance measures the relative deviation between simulated and empirical human variances across three domains.A value closer to 0 indicates superior alignment with natural population diversity.
  • Population Diversity: Variance above ground truth indicates excessive simulated variability, whereas variance collapse corresponds to polarization and behavioral rigidity.The latter reflects convergence toward a narrow, stereotypical value range.
  • Linguistic Fidelity: CVA agents frequently produce smaller Wasserstein distances to ground-truth distributions, while more prompt-driven reasoning rounds do not necessarily improve authenticity.The finding is reported across the detailed linguistic evaluations.

E.2 Quantitative Analysis of the Circular Embedding Structure

The analysis evaluates whether the Value-Guided Verifier recovers Schwartz’s circular value structure and identifies where that structure remains imperfect. The verifier achieves substantial structural alignment, while specific value entanglements and excluded dimensions limit the analysis.

  • Metric: The Circular Inversion Score (CIS) measures the proportion of preserved pairwise relationships after optimally rotating the observed value sequence against Schwartz’s theoretical order.The metric projects value embeddings into two dimensions, sorts them by angle, and minimizes rotational inversion distance before normalization.
  • Metric: 1.00 represents a perfect fit for the CIS metric, which evaluates alignment with the theoretical circular value structure.
  • Results: 0.75: the trained verifier recovered the latent circular structure of human values with a Circumplex Index of Structure score of 0.75.
  • Scope Boundary: Power and Security were excluded because their generated measurements showed noun probabilities above 10%, indicating unstable representations relative to other values.
  • Structural Strengths: The verifier largely preserved adjacency within the Self-Transcendence and Conservation sectors, distinguishing social-focus values from stability-focus values.Universalism and Benevolence formed the Self-Transcendence adjacency, while Tradition and Conformity formed the Conservation adjacency.
  • Deviations and Causes: Achievement shifted toward Self-Direction, while Hedonism became adjacent to Tradition, indicating semantic entanglements in the learned value topology.The passage attributes these deviations possibly to conflation between individual success and intellectual autonomy, and to data sparsity in opposing pairings.

F.2 Verifier Architecture and Training

The Value-Guided Verifier scores candidate actions for consistency with context and value profiles using separate encoders and cross-attention. It is trained with ranked candidate actions to prefer behaviors closer to human ground truth.

  • Architecture: The Value-Guided Verifier explicitly models human preference selection for candidate actions.
  • Architecture: A multiencoder design separately embeds the action, context, and value profile, then uses context-guided cross-attention to refine the value representation.
  • Architecture: The concatenated refined-value and action representation is fed into an MLP that predicts a value consistency score s(A, C, V).
  • Training Objective: For each context, five candidate actions are sampled from the DPO-tuned model at temperature 0.8 and ranked by linguistic similarity to ground-truth behavior.
  • Training Objective: Pairwise ranking loss trains the verifier to maximize the score margin between preferred and rejected actions.The training tuples contain context, value profile, preferred action, and rejected action; the sigmoid function is used in the objective.
  • Training Objective: The verifier assigns higher consistency scores to actions that better align with human ground truth under the given value profile.
Loading 2604.05939v1…