PaperlayerView exact revision on arXiv

Source-linked AI summary

Generating Computational Cognitive Models using Large Language Models

Milena Rmus, Akshay K. Jagadish, Marvin Mathony, Tobias Ludwig, Eric Schulz

arXiv:2502.00879v3cs.LG

TL;DR

Handcrafted cognitive models take domain knowledge, coding expertise, and time, and researchers’ theoretical commitments may narrow the models they consider. GeCCo uses open-source LLMs to generate and refine models across four cognitive domains. The generated models matched or surpassed domain-specific literature models, while remaining interpretable.

  • Problem

    Handcrafting cognitive models requires substantial expertise and time, and researchers’ theoretical commitments may narrow the models considered.

    Source: §1 · p. 2

  • Method

    GeCCo prompts open-source LLMs to generate cognitive models as Python functions and iteratively refines them using predictive-performance feedback.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across four cognitive domains, LLM-generated models matched or surpassed domain-specific literature models and captured as much explainable variance as CENTAUR.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings suggest LLMs can advance cognitive modeling by democratizing access to complex model discovery and accelerating research.

    Source: §8.2 · p. 10

    The evaluated domains do not span all cognitive modeling, and higher-dimensional data or many-parameter models may be challenging; evaluation on naturalistic datasets remains needed.

    Source: §8.2 · p. 10

Abstract

from arXiv · show

Computational cognitive models, which formalize theories of cognition, enable researchers to quantify cognitive processes and arbitrate between competing theories by fitting models to behavioral data. Traditionally, these models are handcrafted, which requires significant domain knowledge, coding expertise, and time investment. However, recent advances in machine learning offer solutions to these challenges. In particular, Large Language Models (LLMs) have demonstrated remarkable capabilities for in-context pattern recognition, leveraging knowledge from diverse domains to solve complex problems, and generating executable code that can be used to facilitate the generation of cognitive models. Building on this potential, we introduce a pipeline for Guided generation of Computational Cognitive Models (GeCCo). Given task instructions, participant data, and a template function, GeCCo prompts an LLM to propose candidate models, fits proposals to held-out data, and iteratively refines them based on feedback constructed from their predictive performance. We benchmark this approach across four different cognitive domains -- decision making, learning, planning, and memory -- using three open-source LLMs, spanning different model sizes, capacities, and families. On four human behavioral data sets, the LLM generated models that consistently matched or outperformed the best domain-specific models from the cognitive science literature. Taken together, our results suggest that LLMs can generate cognitive models with conceptually plausible theories that rival -- or even surpass -- the best models from the literature across diverse task domains.

1 Introduction

Handcrafted cognitive models can be costly to develop, and researchers’ theoretical commitments may narrow the models they consider. GeCCo uses LLMs to generate and iteratively refine cognitive models, which matched or surpassed literature models across four domains.

1 Introduction

  • Handcrafting models requires literature review, theory formulation, and implementation, while researchers’ theoretical commitments may narrow the model space and cause better explanations to be missed.The cited risk is that potentially better explanations of the data may be overlooked.Source: §1 · p. 2
  • LLMs can process behavioral data and task descriptions, use in-context domain knowledge to generate hypotheses, and synthesize executable programs for model generation.These abilities have also been used to generate and evaluate probabilistic programs for statistical modeling.Source: §1 · p. 2
  • GeCCo prompts an LLM with task instructions, participant data, format guardrails, and a code template, then uses held-out-data BIC feedback to refine candidate models over 10 iterations.Three candidates are fitted per iteration; the best model across iterations is evaluated against a baseline and with posterior predictive checks.Source: Figure 1 · p. 2, §1 · p. 2
  • Across decision making, learning, planning, and memory, LLM-generated models matched or surpassed the best domain-specific literature models.The study also reports that they captured as much explainable variance as CENTAUR while remaining interpretable.Source: §1 · p. 3

2 Methods

GeCCo generates candidate models in repeated LLM sampling rounds, fits them on held-out data, and evaluates the best candidate against competing models. The evaluation spans four paradigms and uses BIC and exceedance probability to assess model performance.

2 Methods

  • Each of 10 iterations generates three candidate models; candidates are fitted on held-out validation data, and the best model is evaluated on a separate test set.The prompt includes task instructions, participant data, guardrails, a domain-specific template, and feedback from iteration 2 onward; 20 random starts are used for fitting.Source: §2 · p. 3
  • Three open-source LLMs were tested in-context: Llama-3.1-Instruct-70B, DeepSeek-R1-Distill-Llama-3.1-70B, and Qwen2.5-72B-Instruct.They span model families and reasoning capabilities; a task domain takes at most eight hours on four Nvidia A100 GPUs.Source: §2 · p. 3
  • GeCCo is evaluated on decision-making, learning, planning, and memory paradigms, selected to span increasing complexity in parameters and formalized cognitive processes.The paradigms use noisy human behavioral data, where no single model fully captures the underlying processes.Source: §2 · p. 3
  • BIC balances model fit and complexity by penalizing parameter count, while exceedance probability estimates the posterior probability that a model explains the data better than competitors.The pipeline can use other quality measures, including AIC.Source: §2 · p. 4

3 Experiment 1: Decision making

In a multi-feature choice task, LLM-generated models outperformed the best literature model, and the best candidate captured human choices while using one parameter to arbitrate among canonical heuristics. Its discount factor also relates to continuum-of-models accounts without requiring a Bayesian framework.

3 Experiment 1: Decision making

  • The task asked participants to choose between two options defined by four feature values and their validities.The paradigm was drawn from Hilbig and Moshagen (2014).Source: §3 · p. 4
  • 39.40 ± 4.54 vs. 51.97 ± 4.52 BICM±SEM: Llama outperformed pWADD, the best literature model, with t(52) = 41.8 and p < .001.R1 also produced a model that outperformed the best literature model; Figure 2B presents the model-fit comparison.Source: Figure 2 · p. 4, §3 · p. 4
  • The best LLM model weights features by their validities and uses one discount-factor parameter to arbitrate among Take the Best, Weighted Additive, and Equal Weighting strategies.pWADD can also arbitrate among these strategies, but uses four parameters; simulated choices from the LLM model aligned closely with human choices.Source: §3 · p. 4, Figure 2 · p. 4
  • The model’s discount factor parallels prior-strength parameters in continuum-of-models accounts, while the LLM-generated model achieves this flexibility without a Bayesian framework.The account interprets heuristics as Bayesian inference under extreme priors.Source: §3 · p. 5

4 Experiment 2: Learning

In a two-armed bandit task, the best LLM-generated model performed slightly better than RW 4α and matched its predictive checks. The model separates chosen and forgone outcomes and adds forgetting of unchosen values.

4 Experiment 2: Learning

  • The Chambon et al. (2020) task is a two-armed bandit with partial-feedback and full-feedback conditions, in which rewards may be observed for chosen and unchosen options.The analyzed task also varied reward probabilities across blocks.Source: §4 · p. 5
  • The literature baseline, RW 4α, uses four learning rates for positive and negative prediction errors on chosen and unchosen actions, alongside a softmax choice policy and perseveration parameter.It updates action values with a delta rule.Source: §4 · p. 5
  • 77.97 ± 5.40 vs. 85.24 ± 7.95 BICM±SEM: Llama’s model performed slightly better than RW 4α; its EXP was 0.97 versus 0.03 for the literature model.Posterior predictive checks found that the LLM model matched RW 4α in both high- and low-reward blocks.Source: Figure 3 · p. 5, §4 · p. 5
  • The LLM model maintains separate value sets for chosen and forgone outcomes; forgetting unchosen values creates temporal asymmetry and removes the need for an explicit choice-stickiness parameter.The forgone-outcome values form a dedicated fictive trace, raising questions about how experienced and counterfactual values are weighted.Source: §4 · pp. 5–6

5 Experiment 3: Planning

In the two-stage planning task, R1’s generated model achieved a lower mean BIC than the Hybrid baseline, and its behavior was consistent with a more model-based strategy.

Task

  • The task tests whether people use reward-driven model-free learning or estimated state-transition structure for planning in a two-stage task.The dataset replicated the original Daw et al. (2011) study.Source: §5 · p. 6

Baseline cognitive model

  • The Hybrid baseline combines SARSA(λ) model-free action values with model-based values updated from transition probabilities, weighted by a free parameter.Source: §5 · p. 6

Results

  • 432.47 ± 19.31 versus 437.38 ± 18.72 mean BIC favored R1 over Hybrid, but the difference was not significant; EXP favored R1, 0.85 versus 0.15.Posterior predictive checks showed action selections consistent with a more model-based strategy, as in humans.Source: Figure 4 · p. 6, §5 · p. 6
  • R1’s model learns transition probabilities and uses separate learning rates for common and rare transitions, an inverse-temperature parameter, and value iteration, while omitting reward discounting.Value iteration updates first-stage action values using the maximum over future-value estimates.Source: §5 · p. 6

Scientific Insight

  • GeCCo’s model offers a parsimonious planning account in which learned transition structures and value updating produce behavior without explicitly separating model-based and model-free systems.The authors present it as an alternative to frameworks relying on multiple canonical components, aligned with calls to move beyond the strict MB/MF dichotomy.Source: §5 · p. 7

6 Experiment 4: Working memory

In the reinforcement learning–working memory task, Llama’s generated model outperformed the RL-WM baseline and captured different learning patterns under low and high cognitive load.

Task

  • Participants learned stimulus–action associations by feedback, with set sizes of three (low load) and six (high load) increasing cognitive load and the delay between stimulus repetitions.Source: §6 · p. 7

Baseline cognitive model

  • The RL-WM baseline combines incremental, capacity-unlimited RL with one-shot WM learning that has perfect previous-trial retention but is capacity-limited and subject to decay.Source: §6 · p. 7

Results

  • 267.25 ± 9.31 versus 275.2 ± 9.21 BICM±SEM favored Llama over RL-WM; EXP also favored Llama, 0.99 versus 0.01.Posterior predictive checks showed close alignment with human behavior.Source: Figure 5 · p. 7, §6 · p. 7
  • Llama’s model placed cognitive-load-dependent learning differences in the RL module rather than the original RL-WM model’s cognitive-load-dependent WM weight.Source: §6 · p. 7

Scientific Insight

  • The model captured faster, asymptotic learning at low cognitive load and incremental, RL-like learning at high cognitive load, patterns typically associated with WM and RL, respectively.The authors note this raises questions about where cognitive-load effects arise and challenges the assumption that RL processes are relatively insensitive to load.Source: §6 · p. 7

7 Control experiments

Control experiments found that generated models recovered synthetic ground truths, matched or exceeded CENTAUR across domains, and varied with prompt components and base model.

Contribution of different model features

  • Llama outperformed Qwen (βM±SEM = 5.624 ± 1.029, p = 0.03), while reasoning ability’s apparent advantage was not significant (βM±SEM = -0.266 ± 1.029, p = 0.07).The authors suggest training data and/or training protocols may explain the Llama–Qwen difference despite Qwen’s larger parameter count.Source: §7 · p. 8

Causal role of different prompt components

  • Feedback was the strongest predictor of post-ablation BIC (βM±SEM = -17.040 ± 2.287), followed by participant data (-12.836 ± 2.287), task description (-9.834 ± 2.287), and code template (-9.359 ± 2.287).Source: §7 · p. 8

Data contamination analysis

  • No contamination evidence appeared: acceleration terms were 0.0171 (decision), 0.1147/0.1409 (full/partial learning), 0.1718 (planning), and 0.6481 (working memory).All values were below the recommended threshold of 1.Source: §7 · p. 8

Recovering ground truth models in synthetic datasets

  • On synthetic decision and learning data, generated models closely matched ground truth, recovering Take the Best and Tallying heuristics and Rescorla-Wagner variants with two learning rates or stickiness.Source: §7 · p. 8

Explainable variance captured in human behavior

  • LLM-generated models matched or exceeded CENTAUR’s held-out negative log-likelihood across four domains, suggesting they captured most explainable variance while remaining interpretable.CENTAUR serves as a proxy for maximum explainable variance; Table 1 reports mean negative log-likelihoods and paired t-tests across the domains.Source: Table 1 · p. 8, §7 · p. 8

8 Discussion

GeCCo uses open-source LLMs to generate Python cognitive models and refine them with feedback; across four domains, its models matched or outperformed domain-specific baselines. The discussion situates this approach among automated model-discovery and code-generation research while noting limitations in domain coverage and evaluation settings.

8 Discussion

  • Across four cognitive domains, GeCCo’s LLM-generated models matched or outperformed domain-specific baselines, and control experiments found feedback causally most important among prompt components.The discussion also reports recovery of ground-truth models from simulated data and interpretable models with greater predictive power than CENTAUR.Source: §8 · p. 9

8.1 Related work

  • Prior model-discovery efforts often searched predefined model spaces, while GeCCo offers a lightweight, open-source, in-context approach reported to converge in hours rather than days.Related work also describes LLMs generating executable code and mathematical or probabilistic programs.Source: §8.1 · p. 9

8.2 Discussion and Limitations

  • GeCCo combines in-context LLM proposals with offline optimization, avoids fine-tuning, and is presented as a way to explore broader model spaces more efficiently.Its prompt includes paradigm descriptions, text-formatted data, Python function specifications, a code template, and evaluation feedback on later iterations.Source: §8.2 · p. 9, Figure 6 · p. 14
  • A domain-specific model template could bias proposals, although removing it had the smallest impact on downstream performance and an LLM-generated template fit the data just as well.Current feedback uses offline fit metrics such as BIC and could be supplemented with LLM feedback on theoretical plausibility, parsimony, alternatives, or code quality.Source: §8.2 · p. 9
  • The evaluated domains do not span all cognitive modeling; language processing, perception, higher-dimensional data, many-parameter models, and naturalistic datasets remain future evaluation directions.The authors note that higher-dimensional data and models with many parameters may be challenging.Source: §8.2 · p. 10

B Prompts

The appendix prompts specify task data, model-generation instructions, and Python-function requirements across decision making, learning, planning, and working memory. A synthetic decision-making setup additionally tests model identification and generation under varying noise and prompting strategies.

B.1 Decision making

  • The decision-making prompt describes a four-expert product-choice task and requests distinct executable models returning negative log likelihood, with parameter roles and bounds specified.The four expert validities are 0.9, 0.8, 0.7, and 0.6; the function template receives choices, option features, validities, and parameters.Source: §B.1 · p. 15

B.2 Learning

  • The learning prompt uses a two-action bandit with chosen and unchosen rewards, whose reward distributions remain fixed within blocks but may change between blocks.It requests distinct executable models that use their parameters and return the negative log likelihood of observed actions.Source: §B.2 · p. 16

B.3 Planning

  • The planning task has two stages: participants choose a carpet, reach a mountain, then choose a genie for a probabilistic chance of gold.Carpets usually lead to particular mountains, but occasional wind-driven transitions make transition knowledge potentially relevant to later choices.Source: §B.3 · p. 17

B.4 Working memory

  • The working-memory task varies set size between three and six state-action pairs, resets states each block, and provides binary feedback.Its stated objective is to understand learning differences across the two cognitive-load conditions.Source: §B.4 · p. 18

C Decision making: Sanity checks using synthetic data

  • Synthetic decision tasks contrast Take the Best, which prioritizes one feature, with Tallying, which counts feature wins across three features.The 80 problems balance which option is superior, use unambiguous cases for identifiability, and include noise levels 0, 0.25, and 0.5.Source: C Decision making: Sanity checks using… · pp. 19–20
  • The synthetic-data prompts test model identification and generation using vanilla, Step-Back, and Chain of Thought strategies.For identification, candidate Python code accompanies simulated choices; generation prompts specify the desired function structure.Source: C Decision making: Sanity checks using… · p. 20

C.1 Results

On synthetic decision data, the LLM identified the source heuristic perfectly without noise and remained above chance at noise 0.25, while reaching chance at noise 0.5. Generated models matched the ground-truth choices perfectly on evaluation tasks, with noisier data yielding more varied proposals.

C.1 Results

  • At noise 0, source-model identification was perfect; at noise 0.25, accuracy was 0.86 (SEM = 0.02) for Take the Best and 0.71 (SEM = 0.06) for Tallying.At noise 0.5, both heuristics’ predictions were at chance level; the caption describes this as the point where the heuristics are indistinguishable.Source: §C.1 · p. 20, Figure 7 · p. 21
  • LLM-generated models matched ground-truth heuristic choices with perfect accuracy on evaluation tasks, although Tallying code summed feature values rather than counting superior features.The sum-based alternative can diverge from true Tallying when feature values are not normalized or in the same range.Source: §C.1 · p. 20
  • With noisy data, generated heuristics deviated further from the source rules: Take the Best retained its second-feature priority with a criterion, while Tallying proposals varied.Reported noisy Tallying strategies included choosing by the highest overall or minimum feature value.Source: §C.1 · pp. 20–21

D Learning: Sanity checks using synthetic data

The synthetic-learning sanity check uses two-armed bandit data to test model recovery and model identification. The LLM-generated identification function searches model parameters, while generated models are checked against simulated behavior.

Learning models

  • The tested learning candidates are vanilla RW, RW with separate positive- and negative-feedback learning rates, and RW with stickiness.RW updates the selected action value from reward history; its softmax policy transforms action values into choice probabilities.Source: D Learning: Sanity checks using… · pp. 21–22

Synthetic data generation

  • The benchmark simulates 100 agents per learning model, each completing 150 trials of a two-armed bandit task with plausible parameter values.The task has two options with fixed reward probabilities and binary rewards.Source: D Learning: Sanity checks using… · pp. 21–22

Evaluation of generated models

  • Figure 8 shows successful differentiation of the two learning models, and the generated model based on RW + κ fits better than random guessing and vanilla RW.The simulation also captures the underlying tendency to choose more rewarding actions.Source: Figure 8 · p. 22

LLM Prompting and model identification

  • The LLM-generated ModelIdentification function uses differential evolution to select the source model with the smaller negative log-likelihood.It takes predefined arguments such as action and reward lists and is manually evaluated for identification accuracy.Source: D Learning: Sanity checks using… · p. 22

Proposed model parameters

  • Table 2 gives examples of parameters in LLM-proposed models across sampling runs and notes that their mechanisms are documented in prior research.Source: Table 2 · p. 23

D.1 Results

Across synthetic learning data, GeCCo recovered the two-learning-rate RW model but not the RW + κ ground truth. For RW + κ, it proposed a different model that fit better than two comparison models and reproduced the rewarding-choice rate closely.

Model generation

  • GeCCo ran 10 sampling iterations, generating three models per iteration and using automatically constructed fit feedback in subsequent prompts.The comparison used the best generated models across runs.Source: §D.1 · p. 23
  • BIC 78.40 (SEM = 1.3) for the LLM-generated RW + α± model versus 78.78 (SEM = 5.3) for ground truth; the LLM recovered RW + α±.Source: §D.1 · p. 23
  • For RW + κ data, the LLM did not recover ground truth; its value-decay model fit better than random guessing and vanilla RW and approximated the rewarding-choice rate.The ground-truth and generated-model proportions choosing the more rewarding action were 0.69 (SD = 0.13) and 0.68 (SD = 0.13), respectively.Source: §D.1 · p. 23
  • Across other sampling runs, the proposed parameters were described as reasonable and among those commonly cited in cognitive modeling literature.Source: §D.1 · p. 23

E Human experiments

The human-experiment materials span decision making, learning, planning, and working memory, alongside descriptions of the tasks and candidate model mechanisms. The supplied passages also report prompt ablations and a contamination check.

E.1 Decision making

  • The decision-making task involved 79 participants choosing between two options with four binary features over 96 trials.Figure 9 reports that all prompt components contributed to generated-model performance, with feedback removal having the largest effect; the contamination analysis found no evidence for contamination under its stated threshold.Source: §E.1 · pp. 23, 25, Figure 9 · p. 24, Figure 10 · p. 24

E.2 Learning

  • The learning experiment used a two-armed bandit with partial- and full-feedback conditions, with the study focusing on counterfactual feedback.Participants completed 16 blocks, with changing reward probabilities and binary outcomes.Source: §E.2 · p. 25

E.3 Planning

  • The magic-carpet two-step task uses probabilistic first-stage transitions and deterministic second-stage choices to distinguish model-free from model-based strategies.Common and rare transitions occur with probabilities of 70% and 30%; a model-based planner uses transition structure when choosing whether to switch after a rare transition.Source: §E.3 · pp. 25–26

E.4 Working memory

  • The working-memory task varies stimulus-action set size between three and six, with each stimulus repeated nine times per block and deterministic feedback.Source: §E.4 · p. 26

F.1 Decision making: Take The Best (TTB), Equal weighting (EQW), Weighted sadditive hueristic (WADD)

  • The decision-model candidates include TTB, EQW, WADD, and pWADD, which differ in feature use and weighting.TTB relies on a single prioritized feature; EQW sums feature comparisons, WADD weights them by validity, and pWADD uses inferred weights probabilistically.Source: §F.1 · p. 26

F.2 Learning: Rescorla Wagner model with 4 learning rates, softmax policy and perseveration

  • The Chambon winning model uses separate learning rates for positive and negative prediction errors on chosen and unchosen actions.Its policy also includes exploration and perseveration parameters.Source: §F.2 · pp. 26–27

F.3 Planning: The Model-based/Model-free Hybrid model

  • The planning model combines model-free and model-based action values, with a weighting parameter controlling the relative influence of model-based planning.Its first-stage values incorporate reward prediction errors from both stages, while the model-based values use second-stage action values.Source: §F.3 · p. 27

F.4 Working memory: The Reinforcement Learning - Working Memory model

  • The working-memory model combines RL and WM policies, with reliance on WM varying by stimulus-response set size.The RL module follows RW learning; the WM module stores previous-trial information and includes association and forgetting mechanisms.Source: §F.4 · p. 28

G Partial feedback (learning) experiment

This section examines GeCCo’s models for Chambon et al.’s partial-feedback learning task, including how the best model represented learning from positive and negative feedback. It also describes the previously collected data and in-context evaluation setup, as well as the LLMs and prompting.

G Partial feedback (learning) experiment

H Posterior predictive checks

  • Posterior predictive checks simulated participant behavior from best-fitting parameters and compared it with human data; learning checks examined reward conditions and early versus late trials.For learning, simulations used participant-specific parameters and high- or low-reward block contingencies.Source: H Posterior predictive checks · p. 29

I Using AIC for the best model selection in the GeCCo pipeline

  • AIC-based model selection closely replicated BIC-based results in decision making and planning.Figure 13 compares the best GeCCo-generated models across the two feedback variants.Source: Figure 13 · p. 30

J Using LLM-generated template in the prompt

  • Using an LLM-generated rather than hand-crafted template produced comparable performance in decision making and learning.Figure 14 compares the best GeCCo-generated models under the two template approaches.Source: Figure 14 · p. 30

NeurIPS Paper Checklist

  • The checklist states that the study tested four cognitive domains and notes limitations including limited domain coverage, prompt sensitivity, and restricted exploration of feedback options.The checklist also says prompts, model and fitting details, dataset examples, and data and code availability are provided for reproducibility.Source: NeurIPS Paper Checklist · pp. 31–33
  • The authors reused data collected by other researchers and conducted all experiments in-context, without model training.They were not involved in data collection and described participant tasks as fully as the original papers did.Source: NeurIPS Paper Checklist · p. 37
  • The authors disclosed the LLMs’ family, size, and prompting because LLMs were central to the experiments.Source: NeurIPS Paper Checklist · p. 38
Loading 2502.00879v3…