Source-linked AI summary
Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans
Akshay K. Jagadish, Younes Strittmatter, Nori Jacoby, Eric Schulz, Nathaniel Daw, Thomas L. Griffiths, Suyog H. Chandramouli
TL;DR
The paper studies theory discovery using Centaur simulations in a multi-attribute decision-making task near, but outside, Centaur’s training distribution. AutoCog represents executable theories and evaluates them on simulated experiments and held-out human experiments, enabling like-for-like comparison with theories found on people.
Problem
Centaur was trained on binary-cue multi-attribute decisions, whereas this study uses a cardinal-valued variant near but outside that training distribution.
Method
AutoCog represents each theory with a verbal mechanism, executable prediction function, parameters, and response policy, then evaluates theories across simulated and held-out human experiments.
Results
The evaluation covers fifty Centaur experiments and ten held-out human experiments, using the same pipeline for comparisons with human-loop theories.
Takeaways & Limitations
Centaur simulations remained useful for discovery in this setting, which has no feedback or revealed correct answer and uses independent trials.
Abstract
from arXiv · showhide
Behavioral foundation models have been proposed as stand-ins for human participants across settings, but it is unclear whether theories discovered on them generalize to humans or merely characterize the simulator. We ran the Automated Cognitive Scientist (\textsc{AutoCog}), a closed-loop discovery system in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between competing theories, and synthesize successors, entirely on behavior simulated by Centaur, a foundation model of human behavior. In a multi-attribute decision-making setting, the theories \textsc{AutoCog} found on Centaur generalized to human data: they outperformed canonical theories on ten held-out experiments and were rivaled only by theories found by running the same loop on people. We argue that this succeeds despite the simulator's inevitable imperfections because a discovery loop that arbitrates between competing theories demands less of its simulator than estimation does. The simulator only needs to capture the regularities that distinguish the theories, and not necessarily reproduce behavior precisely. Imperfect simulators can therefore widen the search over theories, with human data then testing whether the surfaced theories generalize.
Distance from Centaur’s training distribution
The studied cardinal-valued decision task was near, but not within, Centaur’s training distribution. Several task properties plausibly helped its simulations remain useful for theory discovery.
- Distance from Centaur’s training distribution: Centaur was trained on binary-cue multi-attribute decisions, whereas these experiments used a cardinal-valued variant.The task therefore lay near, though not within, Centaur’s training distribution.
- Distance from Centaur’s training distribution: The cardinal-valued task directly extended paradigms used to fine-tune Centaur.It extended the binary-valued multi-attribute decision task developed by Hilbig and Moshagen.
- Distance from Centaur’s training distribution: No feedback or correct answer was provided, preventing an accuracy shortcut.Trials were also independent, so sequential dependencies on which Centaur diverges from people did not arise.
- Distance from Centaur’s training distribution: The authors expect distance from the training distribution to matter in degree and recommend human participants when that distance becomes consequential.They identify training-distribution distance as one condition for switching from simulated to human data.
Theory representation
AUTOCOG represents each theory as a verbal mechanism paired with executable code conforming to a fixed interface. The interface standardizes parameters, prediction, and response generation across theories.
- Theory representation: Each theory combines a verbal mechanism statement with an executable model satisfying a fixed interface.This shared representation lets AUTOCOG compare theories through a common computational protocol.
- Theory representation: The interface requires a Parameters block, a predict function returning two-option choice probabilities, and a policy function mapping probabilities to responses.Cue validities are supplied by the experiment rather than declared as free parameters.
- Theory representation: Every theory uses a softmax choice rule with inverse temperature beta and lapse rate epsilon.These shared parameters therefore appear in every theory block.
- Theory representation: AUTOCOG generates the verbal description, predict-function body, and parameter specifications within the manually defined interface.
Run details
The study used Centaur to generate responses for designed experiments and replayed theories on both simulated and held-out human experiments. A fixed evaluation pipeline produced like-for-like comparisons.
- Run details: Centaur supplied 25 synthetic participants for each designed experiment, with about 96 trials per participant.Each trial consisted of a single sampled next token from a prompt formatted like Centaur’s training data.
- Run details: Evaluation replayed each theory across 200 simulated participants with parameters drawn once from declared ranges.Choice probabilities were computed for each option pair using random seed 0.
- Run details: Centaur inference ran on one NVIDIA H200 GPU and the full run took roughly 24 GPU-hours.The runtime was estimated rather than logged.
Seed heuristics
The seed heuristics differ in how they use feature evidence: Take-The-Best follows validity order and stops at one discriminating cue, whereas Tallying counts wins across all features.
- Take-The-Best: Take-The-Best consults cues in descending validity order and stops at the first cue that discriminates between options.The discriminating cue alone determines the choice; if none discriminates, the model guesses.
- Take-The-Best: Take-The-Best ignores lower-validity cues after a higher-validity cue discriminates, as well as the magnitude of the winning difference.Its response noise uses beta and epsilon through a softmax and lapse mixture.
- Tallying: Tallying counts strict feature-wise wins across all cues, with ties contributing to neither option.Every cue contributes equally, and equal totals produce a uniform choice before lapse mixing.
- Tallying: Tallying discards cardinal magnitudes and is invariant to monotone rescaling of individual features.Unlike Take-The-Best, it does not privilege any feature or allow one large difference to dominate through magnitude.
- Shared response model: Both heuristics use beta in [0.1, 20.0] and epsilon in [0.0, 0.5].
Discovered theories
AutoCog surfaced two theories that combine contextual cue processing with probabilistic or compensatory decision mechanisms. CRAN normalizes cue differences within the current context, while RCSL orders cues by validity and allows soft stopping followed by fallback integration.
- CRAN: CRAN normalizes each feature difference by the largest absolute difference in the current stimulus context.A nonlinear contrast parameter suppresses smaller relative differences when another cue has a massive advantage, while scaling preserves absolute magnitude.
- CRAN: CRAN combines validity-based weighting with a nonlinear transformation controlled by tau.Validity weights are shrunk toward equal weighting before the tau transformation and renormalization.
- Choice generation: Both theories convert accumulated option scores into probabilistic choices using a softmax with a lapse-rate mixture.The lapse rate mixes the softmax probabilities with a uniform distribution over options.
- RCSL: RCSL evaluates cues in descending validity order and assigns each cue a probability of terminating the search.The stopping probability depends on relative cue differences, alpha, theta, and lambda_stop.
- RCSL: When no cue stops the search, RCSL adds a compensatory fallback based on validity-weighted relative cue differences.Expected evidence from stopping is combined with fallback evidence before option scores are converted into choice probabilities.