Source-linked AI summary
Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence
Mengru Wang, Junfeng Fang, Shuofei Qiao, Zhenqian Xu, Haoming Xu, Haoxiong Wang, Shumin Deng, Linyi Yang, Zhixiang Cui, Xin Xu, Yunzhi Yao, Buqiang Xu, Fei Shen, Haozhe Luo, Yunxiang Wei, Ningyu Zhang, Julian McAuley, Tat Seng Chua, Huajun Chen
TL;DR
AI models’ capabilities and risks are difficult to understand because mechanistic exploration remains hard to scale and existing automation rarely develops general theories. Mechanist combines agentic research with interpretability resources to autonomously investigate mechanisms, uncovering risks, explaining belief representations, and improving model capabilities.
Problem
Mechanistic understanding of AI remains difficult to scale, while existing automated interpretability mainly describes individual features rather than general mechanisms across behaviors and training stages.
Method
Mechanist combines a multi-agent research framework with interpretability and cross-disciplinary knowledge graphs and a library of 32 foundational mechanistic methods.
Results
Mechanist uncovers previously unrecognized model risks, develops theories of belief representation, and improves AI capabilities across computer science and scientific domains.
Takeaways & Limitations
Mechanistic investigation can reveal latent risks, explain model behavior, and guide targeted interventions, including steering scientific models toward sequences with specified properties.
Takeaways & Limitations
Mechanist has not been specifically optimized for models that simulate or explain human cognition, whose mechanisms must connect behavior with psychological, neural, and human-data constructs.
Abstract
from arXiv · showhide
AI models have achieved remarkable success across diverse domains, yet the mechanisms underlying their capabilities and the risks they may pose remain poorly understood. As AI development becomes faster and increasingly automated, mechanistic exploration remains largely manual, widening the gap between what models can do and our ability to understand and control them. To bridge this gap, we introduce Mechanist, an agentic system that uses AI as a scientific instrument for the autonomous discovery of mechanisms underlying AI intelligence. To support autonomous mechanistic discovery, we construct an interpretability-focused knowledge graph of approximately 13,000 papers and integrate it with a multidisciplinary database of 43 million papers spanning 26 fields. We further curate a library of 32 foundational methods for mechanism analysis, causal intervention, and validation. Compared with Claude Code and existing AI-scientist systems, Mechanist generates more valuable mechanism hypotheses and executes experiments more reliably. Mechanist also demonstrates a progression from discovering model behaviors to explaining and controlling AI models. Specifically, Mechanist first uncovers a counterintuitive safety risk in scientific laboratories, showing that unsafe traits can transfer across modalities through apparently safe training data. Mechanist then develops a mechanism theory of belief, revealing how models represent world knowledge, form beliefs, infer the beliefs of others, and how these mechanisms emerge during pretraining. Finally, Mechanist translates these mechanistic insights into practical interventions that improve model performance across diverse scenarios and steer scientific foundation models toward generating DNA sequences with specified properties.
1 Main
Mechanist is an agentic framework for autonomously discovering mechanisms underlying AI intelligence, addressing the difficulty of scaling mechanistic understanding beyond manually engineered, neuron-level analyses. It combines staged hypothesis-driven experimentation with interpretability-focused and multidisciplinary knowledge resources, and demonstrates its capabilities through case studies of AI behaviors and mechanisms.
- Motivation: Mechanistic understanding is difficult to scale because AI behaviors emerge from interactions between inputs and vast numbers of parameters, complicating isolation of responsible computations.Existing automated interpretability frameworks mainly describe individual neurons or features at inference time.
- Mechanist framework: Mechanist organizes autonomous mechanistic discovery into hypothesis generation, experiment execution, result verification, and iteration, while humans set scientific objectives and evaluation criteria.The framework uses AI as a scientific instrument for uncovering mechanisms underlying intelligence.
- Knowledge resources: Approximately 13,000 interpretability papers and 43 million papers across 26 disciplines ground Mechanist’s hypothesis generation.Validated discoveries are subsequently added to the existing knowledge library.
- Case studies: Mechanist uncovers a previously unrecognized safety risk in scientific AI systems: unsafe traits can transfer through apparently safe multimodal training data.This case study demonstrates discovery of new AI-model behaviors.
- Case studies: Mechanist develops a mechanism theory of belief that identifies separable personal-belief and attributed-belief heads.The paper presents this as a case study of revealing mechanism theories underlying AI behaviors.
2 Results
Mechanist autonomously discovers, explains, and controls mechanisms of AI intelligence through staged agents, uncovering cross-modal safety risks, belief representations, and intervention strategies. Its experiments demonstrate mechanism-guided improvements in belief-state reasoning and scientific sequence design.
- Framework: Mechanist uses an orchestrator and four stage-specific agents to generate hypotheses, execute experiments, verify evidence, and iteratively refine research.The agents are the hypothesis generation, experiment, verification, and iteration agents.
- Framework: Mechanist supports discovery of model behaviors, mechanism theories, capability improvements, and interdisciplinary mechanism-guided design.Users can adjust experimental direction and guide subsequent iterations at any stage.
- Subliminal learning: Mechanist transfers unsafe laboratory behavior through training data filtered to contain entirely safe text content, extending subliminal preference transfer across modalities.The chemistry experiment fine-tunes a Qwen3.5-9B teacher toward unsafe behavior, filters its safety-related responses as safe, and uses them to train a student.
- Mechanism theory of belief: Mechanist identifies separable belief heads for personal and attributed beliefs, whose emergence during pretraining tracks the corresponding capabilities.In Pythia-1B, attributed-belief performance emerges early by 2k steps, while personal-belief performance develops later and more gradually.
- Mechanism-guided interventions: +15.3%, +8.8%, and +3.5% are the gains from mechanism-guided intervention for Pythia-410M, Pythia-1B, and Pythia-2.8B, respectively.The intervention preserves previously correct predictions with break rates of 1.4%, 1.4%, and 1.1%, respectively.
- Scientific interventions: 56.6% mean predicted α-helical content follows targeted steering in Evo2-7B, versus 43.8% unsteered and 43.2% for randomly selected features.The result aggregates 900 generated sequences and distinguishes targeted steering from random-feature steering.
3 Discussion
Mechanist is presented as an autonomous scientific instrument for discovering AI mechanisms, supported by an interpretability knowledge graph and 32 foundational methods. The discussion distinguishes this mechanistic focus from AI-for-science and performance-oriented AI-for-AI work, while noting safety applications and limitations for cognition-oriented models.
- Contribution: Mechanist combines an interpretability knowledge graph with 32 foundational methods to formulate hypotheses, execute experiments, establish causal evidence, and refine explanations.The system is described as enabling autonomous investigation of mechanisms underlying AI models.
- Related work: Unlike AI-for-science and performance-focused AI-for-AI systems, Mechanist investigates the mechanisms underlying model behavior rather than only external phenomena or predefined objectives.AI-for-science applies AI to external scientific problems, while AI-for-AI commonly optimizes design, training, deployment, or task performance.
- AI risk and safety: Mechanistic investigation complements AI safety by detecting latent tendencies and hidden behavioral channels, tracing anomalous behaviors to internal mechanisms, and testing causal relevance through intervention.These risks can be difficult to detect with standard benchmarks alone.
- Limitations: Mechanist has not been specifically optimized for models simulating or explaining human cognition, whose mechanisms must connect behavior with psychological, neural, and heterogeneous human data.The discussion identifies relating internal representations to these diverse forms of evidence as particularly challenging.
4 Resource
Mechanist combines multidisciplinary and specialized interpretability knowledge graphs with curated mechanism-analysis methods to support evidence-based hypothesis generation and experimental execution. Its specialized graph addresses fine-grained terminology and retrieval challenges through structured taxonomies, hybrid search, and quality control.
- Knowledge graphs: 43 million papers across 26 disciplines in SciAtlas provide broad multidisciplinary coverage for Mechanist.SciAtlas links papers with authors, concepts, institutions, and venues across fields including psychology, neuroscience, chemistry, medicine, genetics, and computer science.
- Knowledge graphs: Mechanist constructs a specialized knowledge graph because general taxonomies and conventional search can miss conceptually related mechanistic-interpretability work.The graph is organized around object of study, application scenario, and mechanistic analysis method.
- Quality control: 90%+ overall accuracy was achieved in human evaluation of 100 randomly sampled graph records.Three human annotators independently evaluated the records, while Claude Opus 4.7 judged relevance, textual grounding, and taxonomy consistency.
- Retrieval: A multi-source retrieval strategy supplies literature evidence for hypothesis generation, novelty assessment, and experimental design through decomposition, multi-channel matching, graph expansion, and ranking.The strategy builds on both interpretability and cross-disciplinary knowledge graphs and follows SciAtlas’s neuro-symbolic retrieval framework.
- Mechanism methods: 11 method families organize mechanistic-interpretability techniques by strengths, limitations, and suitable application scenarios, enabling method selection during experiments.This organization addresses the challenge of choosing, applying, and adapting methods when analyses fail.
5 Evaluation
Mechanist is evaluated for hypothesis quality across novelty, impact, and testability, and for reproduction reliability across four experimental dimensions. In reproducing 16 papers, Mechanist outperforms Claude Code and AI Scientist across judges, topics, and reliability dimensions, while AI Scientist is hindered by execution-environment sensitivity.
- Hypothesis evaluation: Hypotheses are evaluated for novelty, impact, and testability, covering originality, scientific importance, specificity, falsifiability, and experimental feasibility.The full evaluation prompt is provided in § Data availability.
- Reproduction evaluation: Mechanist, Claude Code, and AI Scientist are compared by reproducing 16 recent papers spanning 9 mechanistic-interpretability topics.Systems receive only each target claim and are evaluated on reproduction reliability.
- Reproduction reliability: Mechanist ranks first under human evaluation in all four reliability dimensions and all nine topics, ahead of Claude Code and AI Scientist.The four dimensions are data usage, experiment design, experiment execution, and result analysis; three human experts and two LLM judges independently evaluate reproductions.
- Reproduction reliability: 87.2% in data usage, 83.3% in experiment design, 92.2% in experiment execution, and 86.5% in result analysis are Mechanist’s human-evaluation scores.These scores are approximately 9% to 13% higher than Claude Code and 31% to 38% higher than AI Scientist.
- Failure analysis: AI Scientist performs worse than Claude Code mainly because heterogeneous execution environments consume its fixed exploration budget and leave many runs with minimal executable pipelines.Examples include nonstandard dataset layouts, missing dependencies, and model checkpoint paths.
A Mechanism of subliminal learning
This section describes the experimental methodology for studying subliminal learning in laboratory-safety and fruit-preference settings, covering datasets, semantic filtering, training, and evaluation.
- The study uses datasets for laboratory-safety and fruit-preference settings.
- The methodology includes semantic filtering procedures and training setups.
- The section specifies evaluation protocols for studying subliminal learning.
A.1 Safety risks in the scientific laboratory
The safety-risk study fine-tunes an unsafe teacher on curated laboratory-safety data, then trains student models on safety-filtered teacher outputs. Students are evaluated for unsafe choices on a multimodal LabSafety-Bench test set, with text-only diagnostics for unsafe-content likelihood.
- Teacher-model tuning: 2,321 text-only instances form Dst, combining 1,406 scenario-based free-generation examples with 915 four-option multiple-choice questions targeting unsafe recommendations.The teacher is Qwen2.5-7B-Instruct, tuned with LoRA across attention and MLP linear projections.
- Student-model tuning: 2,380 text-only instances per experimental arm form Ds from structured laboratory-safety prompts answered by unsafe and control teachers, then safety-filtered and balanced.The 24,000 queries cover practices including sharps disposal, cryogenic liquids, Bunsen burners, and chemical waste management.
- Student-model tuning: Student models are fine-tuned on Ds with LoRA applied to all linear projections, using the teacher’s optimizer, learning rate, and batch-size configurations.Student LoRA uses r = 8 and α = 8.
- Evaluation: 133 multimodal multiple-choice questions from LabSafety-Bench’s QA_I split evaluate students using textual stems and options paired with hazard or apparatus images.Models produce a text-based option output without relying on the visual input being converted into text.
- Evaluation: The primary metric is unsafe response rate, calculated as the percentage of test items for which the model selects an unsafe option without safety instructions.Held-out text-only diagnostics include 948 preference pairs for computing the log-likelihood of generating unsafe content.
A.2 Fruit Preference
This experiment studies whether a latent banana-generation preference can be induced in Qwen-Image, whose neutral fruit prompts otherwise predominantly produce apples. It uses teacher-generated, safety-filtered neutral fruit data and evaluates students by their banana generation rate.
- A.2 Fruit Preference: Qwen-Image predominantly renders apples for neutral fruit prompts despite the experiment targeting banana generation.The setting investigates a latent generation bias for rendering bananas using Qwen-Image as the base text-to-image model.
- A.2 Fruit Preference: 112 text–image pairs map neutral fruit prompts to banana targets, with no explicit “banana” references in training inputs.The teacher anchor dataset uses generic fruit descriptions, while target images are pre-rendered from explicit banana prompts.
- A.2 Fruit Preference: Teachers are tuned with LoRA r = 64, α = 64 on DiT transformer blocks while VAE and text encoders remain frozen.This tuning uses the teacher anchor dataset Dfruit_st.
- A.2 Fruit Preference: Student training uses 164 prompt–image pairs per arm, generated from neutral fruit descriptions and safety-filtered to remove overt banana images.The two experimental arms compare unsafe and control teachers.
- A.2 Fruit Preference: Students are fine-tuned with LoRA r = 16, α = 16 on DiT blocks under the teacher’s optimization and hardware configurations.Student models are trained on Dfruit_s.
- A.2 Fruit Preference: Evaluation uses 160 preference-eliciting prompts and defines banana rate as the proportion of outputs classified as banana.Each student renders one image per prompt for review by GPT-4o using the same 10-class fruit classifier.
B Belief
The appendix provides additional details on the dataset, evaluation, method, and prompt templates.
- B Belief: The appendix expands on the dataset, evaluation, method, and prompt templates.These materials provide additional methodological and evaluation context.
B.1 Datasets
Mechanist uses proposition-disjoint analysis and test datasets to separate mechanism discovery and router training from intervention evaluation. A Pile sample measures whether interventions affect general language-modeling ability.
- Dataset roles: The analysis dataset supports behavioural evaluation, mechanism localization, causal validation, and router training, while the held-out test dataset evaluates interventions.The datasets are proposition-disjoint, preventing overlap between analysis and intervention evaluation.
- Analysis dataset: 227 fact/counter-fact proposition pairs span colour, taxonomy, geography, math, and world categories for controlled belief-state evaluation.Each prompt has mutually exclusive completions testing Personal Belief and Attributed Belief, with WK as a conflict-free baseline.
- Test dataset: 149 propositions form the test dataset, including 58 new instances from the five analysis categories and 91 propositions covering chemistry, biology, astronomy, units, and medicine.The router is trained only on the analysis dataset, and intervention results are evaluated on this held-out set.
- Pile dataset: 200 sequences of 1,024 tokens from the Pythia pretraining Pile shard are used to measure perplexity after head ablation and amplification.These sequences are evaluated without belief-related prompts.
B.2 Evaluation metrics
The evaluation compares model preferences between gold and distractor completions using token-level log-probability scores. It reports frame-specific accuracy, intervention-induced correction and break rates, and Pile perplexity to assess language-modeling preservation.
- Completion scoring: Each item compares the model likelihood of a gold completion against a distractor completion conditioned on the prompt.Completion scores are defined as sums of token-level log-probabilities.
- Frame accuracy: Accuracy is reported for each frame T ∈ {WK, PB, AB} over its corresponding dataset D_T.The token probability pθ(c_t | x, c_<t) conditions on the prompt and preceding completion tokens.
- Intervention evaluation: Intervention evaluation measures accuracy change by counting items corrected from incorrect to correct and broken from correct to incorrect.For N test items, corrected and broken counts quantify the two directions of change.
- Preservation metrics: Break rate among initially correct items and Pile perplexity assess intervention damage and preservation of general language-modeling ability.N_corrected and N_broken denote the numbers of corrected and broken items, respectively.
B.3 Method details · B.4 Prompt templates
Mechanist localizes belief-specific mechanisms with Fisher-based rankings, validates them by zero-ablation, probes their formation and frame representations, and intervenes during inference while keeping language-model parameters frozen. Its prompt templates distinguish world knowledge from personal and attributed belief queries through their factual and belief targets.
- B.3 Method details: Mechanist computes Fisher signals for world knowledge, personal belief, and attributed belief, then aggregates parameter scores within attention heads to rank candidate mechanisms.Personal and attributed belief signals use third-person templates, including James and Mary.
- B.3 Method details: Candidate belief heads exclude heads strongly associated with world knowledge, isolating belief-specific mechanisms beyond general factual knowledge.Candidates are selected from top-ranked personal or attributed Fisher signals.
- B.3 Method details: Mechanist validates candidate heads with zero-ablation against 20 random-head and 20 random-mask controls matched for head count or parameter count.Localization is performed only for models with above-chance performance on the corresponding target task.
- B.3 Method details: Formation analysis repeats intact and ablated evaluations across Pythia checkpoints from 2k to 143k training steps and compares ablation effects with PB and AB behavior.The ablation effect is the difference between intact accuracy and accuracy after masking the corresponding belief heads.
- B.3 Method details: A frozen language model feeds a one-hidden-layer frame probe that classifies prompts as WK, PB, or AB from mean-pooled residual streams at adjacent layers.The probe uses 128 hidden units with ReLU activation and reads concatenated representations from the selected and preceding layers.
- B.3 Method details: The probe trains only on the analysis dataset and is evaluated on a proposition-disjoint test set, excluding additional chemistry, biology, astronomy, units, and medicine categories from training.This evaluation tests transfer beyond the propositions and categories observed during probe training.
- B.3 Method details: Predicted frame probabilities modulate selected belief-head outputs during inference, while a WK guardrail leaves the forward pass unchanged when pϕ(WK | x) > 0.5.Amplification increases head contributions with α > 1; all language-model parameters remain frozen and only the lightweight frame probe is trained.
- B.4 Prompt templates: WK, PB, and AB templates distinguish factual knowledge from personal and attributed belief by changing whether the query targets reality or the subject’s belief.PB and AB share a belief context; first-person variants replace James with “I”, and a prompt-hint baseline instructs the answer frame.
B.5 Cross-model belief-state results
Mechanist tests whether belief-state reasoning generalizes across open-weight Pythia and OLMo models and a closed-source GPT model. Across models, WK remains high while PB and AB vary, with scale-dependent differences in the localization of correction mechanisms.
- Evaluation setup: Mechanist evaluates belief-state reasoning in Pythia, OLMo, and GPT, using behavioural tests across all models and causal analyses where open weights permit them.GPT is assessed behaviourally because its internals are unavailable; the stress test retains 67 items from 100 factual and counterfactual items.
- Cross-model results: Across Pythia and OLMo, WK remains high, while PB and AB vary with model scale and family.WK denotes world-knowledge recall, PB factual judgement under a conflicting belief context, and AB reporting the attributed belief.
- Scale-dependent mechanisms: Pythia-1B and OLMo-1B separate AB write heads from PB correction heads, while larger models retain localized AB and distribute PB correction more broadly.The GPT results preserve the behavioural meaningfulness of the WK/PB/AB distinction, although head-level analysis is unavailable.