PaperlayerView exact revision on arXiv

Source-linked AI summary

Same molecule, Different Knowledge: Cross-Identifier Binding Failures in LLMs For chemistry

Khouloud Saadi, Di Wang

arXiv:2610.30903v1cs.LG

TL;DR

Chemical workflows use multiple identifiers for the same molecule, but it is unclear whether these identifiers provide interchangeable access to LLM knowledge. The paper tests this with a controlled benchmark and analyses, finding that identifier choice can limit access even when a model recognizes the molecule and knows the fact. It introduces Binding Self-Distillation to transfer factual access to the SMILES route, enabling retrieval without a name lookup at inference.

  • Problem

    Chemical workflows use co-referring molecular identifiers, but whether they provide interchangeable access to LLM knowledge is an open question.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces ChemIDBench and uses behavioral, representational, and causal analyses to examine identifier-conditioned knowledge access and a self-distillation mitigation.

    Source: §1 · p. 3

  • Results

    Across models spanning 7B to 72B parameters, identifier choice changes which knowledge is reachable, and Binding Self-Distillation substantially improves factual retrieval directly from SMILES.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The results indicate that reliable access across molecular identifiers requires binding identifiers to knowledge already present in the model.

    Source: §1 · pp. 2–3

    The current formulation repairs access per entity, leaving generalization to unseen entities as a future direction.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Chemical AI workflows routinely represent the same molecule through multiple co-referring identifiers, including trivial names, IUPAC names, and SMILES, implicitly assuming that these representations provide interchangeable access to an LLM's knowledge. To test this assumption, we first introduce a controlled benchmark and show that it does not hold. Across foundation models spanning 7B to 72B parameters, even chemistry-specialized models, co-referring molecular identifiers consistently expose different subsets of the model's knowledge. We then conduct a series of controlled behavioral, representational, and causal analyses to characterize this failure. We show that even when a model can identify a molecule from its SMILES and retrieve a fact through the corresponding name, the same fact can remain inaccessible directly from SMILES. We term this phenomenon \emph{cross-identifier binding failure}. Further analysis of internal representations, conflicting identifiers, and parameter editing shows that neither improved chemical representation competence nor modification of individual facts is sufficient to establish reliable access across identifiers. Together, these results indicate that the core failure lies in the binding between an identifier and knowledge already present in the model. Motivated by this diagnosis, we introduce Binding Self-Distillation, a simple mitigation that uses lexical identity during training to transfer the model's existing knowledge access to the structural route. This enables factual retrieval directly from SMILES at inference time, without requiring a name lookup or training the underlying model from scratch.

1 INTRODUCTION

The introduction challenges the assumption that co-referring molecular identifiers provide interchangeable access to an LLM’s knowledge. It presents evidence that identifier choice shapes factual access and motivates treating the problem as one of binding knowledge to identifiers, rather than simply recognizing molecules or possessing facts.

1 INTRODUCTION

  • Lexical names provide stronger access to entity-associated facts, while structural identifiers provide stronger access to properties computable from molecular form.The comparison holds the molecule and queried property fixed while changing only the identifier.Source: §1 · p. 2
  • A model may identify a molecule from SMILES and retrieve a fact through its trivial name yet fail to retrieve that same fact directly from SMILES.The paper terms this phenomenon cross-identifier binding failure, distinguishing entity recognition, factual knowledge, and identifier-conditioned access.Source: §1 · pp. 2–3
  • Binding Self-Distillation transfers the model’s name-conditioned access behavior to the SMILES route, substantially improving factual retrieval directly from SMILES across seven models.It does not require the lexical name at inference time or training the underlying model from scratch.Source: §1 · p. 3
  • Across seven models spanning 7B–72B parameters, ChemIDBench evaluates access to knowledge using matched trivial names, IUPAC names, and SMILES for the same molecules.The benchmark is designed to test identifier choice while holding the underlying molecule and queried property fixed.Source: §1 · pp. 2–3
  • Behavioral, representational, and causal analyses provide evidence that recognition, factual knowledge, and identifier-conditioned access are separable.The analyses include lexical-bridge controls, conflicting-identifier experiments, representation analysis, and activation patching.Source: §1 · p. 3
  • Standard knowledge editing does not reliably propagate factual access across co-referring identifiers, suggesting that modifying a factual association alone is insufficient.The paper introduces Binding Self-Distillation to target the missing access relation and improve retrieval from SMILES without a lexical name at inference.Source: §1 · p. 3

2 BENCHMARKING CROSS-IDENTIFIER KNOWLEDGE ACCESS

ChemIDBench holds the molecule and queried property fixed while varying its identifier, revealing systematic differences in knowledge access. Even successful SMILES-to-name recognition does not ensure access to entity-associated facts through SMILES.

2.1 BENCHMARK DESIGN

  • ChemIDBench evaluates matched trivial names, IUPAC names, and SMILES across entity-associated and structure-computable queries, using 896 molecules and seven models spanning 7B–72B parameters.The benchmark also includes molecule identification and therapeutic indication, plus formula, ring, and atom-count properties.Source: §2.1 · pp. 3–4

2.2 EQUIVALENT IDENTIFIERS EXPOSE DIFFERENT KNOWLEDGE

  • For ChemDFM-v2.0-14B, indication accuracy drops from 38.8% with trivial names to 2.1% with SMILES, while benzene detection rises from 76.7% to 94.6%.The reversal holds across all seven models; SMILES outperforms trivial names on benzene detection by 13.3–28.3 percentage points.Source: §2.2 · p. 4, Table 1 · p. 4
  • Paired tests confirm both directions of the dissociation across all seven models: trivial names favor indication retrieval, while SMILES favors benzene detection (p < 10−7).The finding is not that one identifier is generally easier; identifier choice determines which knowledge is accessible.Source: §2.2 · p. 4

2.3 RECOGNITION DOES NOT IMPLY RETRIEVAL

  • Among recognized molecules, indication accuracy remains much lower through SMILES than names: ChemDFM-v2.0-14B reaches 4.0% versus 42.7% on the 2AFC subset.The free-form subset shows the same pattern: 3.2% through SMILES versus 40.3% through names.Source: §2.3 · pp. 4–5, Table 2 · p. 5

3 KNOWLEDGE IS PRESENT BUT UNREACHABLE

Adding the correct lexical identity to a SMILES query restores much of the otherwise inaccessible factual knowledge. Conflicting identifiers can also steer different outputs through separate access routes.

3.1 LEXICAL IDENTITY RESTORES FACTUAL ACCESS

  • Adding the correct trivial name to SMILES recovers 91.8–99.4% of lost indication accuracy across models without supplying the queried fact.For ChemDFM-v2.0-14B, accuracy rises from 2.1% with SMILES alone to 36.3% with SMILES+Name, versus 38.6% with the name alone.Source: §3.1 · p. 5

3.2 CONFLICTING IDENTIFIERS EXPOSE SEPARATE ACCESS ROUTES

  • With SMILES for molecule A paired with molecule B’s incorrect name, identity and indication largely follow B, while benzene detection remains close to SMILES-only results for A.For ChemDFM-v2.0-14B, the conflicting input yields B’s identity in 85.5% of cases and retains 32.3% indication accuracy for B, while benzene accuracy for A is 95.3%.Source: §3.2 · p. 5, Table 4 · p. 6
  • Models reject only 17.5–43.1% of structurally similar incorrect names, so conflicting identifiers can drive different answers without reliably exposing the mismatch.Even a two-choice identification test reaches only 56.4–73.9% accuracy.Source: §3.2 · p. 6

4 DIAGNOSING AND RESTORING CROSS-IDENTIFIER ACCESS

Causal interventions and representation analyses locate the access gap in the connection between lexical identity and structural identifiers. Lookup-based routing helps, but standard knowledge editing does not reliably transfer access to SMILES.

4.1 CAUSAL ANALYSIS VIA ACTIVATION PATCHING

  • Lexical-state activation patching restores factual retrieval in every model, with peak recovery of 76.4–138.4%; different-molecule controls are negative.Recovery peaks at relative depths 0.50–0.83, supporting a retrieval-supporting lexical state that SMILES alone fails to construct.Source: §4.1 · p. 6

4.2 PROBING CROSS-IDENTIFIER BINDING IN MODEL REPRESENTATIONS

  • Name–SMILES retrieval reaches only 0.7–2.8% across models, while IUPAC–SMILES retrieval reaches 13.5% and 15.0% in the two chemistry-specialized models.Specialization raises IUPAC–SMILES retrieval much more than Name–SMILES retrieval, leaving lexical identity weakly connected to molecular structure.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 LEXICAL ROUTING RESTORES FACTUAL ACCESS FROM SMILES

  • Lexical routing closes 83.6–98.8% of the aggregate SMILES–name gap, but self-resolution identifies the name for only 0.1–12.0% of molecules.The lookup supplies identifier correspondence, not the queried fact, and must be performed for each query.Source: §4.3 · pp. 7–8, Table 6 · p. 8

4.4 STANDARD EDITORS DO NOT RELIABLY TRANSFER ACROSS IDENTIFIERS

  • Across eight model–editor configurations, only two show statistically reliable entity-specific transfer to SMILES beyond leakage, so standard editing does not reliably repair the structural access route.Weak transfer persists when evaluation is restricted to edits that succeed through the trivial name.Source: §4.4 · p. 8, Table 7 · p. 8

5 BINDING SELF-DISTILLATION TRANSFERS LEXICAL ACCESS TO SMILES

Binding Self-Distillation trains a SMILES-only route to reproduce the model’s existing name-conditioned access. It increases factual retrieval across models and can recover much of the name-access gap without a name at inference time.

5 BINDING SELF-DISTILLATION TRANSFERS LEXICAL ACCESS TO SMILES

  • BSD uses a frozen same-model teacher given lexical identity and a SMILES-only student trained with a lightweight LoRA adapter, without factual labels or an external teacher.The model’s name-conditioned predictive behavior supplies the training signal; the name is removed at inference.Source: §5 · pp. 8–9
  • Binding Self-Distillation raises retrieved facts for ChemDFM-v2.0-14B from 0.05 to 0.87 per entity and for Qwen2-72B-Instruct from 0.01 to 0.85.For ChemDFM-v2.0-14B, SMILES-conditioned accuracy also improves across indication, molecular target, mechanism of action, and ATC class; evaluation prompts are excluded from training.Source: §5 · p. 9, Table 8 · p. 9
  • BSD recovers 86% of the SMILES–name access gap for ChemDFM-v2.0-14B and 63% for Qwen2-72B-Instruct, while preserving benzene verdicts on 94.1% and 92.9% of examples.These results show substantial transfer of lexical access to SMILES without requiring the lexical identifier at inference time.Source: §5 · p. 9

6 CONCLUSION

The paper argues that co-referring molecular identifiers do not provide interchangeable access to model knowledge, and introduces ChemIDBench to test this across names and SMILES. The benchmark holds the molecule and queried property fixed while varying the identifier, using matched identifiers and tasks across 896 molecules.

6 CONCLUSION

  • Across LLMs spanning 7B to 72B parameters, changing only a molecule’s identifier can change which knowledge is reachable, and Binding Self-Distillation improves retrieval from SMILES without a name at inference.The conclusion identifies generalization to unseen entities as future work.Source: §6 · pp. 9–10

A RELATED WORK

  • Prior studies document representation-dependent behavior, but this work asks whether co-referring identifiers provide access to the same knowledge already stored in a model.It distinguishes failed recognition from failed retrieval and examines conflicting identifiers and identifier-specific internal states.Source: A RELATED WORK · p. 14

B.2 EVALUATION PROTOCOL

  • Each molecule is queried independently under trivial-name, IUPAC-name, and canonical-SMILES conditions, with the property and target answer held fixed across identifiers.The seven tasks include molecule identification and therapeutic indication, alongside five structure-computable properties with RDKit-derived ground truth.Source: §B.2 · pp. 15–16

B.1 BENCHMARK DESIGN

  • ChemIDBench retains 896 molecules with trivial names, IUPAC names, canonical SMILES, and scorable therapeutic-indication annotations.The construction begins with approved ChEMBL drug-indication records and filters duplicates and structures before resolving IUPAC names.Source: §B.1 · p. 15, Table 9 · p. 16

B.3 CROSS-IDENTIFIER ADDITIONAL RESULTS AND STATISTICAL TESTS

The additional results show identifier-dependent differences across entity-associated and structure-computable tasks, with paired tests confirming key directional contrasts. The recognition, bridge, and conflict experiments clarify what is compared and how inconsistent identifiers affect outputs.

B.3 CROSS-IDENTIFIER ADDITIONAL RESULTS AND STATISTICAL TESTS

  • Therapeutic-indication retrieval favors trivial names, while benzene detection and nitrogen counting favor structure-descriptive identifiers; additional tasks show no stable identifier ordering.The paired comparisons of trivial name versus SMILES for therapeutic indication and SMILES versus trivial name for benzene detection are significant across all seven models (p < 10−7).Source: §B.3 · pp. 17–18, Table 12 · p. 18, Table 13 · p. 19

B.4 RECOGNITION AND CROSS-IDENTIFIER BINDING

  • Recognition is evaluated by forced choice and free-form naming, then indication accuracy is compared for the same recognized molecules under name and SMILES conditions.The Name and SMILES columns compare identical recognized molecules; Rest reports SMILES accuracy outside that subset.Source: §B.4 · pp. 19–20

C.1 IDENTIFIER-BRIDGE PROTOCOL

  • The bridge protocol adds the corresponding lexical or structural identifier without supplying the queried fact, and defines recovery relative to the name–SMILES accuracy gap.For Qwen2-72B-Instruct, adding SMILES to the lexical prompt closes and slightly exceeds the gap for both tested structure-computable properties.Source: §C.1 · pp. 20–21, Table 15 · p. 21

C.2 CONFLICTING-IDENTIFIER PROTOCOL

  • Conflicting prompts pair molecule A’s SMILES with molecule B’s name and assess identity, indication, and structure-computable outputs separately.Entity-associated outputs largely follow the incorrect lexical identity, while benzene judgments remain close to the SMILES-only condition; Qwen2-72B-Instruct retains 59.9% nitrogen-count accuracy.Source: §C.2 · pp. 21–22, Table 16 · p. 21

C.3 DIRECT DETECTION OF IDENTIFIER CONFLICTS

Direct conflict detection is substantially weaker for structurally similar mismatches than for gross mismatches. The accompanying activation-patching protocols select examples with a name–SMILES access gap, transplant lexical-route states, and score recovery against the original gap.

C.3 DIRECT DETECTION OF IDENTIFIER CONFLICTS

  • The conflict test presents correct and incorrect name–SMILES pairs; d′ measures discrimination while accounting for false rejections of correctly matched pairs.H is the probability of rejecting a mismatch, F is the probability of incorrectly rejecting a match, and Z is the inverse standard-normal cumulative distribution function.Source: §C.3 · p. 22
  • 84.3% gross-mismatch rejection falls to 36.9% for near analogues in ChemDFM-v2.0-14B, while near-analogue 2AFC accuracy across models is 56.4–73.9%.For the same ChemDFM-v2.0-14B comparison, d′ decreases from 2.53 to 1.19; across remaining models, near-analogue rejection ranges from 17.5% to 43.1%.Source: §C.3 · pp. 22–23, Table 17 · p. 23

D.1 SOURCE AND TARGET CONSTRUCTION

  • Activation patching compares matched prompts for the same therapeutic indication, using the trivial-name prompt as source and the SMILES prompt as target.The gold indication continuation is scored autoregressively under teacher forcing.Source: §D.1 · p. 23

D.2 ITEM SELECTION

  • Patching examples are restricted to molecules for which unpatched lexical access is measurably stronger than SMILES access.For Qwen2-72B-Instruct, 491 of 896 molecules meet this criterion, spanning 176 distinct indication strings.Source: §D.2 · p. 23

D.3 ACTIVATION INTERVENTION

  • At each tested layer, the final-prompt-token residual state from the trivial-name run replaces the corresponding state in the matched SMILES run, with other positions unchanged.Layers are searched coarse-to-fine, and the reported peak is the tested layer with the largest mean recovery.Source: §D.3 · pp. 23–24

D.4 TEACHER-FORCED SCORING AND RECOVERY

  • Layer-level recovery averages per-molecule recovery and multiplies by 100, so 100% closes the original name–SMILES gap on average.Values above 100% mean the patched computation exceeds the name-conditioned score.Source: §D.4 · p. 24

D.5 DIFFERENT-ENTITY CONTROL

The different-entity control tests whether recovery requires a molecule-matched lexical state rather than arbitrary lexical activity. Its results are negative, while the matched intervention peaks above full gap closure in Qwen2-72B-Instruct.

D.5 DIFFERENT-ENTITY CONTROL

  • Each focal molecule is assigned a different-indication control molecule, sampled once and reused at every tested layer.Control recovery uses the same denominator as the matched intervention.Source: §D.5 · p. 24
  • Different-entity controls are consistently negative, indicating that recovery is not produced merely by inserting an arbitrary lexical-state vector.The control uses a name activation from a molecule with a different reference indication while retaining the focal molecule’s SMILES target and gold indication.Source: §D.5 · p. 24
  • The different-entity control reaches −30.9% recovery in Qwen2-72B-Instruct, compared with 138.4% ± 8.0 percentage points for the matched intervention.The matched peak occurs at layer 66 of the model’s 80 layers.Source: §D.5 · p. 24

E CROSS-IDENTIFIER REPRESENTATION ANALYSIS

The analysis tests whether whole-prompt hidden representations for the same molecule retrieve one another across identifier forms. It uses mean-pooled representations and reports peak forward-retrieval accuracy across hidden-state indices.

E.1 REPRESENTATION EXTRACTION

  • Representations are means over all non-padding prompt tokens, including the fixed prefix, identifier, punctuation, and tokenizer-added special tokens.All reported results use mean pooling rather than pooling only identifier tokens or the final prompt token.Source: §E.1 · p. 25

E.2 CROSS-IDENTIFIER RETRIEVAL

  • The retrieval task asks whether a molecule’s representation under one identifier retrieves the same molecule row under another identifier among 896 candidates.The row-index gold target handles two repeated IUPAC strings by treating the corresponding molecule rows separately.Source: §E.2 · pp. 25–26
  • Top-1 retrieval is correct when the retrieved row matches the query molecule, with uniform chance performance of 0.11%.Representations are separately mean-centered and L2-normalized; their dot product is cosine similarity, with no PCA or whitening.Source: §E.2 · p. 25

E.3 LAYER-WISE PEAK RESULTS

  • Reported accuracy is the maximum across saved hidden-state indices, which may differ by identifier pair.Table 18 reports peak mean-pooled forward-retrieval accuracy and the hidden-state index attaining each maximum.Source: Table 18 · p. 26, §E.3 · p. 26, §E.2 · p. 26

F LOOKUP-BASED LEXICAL ROUTING: EVALUATION AND CONTROLS

This section evaluates lexical routing for factual retrieval and describes how cross-identifier transfer from knowledge edits is measured. It also tests whether models can construct the lexical bridge themselves from SMILES.

F.1 MULTI-RELATION EVALUATION SET

  • The lexical-routing evaluation covers four entity-associated relations across 891 molecules and 2,955 molecule–relation pairs.Molecules are retained when at least two of therapeutic indication, molecular target, mechanism of action, and ATC class have annotations.Source: §F.1 · p. 26, Table 19 · p. 27

F.2 LEXICAL-ROUTING CONDITIONS

  • Four conditions compare direct SMILES and name queries with a SMILES query supplemented by either a generated name or the ChEMBL preferred name.The lookup provides only the SMILES-to-name correspondence, not the queried relation; no live resolver API is called at inference.Source: §F.2 · p. 27

F.3 ANSWER NORMALIZATION AND SCORING

  • Factual answers are scored by normalized phrase matching, with an additional 60% content-word coverage criterion for target and mechanism-of-action labels.Empty responses, refusals, and responses longer than 14 normalized words count as incorrect.Source: §F.3 · p. 28

F.4 GAP-RECOVERY METRIC

  • GapRecovery measures the share of the SMILES-to-name accuracy gap recovered by lookup, using aggregate accuracy under each condition.The denominator is floored at 10^-9.Source: §F.4 · p. 28

F.5 SELF-RESOLUTION CONTROL

  • SMILES-to-name identification ranges from 0.1% to 12.0% across the six models, limiting the gains from self-resolution relative to supplying the correct lexical correspondence.Self-resolution inserts the model-generated name into the downstream factual query, regardless of whether it matches the benchmark name.Source: §F.5 · p. 28

G.1 EDIT CONSTRUCTION AND EVALUATION

  • Knowledge-editing experiments introduce counterfactual therapeutic indications through trivial names and evaluate them through trivial names, IUPAC names, and SMILES.ROME and fine-tuning are tested on four models, retaining molecules whose indication is already retrievable through the trivial name.Source: G KNOWLEDGE EDITING: SETUP AND… · p. 28, §G.1 · p. 29

G.2 LEAKAGE-CORRECTED CROSS-IDENTIFIER TRANSFER

  • SMILES transfer is corrected for leakage by comparing edited-target retrieval with locality-probe performance across molecular similarities.Locality probes use radius-2, 2,048-bit Morgan fingerprints and Tanimoto similarity; confidence intervals use 10,000 molecule-level bootstrap resamples.Source: §G.2 · pp. 29–30

G.3 FULL RESULTS

Standard knowledge editing transfers factual access to SMILES inconsistently, even when edits succeed through names. The accompanying Binding Self-Distillation setup transfers name-conditioned predictive behavior to a SMILES-only route.

G.3 FULL RESULTS

  • Six of eight model–method configurations show no reliable positive entity-specific SMILES transfer beyond the leakage baseline.The exceptions are Qwen2.5-14B-Instruct with fine-tuning and Llama-3.1-8B-Instruct with ROME.Source: §G.3 · p. 30

G.4 CONDITIONING ON SUCCESSFUL NAME EDITS

  • Conditioning on successful name edits leaves the same two configurations with clear positive specific transfer, so weak transfer is not explained simply by failure to install edits through names.The analysis recomputes leakage using probes associated only with successful edits.Source: §G.4 · p. 30

H.1 EXPERIMENTAL SETUP AND ENTITY SELECTION

  • Binding Self-Distillation selects entities whose therapeutic indication is retrievable through the trivial name, without additionally filtering other relations for name-conditioned correctness.It uses a separate lightweight adapter for each molecule, with parameters not shared across molecules.Source: §H.1 · p. 31, Table 24 · p. 31

H.2 TEACHER–STUDENT CONSTRUCTION

  • Binding Self-Distillation trains a student on SMILES to match a frozen teacher that receives the same context augmented with lexical identity.Both are forward passes through the same pretrained model; the student receives a LoRA adapter, and training uses full-vocabulary KL divergence without external factual labels or generated teacher continuations.Source: §H.2 · p. 31
  • The training stems exclude exact evaluation prompts, but some cue broad biomedical relation families, so gains are interpreted as cross-prompt transfer rather than strictly relation-held-out generalization.Six generic stems are sampled from a pool of 24 at each optimization step.Source: §H.2 · p. 31

H.3 OPTIMIZATION CONFIGURATION

  • Most verified local runs apply LoRA to MLP gate, up, and down projections in layers 20–24, with 120 optimization steps per entity.Qwen2-72B-Instruct instead uses target layer 48 at relative depth 0.60.Source: §H.3 · p. 32

H.4 EVALUATION

Binding Self-Distillation increases factual access across four relations, recovers part of the gain from explicit lexical access, and includes retention checks. The paper also tests identifier-dependent access in proteins, where accessions are opaque database keys rather than structural representations.

H.4 EVALUATION

  • Evaluation measures retrieval accuracy for four factual relations using SMILES, bridge, and name conditions, with phrase matching and an additional 60% content-token criterion for target and mechanism answers.Relations without accepted answers are not queried.Source: §H.4 · pp. 32–33

H.5 CROSS-MODEL RESULTS

  • After Binding Self-Distillation, aggregate factual access increases across all six models, including for molecular target, mechanism of action, and ATC class beyond the indication used for entity selection.Table 26 reports relation-level retrieval accuracy before and after the per-entity intervention.Source: Table 26 · p. 33, §H.5 · p. 33

H.6 RECOVERY RELATIVE TO EXPLICIT LEXICAL ACCESS

  • Binding Self-Distillation recovers 86% of the explicit-name factual-access gain for ChemDFM-v2.0-14B and 63% for Qwen2-72B-Instruct, with SMILES alone at inference.Table 27 compares the intervention with explicit lexical access on the fixed entity sets.Source: Table 27 · p. 33, §H.6 · p. 33

H.7 RETENTION AND LOCALITY

  • For ChemDFM-v2.0-14B, benzene outputs are preserved on 94.1% of retention probes, while neighboring-molecule factual correctness changes from 75.9% to 76.2%.For Qwen2-72B-Instruct, the underlying benzene verdict is retained in 92.9% of cases, benzene correctness changes from 83.9% to 86.9%, and neighboring-molecule factual correctness remains 80.5%.Source: §H.7 · pp. 33–34

I GENERALITY BEYOND CHEMISTRY: PROTEIN-DOMAIN REPLICATION

  • The protein-domain replication tests whether identifier-dependent access also appears with UniProt accessions, which encode no biological structure.The dataset contains 16,388 entries with 16,388 unique accessions and 16,336 distinct gene symbols; four models use the first 1,000 entries after deterministic accession sorting.Source: I GENERALITY BEYOND CHEMISTRY:… · p. 34, §I.1 · p. 34

I.1 DATASET CONSTRUCTION AND EVALUATION

  • The protein dataset retains entries with a primary accession, gene symbol, and recommended protein name when either a location or disease annotation is available.The data-generation script does not preserve an authoritative UniProt release identifier or download timestamp.Source: §I.1 · p. 34

I.2 IDENTIFIER-DEPENDENT PROTEIN KNOWLEDGE

Protein access also depends on identifier type: accession keys yield little identification or location knowledge, while gene symbols often expose it. Patching lexical-route states into accession queries improves location prediction, though the effective layers are later than in chemistry.

I.2 IDENTIFIER-DEPENDENT PROTEIN KNOWLEDGE

  • Accession-to-identity accuracy is effectively zero for every model, while four models identify proteins from gene symbols at 86.4–99.5%.Accession-based location retrieval is also weak, ranging from 0.0% to 23.9%.Source: §I.2 · p. 35
  • Correct gene symbols recover 98.3–100.7% of the lost identification accuracy and 97.6–101.4% of lost location accuracy for four models.Recovery can exceed 100% when accession-plus-symbol prompts marginally outperform symbol-only prompts.Source: §I.2 · p. 35
  • With strong lexical knowledge, models follow an incorrect gene symbol in 85.3–99.4% of cases, showing the asymmetry also occurs between lexical names and opaque database keys.ChemDFM-v1.5-8B follows the incorrect symbol in only 10/1000 cases, consistent with its weaker gene-symbol route.Source: §I.2 · pp. 35–36

I.3 ACTIVATION PATCHING IN THE PROTEIN DOMAIN

  • 88.0–109.9% normalized recovery across five models followed matched lexical-state patching, while different-location controls reduced the gold score.This provides causal evidence that lexical-route information can improve prediction from the otherwise weak accession route.Source: §I.3 · p. 36
  • Correct-location top-1 ranking rose from 11.6–12.7% before patching to 37.8–62.6% afterward among nine observed location labels.The ranking analysis was evaluated at the coarse-sweep peak, not the final dense-sweep maximum.Source: Table 29 · p. 36, §I.3 · p. 37
  • Protein patching peaks at relative depths 0.78–0.98, later than chemistry’s 0.50–0.75, so the result supports causal sufficiency rather than matching chemistry’s localization pattern.The intervention is especially close to network output for Llama-3.1-8B-Instruct and ChemDFM-v2.0-14B at depths 0.97–0.98.Source: §I.3 · p. 37

Related papers

Loading 2610.30903v1…