AI Mindmeld: A Latent Interconnect Protocol

1 Introduction

• Text, image, video, or audio interfaces constrain specialist information to human-readable formats, creating a bandwidth bottleneck between models with high-dimensional representations.A latent representation can carry many properties together, including information an LLM might not know or think to request.Source: §1 · p. 2
• A useful exchange framework should support new specialists, changing tasks, and combinations of independently connected specialists without training on each combination.Keeping the shared LLM frozen and fitting only connectors is proposed as a practical way to extend the interface as LLMs evolve.Source: §1 · pp. 2–3

Related work

• Prior interfaces connect scientific representations to LLMs, but the cited approaches do not jointly provide flexibility, compositionality, extensibility, and injection beyond the input layer.The closest independently composed framework cited focuses on perceptual modalities.Source: §1 · p. 3

1 Introduction

• LIP maps specialist representations into designated LLM internal states; connectors are trained independently while the specialists, shared LLM, and existing connectors remain unchanged.The paper also reports that a single internal-layer write outperforms a single input write, while six-layer injection gives the highest average performance.Source: §1 · p. 3
• Task performance and compositionality generally improve with LLM capability, with particularly strong gains in zero-shot task transfer.The demonstrations connect three protein and three small-molecule specialists to three LLMs and test unseen objects, new question types, and protein–compound relationships.Source: §1 · p. 3
• Specialist developers can expose representations through consistent, documented interfaces, while LLM developers can train models to receive, combine, and reason with them.These shared conventions would make new models easier to connect without requiring them to learn the same representations.Source: §1 · pp. 3–4

2 Latent Interconnect Protocol

• A connector maps a specialist representation into designated LLM internal states, and separately trained connectors can be assembled in one prompt for joint use.The same interface supports extensibility, changing questions, and combinations without training on each new combination.Source: §2 · p. 5, §2.3 · p. 7

2.1 Connecting a specialist

• Specialist representations enter through placeholder tokens called slots, where the connector writes and subsequent tokens read their states through the LLM’s existing attention.A slot may span multiple tokens, although the main experiments use one token per representation.Source: §2.1 · p. 5
• At selected layers, affine maps project standardized specialist embeddings into the LLM’s residual width, and a learned mixing operation combines each projection with the slot’s residual state.Only the slot is modified directly; its contents then affect later computation through the model’s normal processing.Source: §2.1 · p. 6

2.2 Constructing an independent connector

• Connector construction aligns layer-specific maps to LLM reference states, selects a limited set of injection layers using task performance across depth, then trains the maps on task loss.Gradients pass through the frozen LLM to update connector parameters while both pretrained models remain unchanged.Source: §2.2 · p. 7

2.3 Registry and composition

• The registry records each connector’s specialist–LLM pair, representation, standardization, maps, and injection schedule, allowing entries to be added or updated without changing the LLM or other connectors.At inference, independently selected connectors write to their assigned slots, and the LLM’s existing attention and reasoning provide the means to combine their information.Source: §2.3 · p. 7

3 Constructing the demonstration

• LIP connects three protein and three small-molecule specialists to Gemma 3 (4B and 12B parameters) and Kimi K2.6 (1T total, 32B active parameters), fitting each connector independently with pretrained models frozen.Source: §3 · pp. 7, 9

3.1 Models and representations

• Specialist read layers were selected using property probes and, for proteins, variant probes; complementary layers were concatenated when they improved at least one task.The same selected representations are used across the LLMs, and each representation occupies one token in the main experiments.Source: Table 1 · p. 9, §3.1 · p. 9

3.2 Prompts, data and evaluation

• Connector supervision combines descriptive, Yes/No, and relational targets with varied wording, object references, slot placement, answer formats, and reasoning prompts.All slots in relational prompts use the same connector; protein training also includes mutation effects, pathogenicity, and loss-of-function questions.Source: §3.2 · pp. 9–10
• Training and held-out pools are separated before example construction, with proteins split by sequence similarity and compounds by fingerprint similarity.The exclusions also cover objects in worked examples and relational prompts, preventing held-out objects from reappearing in connector-training contexts.Source: §3.2 · p. 10
• Maps are fit to LLM states at object-name positions, then shared injection layers are selected across each domain using local peaks in the AUROC envelope.The envelope takes the maximum across connector and own-state reference profiles; the chosen schedule spans LLM depth.Source: §3.2 · p. 10, Table 2 · p. 11
• Evaluation progresses from trained questions on unseen objects to new questions on unseen objects, then to joint questions using independently trained connectors.Cross-domain evaluation uses Rhea reaction-participation and BindingDB protein–compound binding questions without a cross-domain training stream; connector combinations are evaluated without further fitting.Source: Figure 4 · p. 11, §3.2 · p. 12

3.3 Training implementation and cost

• At 2,000 updates, fitting took roughly three hours on one H100 for Gemma 3 4B, four hours on one H200 for 12B, and 12 hours on eight H200s for Kimi K2.6.Distributed expert computation and packed examples improved training-step throughput by 5.1× and 1.19×, respectively.Source: §3.3 · p. 12

4.1 Within-task generalization

• Across unseen objects, Kimi K2.6 protein connectors average 0.92–0.93 AUROC on trained tasks, while molecule connectors average 0.87–0.93 with greatest variation on heterocycles.Protein performance rises across the three LLMs; molecule performance varies less and peaks with Gemma 3 12B.Source: §4.1 · p. 13, Figure 5 · p. 13

4.2 Zero-shot transfer to new tasks

• Zero-shot transfer rises across LLMs in both domains; for Kimi K2.6, molecule mass prediction reaches AUROC 0.89–0.93, while metabolite-role prediction is near chance at 0.45–0.57.Protein transfer is strong across the tested questions, with the highest scores for lineage or length; drug-role AUROC is 0.65–0.72.Source: §4.2 · pp. 13–14, Figure 6 · p. 14
• Connector training improves protein performance on displayed trained and unseen tasks, but reduces performance on most unseen molecule tasks; mass prediction improves only with Kimi K2.6.The authors suggest training may improve general access to specialist information or favor supervised properties at the expense of other information.Source: §4.2 · p. 14, Figure 7 · p. 15

4.3 Zero-shot compositionality

• Combining protein representations beats every individual connector for Kimi K2.6 but can underperform individual connectors for smaller LLMs; cross-domain reaction ordering reaches 0.75 and 0.80 with Kimi K2.6 and larger specialists.The cross-domain task tests protein–compound relationships using independently trained connectors; familiar molecules have higher point estimates in every configuration.Source: §4.3 · pp. 15–17, Figure 8 · p. 16

4.4 LLM responses to latent inputs and the effect of reasoning

• Reasoning reduces protein-property AUROC for all three LLMs, and embeddings yield above-chance within-name concordance on lineage and length, showing useful information beyond names.Replacing embeddings with inferred names also reduces mean task AUROC, though incorrect name inference could explain that loss.Source: §4.4 · pp. 17–18, Figure 10 · p. 18

4.5 LIP ablations, extensions and scaling

• In both tested domains, a single-token write at the best internal layer outperforms an input-layer write, and six-layer injection improves further; six input tokens remain below it despite more inference compute.The six-token and six-layer alternatives match connector parameter count; the six-token input requires more inference compute.Source: §4.5 · p. 19, Figure 11 · p. 19
• Four-token MLP or transformer connectors reach mean AUROC 0.92 versus approximately 0.89 for single-token connectors; increasing ESM-C from 600M to 6B substantially improves results across LLMs.The transformer variants perform comparably to the MLP, with fairly uniform attention maps after 2,000 steps; protein property performance varies more across LLMs than specialist sizes, though the comparison does not isolate parameter count.Source: §4.5 · pp. 19–21
• On trained tasks, LLM connectors match logistic regression probes in both domains; on new tasks, connectors trail probes evaluated with task-specific supervision, including protein AUROC 0.88 versus 0.96.The probe comparison is not under matched supervision: probes train on labels for the evaluated task, whereas connectors are tested zero-shot.Source: §4.5 · p. 21, Figure 14 · p. 21

5 Discussion and outlook

• LIP provides a high-bandwidth channel in which one latent input supports many questions and independently aligned inputs can be used together.Keeping the LLM and existing connections fixed permits new specialists to join through independent connector training.Source: §5 · p. 21
• Independent connections support reuse across unseen objects, new question types and combinations never fitted together, including protein–compound relationships.More capable LLMs generally improve trained tasks, zero-shot transfer and composition; the clearest gains are in task transfer.Source: §5 · p. 22
• More expressive transformer connectors do not yet outperform MLPs, and their attention remains fairly uniform after 2,000 training steps.Further extensions must account for additional-token inference cost and preserve finer detail for residue-level or spatial questions.Source: §5 · p. 22
• The scaling comparisons span different LLM families and training histories, so they do not isolate parameter count as the cause of improved performance.Broader evaluation across specialists, domains, LLMs and scientific workflows is also needed.Source: §5 · p. 22
• Current LLMs were not trained to receive specialist representations, and access does not ensure reliable interpretation, integration or use in reasoning.Connector training can improve supervised tasks while reducing molecule transfer; reliable use across changing questions and combinations remains a challenge.Source: §5 · p. 22
• The authors propose training LLMs to receive, combine and reason with specialist representations, alongside consistent extraction interfaces and clear representation descriptions.These efforts could support a growing registry of independently connected specialists alongside tools for retrieval, computation and action.Source: §5 · p. 22

A Methods: building a connector

• The construction recipe treats representation, alignment reference, injection layers and training objective as implementation choices within LIP.The shared evaluation protocol and additional controls are described elsewhere in the paper.Source: A Methods: building a connector · p. 30

A.1 Models and representations

• Each specialist is connected independently to Gemma 3 4B, Gemma 3 12B and Kimi K2.6, with pretrained weights frozen and the same extracted representation used across LLMs.Protein specialists are MIMIC, ESM-C 600M and PGLM 1B; molecule specialists are MoLFormer-c3, ChemFM-1B and Uni-Mol2 1.1B.Source: §A.1 · p. 30
• Table 3 summarizes task supervision and reserved categories.Reserved questions may still have related information in description targets.Source: Table 3 · p. 31

A.2.1 Sources and object separation

• Protein and compound annotations come from reviewed UniProtKB/Swiss-Prot and three-star ChEBI, while Rhea supplies reaction relationships for cross-domain evaluation.Protein and molecule splits exclude related objects using sequence-identity and chemical-similarity criteria; overlap with model pretraining is unknown.Source: §A.2.1 · p. 31

A.2.2 Supervised and reserved targets

• Task partitions separate supervised and reserved targets, but withheld questions may remain answerable from descriptions or correlated annotations.Molecule roles are partitioned using ontology ancestry, and class labels include transitive is-a ancestry.Source: §A.2.2 · p. 31

A.2.3 Sampling and relational examples

• Training mixes single-object text, binary, relational and protein-variant examples; connectors are combined only at evaluation, not within training examples.Relational answers identify objects by prompt position, and some odd-one-out draws use hard negatives.Source: §A.2.3 · pp. 31–32

A.3 Question examples and prompt construction

• A compositional prompt grammar varies wording, slot placement and answer format across text, binary, relational and combined-representation questions.Worked examples use other training objects; reserved phrasings are used for evaluation.Source: §A.3 · pp. 32–33

A.4.1 Specialist read profiles

• Ridge probes compare specialist layers and candidate representations using property labels and, for proteins, variant probes; the profiles guide representation choice.The plotted protein and molecule curves primarily support within-specialist layer comparisons because task subsets differ across specialists.Source: §A.4.1 · p. 33, Figure 15 · p. 34, Figure 16 · p. 34

A.4.2 Standardization and the write operation

• Standardized specialist representations map through layer-specific affine maps to reference states; layer-selection curves use connector predictions and the LLM’s own reference states.Reference capture uses object descriptions and names, while names are not supplied when connecting a new object.Source: §A.4.2 · p. 34, §A.4.3 · pp. 34–35, §A.4.5 · p. 35

A.4.4 Name states as an alignment reference

• Name-state predictions gave better injected-question performance than mean description states in the tested comparison.The comparison used the same 9,017 fitting proteins and averaged five Yes/No questions over 200 held-out proteins.Source: §A.4.4 · p. 35, Figure 17 · p. 35

A.5.1 Objective and token weights

• Only connector parameters are trained with teacher-forced negative log likelihood; answer-token weights emphasize annotation content, while prompt tokens receive no loss.Main comparisons use the final checkpoint after 2,000 updates, without selecting an intermediate checkpoint by evaluation performance.Source: §A.5.1 · p. 36, §A.5.2 · p. 37

A.5.3 Training through the frozen LLM

• Kimi and Gemma use different training backends, but gradients pass through the frozen LLM without LLM weight gradients or optimizer states.Packed examples retain separate causal-attention boundaries and restart position indices.Source: §A.5.3 · p. 37

A.6 Applying and combining connectors

• Each slot retains its connector’s own representation definition, normalization, maps, write layers and mixing weights; multiple slots can supply representations without refitting connector parameters.At inference, the checkpoint’s LLM, specialist, extraction procedure, normalization and write schedule must match.Source: §A.6 · p. 37

• The evaluation tests whether the LLM uses the supplied object rather than merely the question and target.Source: B Evaluation methods and supporting… · p. 37
• It assesses transfer to new question types beyond supervised questions.Source: B Evaluation methods and supporting… · p. 37
• It examines whether independently trained connectors can combine information about the same protein.Source: B Evaluation methods and supporting… · p. 37
• It tests access to information beyond object names.Source: B Evaluation methods and supporting… · p. 37

• Matched comparisons use the same eligible objects and questions, while write-scan objects are excluded from fitting selection maps but are not held-out evaluation objects.Source: §B.1 · p. 37
• The own condition supplies the queried object’s embedding; shuffled substitutes another object’s embedding while retaining the question and target, and placeholder leaves the slot uninjected.Source: §B.1 · pp. 37–38
• For binary questions, the Yes-versus-No log-probability contrast yields task AUROC; a positive information-access gap means the correct embedding helps predict a reference answer, not that generated text is accurate.Source: §B.1 · p. 38
• Intervals use matched resampling, but rare labels can leave some resamples without both classes, so narrow AUROC intervals do not establish precise performance.Source: §B.1 · p. 38

B.2 Within-task generalization

• Reserved question wording produced no detected loss in the information-access gap for any protein connector, with paired confidence intervals including zero.Source: §B.2 · p. 38
• Displayed task groups use fixed memberships and equal-weight task AUROCs, summarizing correlated questions rather than independent information dimensions.Source: §B.2 · p. 38

B.3.1 Description exposure

• Low-exposure question types have estimated description-target exposure below 0.1%, while starred types reach 0.32%; these estimates omit other routes and do not establish complete blindness to tested properties.Source: §B.3.1 · pp. 38–39

B.3.2 Tasks and populations

• Transfer populations distinguish held-out molecule structural groups, defined at ECFP4 Tanimoto similarity 0.5, and use fixed thresholds of 329.29 Da for mass and 320 amino acids for protein length.Source: §B.3.2 · p. 39

B.4 Effect of training

• Figure 7 compares the same connector before and after task training on matched questions and objects, isolating the training-stage change but not individual supervision streams.Source: §B.4 · p. 39

B.5 Merging representations of one protein

• Figure 19 compares three independently connected protein specialists with the best single connector and best repeated-source triple on trained and unseen questions across 2,036 proteins; the exploratory comparisons did not meet the prespecified improvement criterion.Source: Figure 19 · p. 39, §B.5 · p. 39

B.6 Protein–compound relations

• Protein–compound tests use Rhea reaction participation and BindingDB binding questions without cross-domain question supervision; annotation-negative pairs are not verified biochemical non-participation.Source: Tasks and populations · p. 40
• Relational ordering compares annotated pairs with mismatched pairs sharing a protein or compound; an additive score from separate protein and compound contributions averages 0.5, so above-chance ordering requires a relationship between inputs.Source: Tasks and populations · p. 40

B.7 Information beyond names

• Name replacement can lose embedding information or use an incorrect generated name, while above-chance within-name concordance indicates distinctions unavailable from the shared name alone.Source: §B.7 · p. 40