Building AI Research Teams for Autonomous Discovery of Linear Attention Architectures

• The team’s rationale is that confined exploration can create local minima, while complementary disciplines may bring mechanisms from one field to limitations recognized in another.The introduction cites fast-weight learning as one source of ideas for linear-attention memory.Source: §1 · pp. 1–2
• The research question is how researchers with different backgrounds can build on one another’s findings and sustain progress under a limited experimental budget.Costly experiments constrain how many ideas can be tested, requiring a balance between exploration and exploitation.Source: §1 · p. 2
• The proposed pipeline combines persistent researchers with distinct disciplinary profiles, private histories, shared experimental evidence, and adaptive allocation of research opportunities.Researchers propose, review, implement, and evaluate ideas, then reflect on outcomes.Source: §1 · pp. 2–3
• At 1.3B parameters, CFDN improves over GDN2 in every evaluation category.The categories include language modeling, common-sense reasoning, and synthetic and real-world retrieval.Source: Figure 1 · p. 2, §1 · p. 3
• Controlled comparisons find that disciplinary profiles broaden explored mechanisms, private histories support refinement beyond shared evidence, and adaptive selection beats uniform selection within the same iterations.Optimizing solely for language modeling transfers less consistently across evaluations.Source: §1 · p. 3

2 METHODOLOGY

• The overall design uses researchers with complementary perspectives to explore a shared problem and develop ideas through accumulated experience and experimental feedback.Source: §2 · p. 3, §2.1 · p. 3

2.1 SCIENTIFIC PERSPECTIVES AND ACCUMULATED KNOWLEDGE

• Researchers receive disciplinary profiles with prior knowledge, scientific insights, and research habits, while private histories preserve hypotheses, reasoning, and outcome reflections.A shared record of proposals, reviews, results, and conclusions lets researchers extend or combine one another’s findings.Source: §2.1 · pp. 3–4

2.2 AN ITERATIVE PROCESS FOR DEVELOPING AND TESTING IDEAS

• Each iteration selects one researcher, develops and reviews a proposal, refines and tests approved designs, then records reflections and shares lessons.Discounted UCB selection balances historical rewards with opportunities for less-tested researchers, while recent contributions receive more weight.Source: §2.2 · pp. 4–5

2.3 EVALUATING ARCHITECTURES AND REWARDING RESEARCH PROGRESS

• Progress is measured through language-modeling loss, common-sense accuracy, and retrieval performance, with lower loss and higher accuracy or retrieval indicating improvement.Researcher reward is the nonnegative gain in model utility over the best architecture known at the iteration’s start; failed or rejected proposals receive zero.Source: §2.3 · pp. 5–6

3.1 EXPERIMENTAL SETTING

• The search used 16 researchers over 80 iterations; each researcher was invoked exactly once in the first 16 iterations, and subsequent researcher selection followed the UCB rule.Candidates were searched at approximately 150M non-embedding parameters and trained from scratch on FineWeb-Edu with training tokens adjusted to match GDN’s total FLOPs. Search feedback used LM loss, 256 examples from each CSR dataset, and 64 examples per NIAH task at 1K, 2K, and 4K context; RWR was not evaluated during search.Source: §3.1 · p. 6

3.2 DISCOVERED ARCHITECTURE AND VALIDATION AT LARGER SCALE

• At 150M, CFDN improves all three search targets over GDN2: LM perplexity falls from 12.21 to 11.90, CSR accuracy rises from 44.48% to 44.63%, and retrieval rises from 28.91% to 50.00%.In the full evaluation, it outperforms all baselines on both language-modeling tasks and more considerably on NIAH, while its CSR performance is slightly below GDN2 but above GDN and KDA.Source: §3.2 · p. 7, Table 1 · p. 7
• At 1.3B, CFDN achieves the best performance among the compared models in language modeling, common-sense reasoning, synthetic retrieval, and real-world retrieval.The full evaluation reports the lowest perplexity on both language-modeling datasets and the highest mean accuracy in the other categories.Source: §3.2 · p. 7, Table 2 · p. 7

3.3 WHICH ORGANIZATIONAL CHOICES SUPPORT DISCOVERY?

• The organizational comparisons remove disciplinary backgrounds, private histories, or adaptive selection while retaining shared experimental evidence and holding the agent model and experimental budget fixed.Fresh-memory and uniform-selection runs reuse the main run’s first 16 iterations before conducting 64 further iterations.Source: §3.3 · p. 7
• The full system reaches utility 1.04, versus 0.54 without backgrounds, 0.56 with fresh memory, and 0.76 with uniform selection.The main run evaluated 63 candidates, compared with 65–68 in the variants; its strongest proposal appeared at iteration 68, while the no-background run peaked at iteration 22 and did not improve thereafter.Source: §3.3 · p. 8, Figure 3 · p. 8

3.4 HOW DOES THE SEARCH OBJECTIVE STEER DISCOVERIES?

• LM-only search yields lower LAMBADA perplexity, while three-target search performs better on the other metrics, showing that objective choice affects which capabilities the search favors.The comparison holds researcher profiles, private memory, adaptive selection, and experimental budget fixed; LM-only excludes CSR and NIAH from its objective.Source: §3.4 · p. 8, Table 3 · p. 8

3.5 HOW RESEARCHERS DEVELOP THE DISCOVERED ARCHITECTURE

• Four researchers develop CFDN by adding a recurrent context vector to GDN2’s matrix memory, learning its address origin, and normalizing the correction by translated-address energy.The vector supplies a query-independent contribution, while the matrix provides a query-dependent read.Source: §3.5 · p. 8
• Updating the matrix before the context vector improves LM perplexity but weakens CSR and retrieval; a later revision updates shared context first and produces CFDN.The researchers use the degraded result to reconsider which memory should absorb prediction error first.Source: §3.5 · p. 8

Autonomous research agents

• Prior autonomous-research systems use fixed pipelines, agentic tree search, or evolutionary search, with a single agent or fixed pipeline proposing every candidate.Examples include Lu et al. (2024), Yamada et al. (2025), and Novikov et al. (2025).Source: §4 · p. 9

Research teams

• Research teams in prior work vary in member backgrounds, retained histories, and shared records, but do not test whether private memory adds value once findings are shared or compare backgrounds by implemented experimental outcomes.The cited systems include VirSci, ASCollab, ScienceClaw, CORAL, and AgentRxiv.Source: §4 · p. 9

Adaptive allocation

• Prior adaptive-allocation systems choose an operator, candidate, branch, or acting agent, whereas this work adaptively chooses which researcher proposes and tests the next idea.The paper frames that choice as consequential because each researcher brings a distinct background and record of prior work.Source: §4 · p. 9

Architecture discovery

• Existing architecture-discovery systems use fixed-role workflows or independent searches; this work fixes the experimental budget and varies team organization and the search objective.The related systems include NADER, AIRA-Compose, AIRA-Design, and ASI-Arch.Source: §4 · p. 9

5 CONCLUSION

• The framework combines diverse scientific backgrounds, persistent memory, shared evidence, and adaptive allocation; applied to linear attention, it discovers CFDN, which improves over GDN2 at two scales.Organizational comparisons and research trajectories highlight heterogeneous agents, adaptive allocation, and shared experimental records.Source: §5 · p. 9

Model configurations

• Model configurations describe GDN2’s recurrent-head geometry, with the outer backbone fixed within each scale; scale labels exclude trainable input embeddings and output vocabulary projections.The output projection is included in reference FLOPs.Source: Table 4 · p. 14

Training hyperparameters

• All architectures share optimizer settings and global batch size, while microbatch size denotes sequences processed per device in one forward/backward pass.Source: Table 5 · p. 14

Common small-scale data stream

• The common small-scale data stream packs tokenized FineWeb-Edu documents into 4,097-ID blocks, yielding 4,096 input–target positions per sequence, and candidates use the same saved sequence order.Each optimizer update advances by 512 sequences.Source: A TRAINING AND COMPUTE PROTOCOL · p. 14

Baseline parameterization

• GDN2 uses full query, key, value, erase, and write projections with causal depthwise convolutions and SiLU; KDA shares the surrounding components but uses scalar per-head erase/write gates.The comparison confines architectural differences to recurrent mechanisms within a common surrounding backbone.Source: A TRAINING AND COMPUTE PROTOCOL · p. 14

A.1 ARCHITECTURAL REFERENCE FLOPS AND PRACTICAL CONSTRAINTS

• Training tokens are set by the compute budget divided by reference FLOPs per token, rounded down to complete updates.The rounding keeps each candidate within its nominal budget.Source: §A.1 · p. 15
• Custom-operation forward counts are multiplied by three as a uniform training-cost approximation, while repeated shared projections incur additional charges.Embedding lookup is excluded and the language-model output projection is included.Source: §A.1 · p. 15
• The reference-FLOPs rule counts projections and memory computation while excluding ordinary token-local processing, optimizer steps, communication, and checkpointing.The passage characterizes this as a consistent architectural proxy, not a hardware instruction count.Source: §A.1 · p. 15
• The projection term uses an audited weight inventory, while custom-operation terms retain counts approved in scientific review.A proposer derives the breakdown, a technical reviewer checks it, and automated checks measure implemented weights; finite training and parameter counting alone do not prove faithful implementation.Source: §A.1 · p. 15
• 6,453 updates for GDN2 and 6,452 for CFDN are allocated under the small GDN compute anchor.Their reference costs are 1,150,353,408 and 1,150,500,864 FLOPs/token, respectively.Source: §A.1 · p. 15
• Analytical compute is supplemented by same-hardware speed checks, and implementation repairs are limited to three validation rounds and six active hours.Candidates must meet 1.2× baseline step-time and elapsed-time thresholds at four training milestones.Source: §A.1 · p. 15

B EVALUATION DEFINITIONS AND COMPLETE RESULT TABLES

• Search-scale language-modeling perplexity averages token-mean negative log-likelihoods from WikiText-2 and LAMBADA equally before exponentiation.WikiText-2 uses rolling 4K windows; LAMBADA scores final-word continuations.Source: B EVALUATION DEFINITIONS AND COMPLETE… · p. 15
• The 1.3B real-world retrieval evaluation covers six datasets and reports their unweighted mean accuracy; this suite is reserved for final evaluation, not search feedback.The six datasets are SWDE, SQuAD, FDA, TriviaQA, Natural Questions, and DROP.Source: B EVALUATION DEFINITIONS AND COMPLETE… · p. 16
• Tables 6–7 report per-dataset search-scale LM and CSR measurements used to determine Table 1 utilities; Tables 8–9 give complete 150M LM/CSR and generated-answer NIAH evaluations.Each Table 9 cell contains 500 frozen requests, and the mean weights all 14 cells equally.Source: B EVALUATION DEFINITIONS AND COMPLETE… · p. 16, Table 6 · p. 16, Table 8 · p. 17
• Search-scale retrieval feedback uses 64 examples for each of four NIAH tasks at 1K, 2K, and 4K contexts, with exact teacher-forced continuation scoring.The 768 requests are summarized equally across 12 task/length cells.Source: B EVALUATION DEFINITIONS AND COMPLETE… · p. 15, Table 7 · p. 16
• Full NIAH evaluation uses generated answers for 500 shared frozen requests in each of 14 cells, with equal weighting across cells.This differs from the search-time teacher-forced proxy.Source: B EVALUATION DEFINITIONS AND COMPLETE… · pp. 15–16, Table 11 · p. 18

C.1 SCIENTIFIC PROFILES AND RESEARCHER INITIALIZATION

• The team uses 16 fixed disciplinary profiles, each a roughly 2,000–2,300-word document combining foundations, design judgments, diagnostic practices, and references.Only the focal linear-attention profile is written specifically for the target architecture family.Source: §C.1 · p. 19
• Researchers retain private conversations and notebooks, then receive summaries and access to shared records at new assignments, allowing both initial profiles and accumulated evidence to inform subsequent judgments.The common starting materials include the profile, problem statement, and shared references.Source: §C.1 · p. 19
• The profile table summarizes each perspective’s organizing question, not a prescribed candidate mechanism or exclusive research territory..Source: Table 13 · p. 19

C.2 RESEARCH PROTOCOL AND ANALYSIS DETAILS

• Proposals specify a hypothesis, expected benefit, falsifying outcome, architecture details, training formulation, and analytical compute breakdown before technical and contextual review.Reviewers are blind to the author and one another’s review; the technical reviewer lacks shared history, while the contextual reviewer uses it to assess duplication and research value.Source: §C.2 · p. 20
• Persistent-memory researchers retain their conversations and notebooks across iterations, whereas fresh-memory researchers retain them only within one iteration despite access to the same shared evidence.A director consolidates shared lessons every ten completed reflections but does not select the next researcher.Source: §C.2 · p. 20
• Adaptive allocation discounts an observation’s weight by half after 32 completed reflections, excludes researchers with unresolved investigations, and uses stable ordering to break ties.Uniform selection follows the first-round order and waits when the next researcher is busy.Source: §C.2 · p. 20
• Fresh-memory and uniform-selection runs branch from the same first 16 investigations and each conduct 64 further iterations with access to common shared evidence.The branch preserves identical proposals, reviews, specifications, implementation evidence, and outcomes.Source: §C.2 · p. 20
• Parent–child utility comparisons include only earlier candidate parents and complete evaluations, while cross-researcher link counts include proposals without completed evaluations.The denominators therefore differ, and the counts describe declared relationships rather than proof of knowledge transfer.Source: §C.2 · p. 20
• Table 14 reports completed evaluations and declared parent relationships across 80-iteration rows, with branch rows including the shared first 16 iterations and 11 completed evaluations.Fractions give the number meeting a criterion over the number eligible.Source: Table 14 · p. 20

C.3 AN EXACT RESEARCHER PROFILE

• Filtering alternates prediction and correction, while the posterior representation determines how uncertainty and new evidence are handled.The exact recursion is an identity; the difficulty lies in representing and updating the posterior.Source: §C.3 · p. 21
• A normalized innovation compares residuals with predicted uncertainty; persistent structure in those innovations can signal model misspecification.The normalized innovation squared has a chi-squared reference distribution under the assumed model, while whiteness tests assess temporal dependence.Source: §C.3 · p. 22
• Large innovations can reflect corruption, latent motion, regime change, or overconfidence, and each explanation implies a different update.The profile distinguishes isolated spikes, coherent cross-measurement innovations, and sustained changes in normalized innovations.Source: §C.3 · p. 23
• Bounded inference requires pruning, merging, or approximating hypotheses, choices that determine which alternatives survive.Hard winner-take-all truncation can discard low-probability hypotheses whose futures differ or whose recovery would be difficult.Source: §C.3 · pp. 24, 27
• Change-point detection maintains uncertainty over regime duration, but bounded implementations must approximate the growing set of run-length hypotheses.A constant hazard implies a geometric regime duration; retaining only the most probable run length creates brittle winner-take-all segmentation.Source: §C.3 · p. 24
• Estimator comparisons should fix decision latency and assess calibration and consistency, not point error alone.The workflow also checks coverage, innovation whiteness, particle diversity, and recovery after mismatch.Source: §C.3 · pp. 25–26

D.1 THE DISCOVERED CONTEXT-FIRST RECURRENCE

• CFDN adds a recurrent context vector to GDN2’s matrix memory, creating shared context and address-dependent contributions to each readout.The context term is equal across queries; the matrix term varies with displacement from a learned origin.Source: §D.1 · pp. 27–28
• CFDN corrects the context vector first, then updates the matrix with the prediction error that remains.The matrix therefore fits corrections beyond what shared context already explains.Source: §D.1 · pp. 27–28
• The joint memory corrections can be combined into a single rank-one update of the augmented memory, retaining parallel chunkwise training compatibility.The update sequence recomputes the error after the context correction before updating the matrix.Source: §D.1 · p. 28
• CFDN learns a reference address for matrix reads and writes, plus a scalar that controls the relative weighting of matrix and context coordinates.Translation changes address energy, so the corrections are scaled to compensate; the scale is one when the learned origin is zero.Source: §D.1 · p. 28
• At the learned origin, the matrix contribution to the readout vanishes, while the context contribution remains query-independent.Passive matrix decay alone leaves that readout unchanged, though later matrix corrections can change it.Source: §D.1 · p. 28