PaperlayerView exact revision on arXiv

Source-linked AI summary

mimeo: Compiling Public Expert Corpora into Agent Skills and Testing What Transfers

Timothy Kassis

arXiv:2609.00453v1cs.AI

TL;DR

The paper asks whether expert files give agents knowledge access, recognizable personas, or transferred judgment. It presents mimeo, which compiles public expert work into source-carrying files, and finds clear grounding benefits but unresolved judgment transfer and distillation value.

  • Problem

    The paper separates whether an expert file improves knowledge access, produces a recognizable persona, or changes an agent’s decisions.

  • Method

    mimeo compiles a named person’s public sources into an inspectable SKILL.md or AGENTS.md file, and the study evaluates four expert profiles with one coding-agent harness and four models.

  • Results

    Grounding reduced documented-position misstatements, while knowledge access improved and judgment transfer remained unresolved because the relevant tests reached their ceiling.

  • Takeaways & Limitations

    The evidence supports mimeo as a compact, inspectable reference on a person, not as a demonstrated transfer of that person’s judgment.

  • Takeaways & Limitations

    The study covers one changing commercial agent model, one harness, four experts, four keyed tasks, and one built artifact per expert.

Abstract

from arXiv · show

Giving an agent a file about a named expert can supply hard-to-find material, produce a recognizable persona, or change what the agent decides. These are different claims. We test each one. mimeo is an open-source tool that finds a person's public work, checks each extracted quotation against the cached source text, and writes a file an agent can load. Eight logged builds averaged 38 model calls; the check rejects 13.2% of extracted quotations. We tested four expert files with one coding-agent harness. Knowledge access was clearest: mimeo answered all 20 obscure, quotation-heavy questions; no closed-book condition answered more than 10. Keyword search (BM25) over the same pages answered 15-17, a gap this sample cannot resolve. Grounding showed one clear benefit: personas written from model memory misstated a documented position on 1-4 of 20 answers under every grader; the plain agent and mimeo never did. Every persona was easy to spot on short open prompts, and adding task material lowered identification by 18-23 points. mimeo was no more identifiable than a from-memory profile. Judgment transfer remained unresolved because both tests hit their ceiling: every condition found 94-97% of the problems planted in engineering tasks and scored 94-100% on 16 new application scenarios. An AI-judged "sounds like the expert" score changed with the judge: two of four preferred answers based on a model's stereotype, while two found no difference on the same text. That is a caution against relying on a single AI judge. The evidence supports mimeo as a compact, inspectable reference on a person, not as a demonstrated transfer of their judgment. Toolkit and expert profiles: https://github.com/K-Dense-AI/mimeo

1 Introduction

mimeo tests whether a sourced expert file provides knowledge access, a recognizable persona, or transferred judgment—three distinct outcomes. The evidence supports improved access and grounding, but not judgment transfer or an advantage over source lookup.

  • Three outcomes are separated: knowledge access, recognizable persona, and judgment transfer.The paper treats these as different claims rather than assuming that quoting an expert implies adopting their decisions.
  • mimeo turns a name and public web sources into a source-carrying agent file, while warning that the result is not a verified account.The tool searches multiple source types, groups ideas, checks quotations against cached text, and revises the draft.
  • Corpus access improves recall of obscure wording, but BM25 retrieval recovers most of that gain and the static profile is not shown better than looking up sources when needed.The paper distinguishes access to public material from evidence that distillation itself improves performance.
  • Grounding prevents documented-position misstatements observed in personas written without sources, while mimeo does not demonstrate transfer of the expert’s judgment.The introduction reports this as the clearest grounding benefit and limits the broader conclusion to an inspectable reference.
  • The work asks whether sourced, multi-page expert profiles transfer anything beyond role labels or task-optimized procedures.This positions mimeo as a test of what transfers from a compiled public record.

2 The mimeo System

mimeo is a cached Python pipeline that compiles public material about a named person into an installable SKILL.md or AGENTS.md file. Its quotation check improves attestation but does not establish attribution or truth, and the released artifacts retain known limitations.

  • mimeo searches eight source categories, extracts labeled ideas and quotations, groups recurring themes, and writes SKILL.md, AGENTS.md, or both.The pipeline first identifies the person and fetches available text before drafting the agent-readable file.
  • Every stage is cached, so interrupted or changed runs repeat only work whose inputs changed.This makes the multi-stage build resumable and incremental.
  • The generated output is prose organized under headings for an agent to read as an instruction file.The examples use named expert headings, core principles, and anti-patterns.
  • Quotation matching removes spans that fail comparison with cached records, but a match makes a quotation attested rather than verified.Matching does not prove who spoke, that the surrounding claim follows, or that the record belongs to the correct person.
  • 13.2% of extracted spans were removed across eight logged reruns averaging 37.8 LLM calls and 7.1 minutes.Matching removed 41 of 311 extracted spans.

3 Results

Across six studies, corpus access most clearly improves obscure recall and grounding, while persona recognition weakens with task material. Judgment transfer remains unresolved because the behavioral tests largely hit their ceilings.

  • The evaluation uses one coding-agent harness, four expert profiles, four LLM graders or judges, and conditions spanning no skill, one-line persona, from-memory profile, and mimeo.Findings are treated as established only when every assigned model agrees.
  • Knowledge access: 35 of 38 E4 quotations appear in the cached corpus under mimeo, compared with 37 of 43 for runtime retrieval and 28–50% for closed-book conditions.These figures concern wording appearing in the cached corpus, not verified attribution or reasoning.
  • Knowledge access: mimeo answers 20/20 single-record questions, while no closed-book condition exceeds 10/20.The questions were drawn from the corpus used to build the mimeo file, so the design gives mimeo a ceiling advantage.
  • Knowledge access: BM25 answers 15–17 of the same questions, leaving mimeo ahead by three to five questions without a statistically separated advantage.The sample is too small to determine whether static distillation beats lookup when needed.
  • Grounding: Ungrounded personas misstate documented positions on 1–4 of 20 answers, whereas the plain agent and mimeo do so zero times under every grader.This is the clearest measure on which source grounding beats a profile written from model memory.
  • Recognizable persona: Every persona is identifiable above the 25% chance rate on short open prompts, while adding task evidence lowers identification by 18–23 points.The persona remains above chance, but task material competes with its signal.
  • Recognizable persona: On richer engineering tasks, mimeo identification is 25–31%, and every resampled interval includes the 25% chance rate.The result is a weak, unresolved persona signal in answers shaped by task materials.
  • Judgment transfer: Engineering-task issue coverage is 94–97% for every condition, leaving little room to observe a judgment-transfer gain.Equivalence tests rule out a ten-point coverage loss but cannot show whether profiles change harder decisions.

4 Discussion

The evidence separates corpus access, persona recognition, and judgment transfer: mimeo improves access to obscure material, but static distillation and judgment transfer remain unresolved. Grounding clearly reduces documented-position errors, while unkeyed persona judgments vary by evaluator.

  • Corpus access: Corpus access helps, but BM25 retrieval over the same records recovers most of mimeo’s gain, leaving distillation itself unresolved.The sample cannot determine whether a static profile beats looking up passages when needed.
  • Evaluation: “Sounds like the expert” judgments depend on the evaluator because the measure lacks an outside key.Two judges prefer stereotype-based answers, while two find no difference on the same text.
  • Future tests: Future tests should vary sources and use context-rich decisions that compare which cues experts and agents notice and when they stop.Recognizable voice alone would not validate judgment transfer.

5 Ethics and Limitations

The paper treats public expert profiles as partial, potentially outdated artifacts whose deployment raises attribution, consent, licensing, and scope concerns. Its evidence is limited to one changing model, one harness, four experts, and four keyed tasks.

  • Ethical risks: Public material is partial and dated, and distillation can freeze abandoned positions or turn conditional remarks into rules.The named person did not write, approve, or endorse the artifact, and cannot review, correct, or remove it.
  • Attribution and deployment: Quotation matching reduces one risk but does not establish who spoke or eliminate attribution errors.Fetched pages are untrusted input, and users remain responsible for source licenses and takedowns.
  • Scope limits: The study covers one changing commercial agent model, one harness, four experts, four keyed tasks, and one artifact per expert.The author wrote the tasks and keys, and the coverage and application tests sat at their ceiling.

6 Conclusion

mimeo converts public records about a named expert into a portable, source-carrying skill file, but the result is an inspectable reference rather than a verified copy of expert reasoning. The study separates recall and persona effects from the still-unestablished transfer of judgment.

  • Contribution: mimeo produces portable, source-carrying SKILL.md or AGENTS.md files from public expert records.The toolkit and profiles are open source.
  • Findings: The evidence supports corpus access and grounding benefits, but does not demonstrate judgment transfer.A static profile’s value beyond retrieval remains unresolved, and judgment-based scores vary by judge.
  • Evidence base: The conclusions draw on 732 agent runs across the reported experiments and judge repetitions.Main E4 and E6 grades, pairwise judgments, and identification judgments were repeated across four judges.

A Related Work

Prior work finds mixed effects from role prompts and agent skills, while mimeo studies a sourced, multi-page expert profile as a distinct intervention. Its design combines public-record grounding, quotation matching, and inspectable skill files, but the evaluation highlights unresolved quality and judge-stability questions.

  • Persona and role prompting: Prior persona studies report mixed effects, with some gains in reasoning or judged preference but no reliable accuracy improvement across broad evaluations.The cited literature includes studies finding useful reasoning and preferred expert-identity answers, alongside a 162-persona evaluation where most effects were indistinguishable from noise.
  • mimeo’s distinction: mimeo differs from short role labels by testing multi-page profiles built from attributed public records with quotations checked against cached source text.The evaluation includes a one-line persona as an explicit baseline and asks whether sourced profiles do anything that ungrounded personas do not.
  • Evaluation caveats: Expert-alignment judgments were unstable: two judges preferred one condition based on stereotype-like judgments, while two found no effect on the same text.No individual comparison survived Holm correction, and the authors do not treat any single AI judge’s score as ground truth.
  • Pipeline design: The tool produces SKILL.md or AGENTS.md through source discovery, idea grouping, quotation checking, and checklist-based revision, with cached stages and URL-safety controls.Its pipeline also resolves ambiguous names before discovery and treats fetched text as untrusted data.
  • Inspectability and limits: The resulting gallery bundles average 6,453 words and 113 source references per expert, but linked sources indicate bookkeeping rather than verified claim support.The paper explicitly distinguishes attested quotations from verified attribution or supported surrounding claims.
  • Pipeline design: Quotation matching rejected 13.2% of extracted spans, while critique-and-revise scores rose by +0.75 for SKILL.md and +1.25 for AGENTS.md on average.The revision stage scored lower than an earlier draft in 3 of 16 files, but the best-scoring draft was shipped.

D.2 Results: non-inferior coverage at the ceiling; quality unresolved

Across engineering coverage and quality comparisons, the conditions are difficult to distinguish because coverage is already near its ceiling, while pairwise quality remains unresolved. Deployment also introduces a bottleneck: the agent often does not load the skill.

  • Coverage: 94–97% of planted issues were found in every condition, leaving only 3–5 points for possible improvement.The primary and second graders agreed on 95.1% of 819 issue verdicts and 96.2% of simpler binary decisions.
  • Coverage: All six equivalence tests rule out differences larger than ten points, but this ceiling-limited non-inferiority does not show unchanged behavior on harder tasks.The practical interpretation is asymmetric because no skill already covers about 95–97% of the issue list.
  • Quality: mimeo’s pairwise vote share was .39–.53 against controls, with no comparison significant after Holm correction.The quality analysis had limited power: 80% conditional power required a vote share around .68–.72.
  • Deployment: The agent loaded the skill in only 5 of 16 on-demand runs, showing deployment adds a bottleneck beyond the file’s contents.Coverage was .979 for on-demand deployment versus .957 for the original always-on pass, but eleven runs never loaded the file.
  • Artifact safeguards: The editorial checklist produced the same self-critique score after clustering, and the behavioral value of clustering was not measured.Clustering reduced extracted items by 1.9×–2.5× while scores remained 7 versus 7.

E.1 E4: direct questions about the corpus

Direct questions show a clear open-book advantage for obscure, single-record material, but they do not establish that mimeo’s distilled file beats plain retrieval. Source grounding also prevents documented-position errors that appear in memory-written personas.

  • Recall: 20/20 single-record questions were answered by mimeo under every grader, while no closed-book control exceeded 10/20.The questions were selected from the same corpus used to build the mimeo file, so mimeo’s ceiling score is partly design-induced.
  • Grounding: Source-grounded mimeo and no-skill answers were never graded incorrect on the 20 tail questions, whereas memory-written personas were incorrect on 1–4 answers under every grader.The rubric combines contradictions with different plausible positions, but all four graders recorded the same persona pattern.
  • Recall: BM25 answered 15–17 questions against mimeo’s 20/20, but exact paired tests did not separate the conditions.The sample is too small to determine whether static distillation beats looking up passages when needed.
  • Familiarity: In the less-famous-expert arm, mimeo answered 17/18 questions versus 6–7 for the controls, although the comparison interval was wide.This arm had no runtime-retrieval condition and does not establish generalization beyond the three less-famous experts.
  • Persona visibility: Every persona was identifiable above the 25% chance rate on short prompts, and adding task materials reduced identification by .177–.229 on average.The packet did not erase the signal, and the experiment does not show that it eliminates steering.
  • Quality: Judged quality provided no reliable panel result, so the experiment does not support a claim that personas reduce advice quality.The primary judge preferred the no-skill answer, but the independent-task analyses were not significant.

F Extended Discussion

The discussion separates retrieval, persona recognition, and judgment transfer, finding clearer evidence for corpus access and grounding than for transferred expert judgment. Persona-alignment results also depend materially on the judge, while task materials weaken identification.

  • Artifact versus retrieval: A static artifact is portable and repeatedly loadable, whereas on-demand retrieval can provide fresher context but requires infrastructure and missed the source on four E4 questions.The evidence does not determine whether distillation is better than looking up passages when needed.
  • Judge-dependent alignment: Four judges score the same 112 answer pairs differently: two prefer stereotype-based answers, while two detect no difference.No judge significantly prefers mimeo, and the split does not establish caricature as its cause.
  • Judge-dependent alignment: Judge disagreement exceeds shuffled-label expectations, with Krippendorff’s α=.18 in E2 and ICC(2,1)=.50 across the reported alignment measurements.The evidence indicates that a single judge’s persona score should not stand in for a panel measurement.
  • Measurement implications: Measures with outside references produce more stable ordering than judged advice quality or expert alignment, which lack an external key.The paper therefore excludes the quality penalty from its conclusions and treats alignment as a measurement result.

G Ethics and Limitations in Full

The paper frames mimeo profiles as generated, incomplete representations rather than endorsements or verified reasoning, and documents ethical, security, identity, and evaluation limits. These boundaries constrain both deployment claims and generalization.

  • Representation and consent: A generated profile may preserve abandoned positions, turn conditional remarks into rules, or make summaries seem more authoritative than their sources.Public records are incomplete, dated, and uneven across languages, venues, and publishers.
  • Representation and consent: Quotation matching catches invented wording but does not establish attribution, source identity, or whether surrounding claims follow.The paper separates attestation from verification and recommends measuring citation support, completeness, and factual precision separately.
  • Grounding and safety: Ungrounded personas misstate documented positions on 1–4 of 20 probes under every grader, whereas no-skill and mimeo do so zero times.The probe category includes different plausible positions as well as direct contradictions, and the test is narrow.
  • Grounding and safety: The released toolkit provides no process for named people to review, correct, or remove profiles.The paper calls this an unresolved deployment defect and recommends contact, response-time, correction-history, and prompt-removal procedures.
  • Grounding and safety: Fetched pages remain capable of influencing extraction or authoring despite data markers and URL restrictions.Because no adversarial prompt-injection benchmark was run, deployment should avoid secrets and sensitive-repository write access.
  • Evaluation scope: Behavioral evidence uses one commercial agent model and one coding-agent harness, whose underlying weights may change under vendor aliases.Future releases should include execution dates, CLI versions, prompts, and provider-returned model metadata.
  • Evaluation scope: The study includes only four experts and four keyed tasks, so its resulting intervals remain estimates over a very small task set.The design avoids pseudoreplication, but adding tasks or experts increases judging demands.
  • Evaluation scope: All four human raters lack expertise across the studied people, so the artifacts are not described as digital twins, clones, or verified expert reasoning.Evaluating judgment in context requires source audits and scenario ratings from people who know the work.

H Evaluation Task Suite

The evaluation suite tests agent judgment across eight engineering tasks spanning review, debugging, design, refactoring, APIs, performance, and incident analysis. Several tasks contain planted issues or weaknesses whose known counts support keyed grading.

  • Task coverage: Eight tasks cover code review, architecture critique, debugging, refactoring, experiment design, API review, performance investigation, and incident postmortem.The prompts target distinct engineering decisions rather than naming an expert.
  • Task coverage: Four keyed tasks contain 21 planted issues per expert: 8 in code review, 3 in debugging, 6 in experiment design, and 4 in performance investigation.Agents and judges never see the issue lists used for grading.
  • Keyed tasks: The code-review task embeds eight issues involving normalization leakage, augmentation aliasing, graph-retaining accumulation, validation gradients, scheduling, warmup, fp16 scaling, and seeding.It uses an approximately 120-line PyTorch training script.
  • Open-ended engineering tasks: The architecture task critiques a 50-user internal RAG tool with seven microservices, three databases, and an agent swarm, alongside ten planted flaws.The requested output includes a counter-proposal.
  • Open-ended engineering tasks: The debugging task combines a plateauing, spiking, diverging training run with three planted causes, four red herrings, and a confounded baseline.The task follows learning-rate decay.
  • Open-ended engineering tasks: The experiment-design task tests an optimizer claim against six weaknesses, including test-set tuning, one seed, narrow workload, omitted costs, and a schedule artifact.The comparison also pits a tuned method against a default baseline.

I Open-Ended Task Suite (E5)

E5 uses six open-ended advisory prompts without expert names or supporting materials, then E5b repeats them with task-specific and organizational evidence. The setup tests whether added context changes persona identification.

  • E5 prompts: Six E5 prompts require firm choices about research agendas, startups, course design, hiring, AI open problems, and PhD advice.The scenarios provide no supporting materials and do not name an expert.
  • E5b intervention: E5b repeats the six requests and adds 1,748–1,970 characters of task-specific evidence plus 5,105 characters of shared organizational record.The added blocks contain constraints, measurements, and competing considerations without expert names or quotations.
  • E5b intervention: Every E5/E5b pair keeps the task, condition, and expert fixed; only the evidence packet changes.This isolates the context intervention within each paired comparison.
  • Deployment conditions: The sandbox loads persona content through CLAUDE.md at session start, while the no-skill condition writes no file.mimeo’s AGENTS.md output is adapted for this deployment.
  • Deployment conditions: The on-demand arm installs SKILL.md and references under .claude/skills and records whether the agent invokes that skill.Skill loading is an observed outcome rather than a fixed setup condition.
  • Persona conditions: The one-line persona condition explicitly instructs the agent to approach every task as Andrej Karpathy would.It identifies Karpathy through his research, education, Tesla, and OpenAI roles.
  • Persona conditions: From-memory profiles match mimeo’s broad format and approximate length, but the comparison does not isolate grounding as a single factor.The authoring model had no sources in front of it when writing the from-memory profile.

L Statistical Procedures

The paper uses paired, resampling-based tests and agreement measures to evaluate coverage, identification, pairwise preferences, and judge consistency. It explicitly accounts for ceiling effects, judge dependence, multiplicity, and data-quality exceptions.

  • Blind identification: The repeated identification analysis shuffles four names, forces a choice, and estimates accuracy intervals by resampling task means.Chance is 1/4; the repeated E2 analysis contains 48 answers per persona condition.
  • Multiplicity: Holm–Bonferroni adjusts the six related pairwise comparisons within each experiment to control at least one false positive.The paper does not correct across experiments or measures.
  • Equivalence and power: TOST tests whether coverage differences stay within 0.10 of the answer key, rather than whether they differ from zero.The margin corresponds to 0.3–0.8 issues across the four task keys.
  • Judge panel: Four judges independently score the same answer pairs, and judge differences are tested by shuffling judge labels within paired observations.The analysis uses 20,000 permutations and paired comparisons because judges evaluate identical pairs.
  • Judge panel: p=.0032 and .0016 show that the judges differ, with three of six paired mean-difference intervals excluding zero.The reported differences include −.143, +.107, and +.103 for specific judge comparisons.
  • Agreement: 94.9% exact agreement on 273 original issue verdicts and 95.1% across 819 verdicts support high grader agreement in the coverage analyses.Agreement on found-versus-not decisions was 95.6% originally and 96.2% across all generations.

N Reproducibility

The released artifact includes the mimeo toolkit and expert profiles, while experiment materials remain unreleased. Reproducibility is supported by cached outputs and deterministic offline analysis, but reruns can change as model providers update their systems.

  • Released artifact: The public release contains the mimeo repository, toolkit, and expert profiles, but not experiment runs, judgments, prompts, keys, scripts, or telemetry patches.The 22 profiles analyzed in Appendix C are committed with the pinned toolkit release.
  • Resumability: Runners skip existing result files, and pipeline stages cache outputs using fingerprints of inputs, prompts, and output schemas.These mechanisms allow interrupted long runs to resume and avoid repeating unchanged stages.
  • Determinism: Agent and judge calls are nondeterministic, but every figure and table is regenerated deterministically from a fixed results tree with a fixed bootstrap seed.Individual verdicts therefore need not reproduce across reruns even when offline analyses do.
  • Expert profile example: The released expert profile excerpt presents Karpathy’s hacker’s perspective as favoring code and physical intuitions over dense mathematics.Its recommendations include building systems from scratch and verifying AI outputs in small concrete chunks.
Loading 2609.00453v1…