Knowledge Graphs as Files: An Agent Harness Where Tool Creation and Tool Use Co-Evolve

• Predefined interfaces constrain the evidence agents can inspect, and batching calls still inherits each API’s evidence limits.Source: §1 · pp. 1–2
• DKGI exposes complete node and edge files so agents can write Python to follow relations, combine node sets, and inspect textual attributes.Agents can interleave code execution with shell search and record inspection, revising code as evidence becomes available.Source: §1 · p. 2
• Execution feedback guides agents’ code revisions, while the revised access code determines what evidence subsequent searches gather.Source: §1 · p. 2
• By at least 6.7 Hit@1 points on every STaRK dataset, training-free DKGI exceeds the strongest published baseline even in the minimal Pi harness.Source: §1 · p. 2
• Controlled ablations and trajectory analyses show benefits from interleaving tool creation and use and characterize how they co-evolve during agent–KG interactions.Source: §1 · p. 3

2.1 Overview and Notation

• DKGI uses a ReAct-style loop in which query reasoning and prior observations guide actions, with shell tools and Python code for graph access.The agent follows relations, intersects node sets, and filters textual attributes directly over graph files.Source: §2.1 · p. 3

2.2 Knowledge Graphs as Files

• The graph export stores node data, per-relation edge lists, and a schema while preserving access to the complete graph.The file layout supports text search and Python extraction of fields and relation endpoints.Source: §2.2 · p. 3

2.3 Graph Operations Written by the Agent

• Agent-written Python composes scans, lookups, traversals, set operations, and ranking, then retrieves selected abstracts for relevance assessment.These operations are generated for the current request rather than supplied as predefined harness functions.Source: §2.3 · pp. 3–4

2.4 Large Outputs as Files

• Shell outputs are saved in full, while head-and-tail display bounds context use and later code can search or filter the saved results.Access code can also return compact candidate lists or selected text fields for subsequent observations.Source: §2.4 · p. 4

2.5 Co-Evolving Toolbox

• Execution evidence guides code reuse, revision, or extended traversal, and revised access code determines what later executions examine.The process develops access code within a task without model training or a tool library learned across tasks.Source: §2.5 · p. 4

2.6 Implementation

• DKGI is instantiated in minimal Pi and feature-rich Codex CLI, with both providing read-only graph files, schema, and writable scratch space.Pi supports controlled ablations, while Codex tests the design with richer runtime support.Source: §2.6 · p. 4

3.2 Main Results

• With GPT-6-Luna in Pi, DKGI exceeds the best training-free baseline in H@1 by 9.4 points on MAG and 13.2 on Prime.It is the strongest training-free method on MAG and Prime; GPT-6-Astra achieves the highest H@1 and MRR on every dataset.Source: §3.2 · p. 5
• With GPT-6-Astra, DKGI exceeds trained GRASP in H@1 by 8.2, 9.0, and 6.7 points on Amazon, MAG, and Prime, respectively.Overall H@1 rises from 71.5 with GPT-6-Luna to 82.1 with GPT-6-Astra; Table 2 reports results on the full 7,108-query test sets.Source: Table 2 · p. 5, §3.2 · p. 6

3.3 Ablation Study

• On the 710-query ablation subset, Full DKGI reaches 75.2 overall H@1, 2.4 points above Graph tools, while shell-only access scores 68.3.Adding Python over several turns to graph-file access improves overall H@1 by 6.9 points, R@20 by 8.5, and MRR by 6.9 versus Shell only.Source: §3.3 · pp. 6–7, Table 3 · p. 6

3.4 How Tool Creation and Tool Use Co-Evolve

• Python makes up 83.3% of calls in the final 20% of trajectories, and 75.0% of programs contain a full node scan.Agents average 4.0 programs per query; 35.5% contain one-hop traversal, and 42.3% contain scoring or ranking, multi-hop traversal, or set operations.Source: §3.4 · p. 7
• Of all programs, 48.4% reuse an ID from an earlier output and 36.5% revise an earlier program; after a failed program, the next revises it in 88.6% of cases.A single-program-per-query variant removes this iterative process and recovers about a tenth of the Python gain on graph files.Source: §3.4 · pp. 7–8
• On queries where graph-tool agents call neighbors at least ten times, Full DKGI leads Graph tools by 9.0 H@1 points and Graph tools + code by 6.0.Those 100 queries are all from MAG and Prime; 85.0% involve programs with scoring or ranking, multi-hop traversal, or set operations.Source: §3.4 · p. 8

With a stronger LLM, the agent reuses earlier

• With GPT-6-Astra, Python appears earlier in trajectories, 59.9% of programs reuse an earlier-output ID, and 92.2% of next programs revise a failed one.Among incorrectly answered queries, gold nodes are returned below first position in 58%, 46%, and 55% on Amazon, MAG, and Prime, respectively.Source: With a stronger LLM, the agent reuses… · p. 8

Related Work

• KG access approaches include reasoning over retrieved paths or subgraphs, predefined graph functions, and code that composes interfaces or processes retrieved graph objects.Examples include semantic parsing, KG-Agent’s tool-calling programs, Code-on-Graph’s subgraph processing, and AvaTaR’s composition of predefined functions.Source: §4 · p. 8
• Outside KGs, agents create executable tools and refine code with execution feedback; SMITH jointly trains tool creation and use and reuses tools across tasks.DKGI instead creates and revises tools within a task without training.Source: §4 · p. 8
• For entity search over typed, multirelational KGs, DKGI is presented as the first harness letting agents implement graph operations over complete node and edge files.Graph-as-Code iteratively processes graph data for node classification, while GRA explores hybrid KGs through predefined Unix-style tools.Source: §4 · p. 8

• DKGI with GPT-6-Astra achieves the highest H@1 and MRR on every STaRK dataset.The evaluation is training-free.Source: §5 · p. 8
• With GPT-6-Luna, DKGI achieves higher overall H@1 than predefined graph tools, whether called directly or composed in code.Source: §5 · p. 8
• Multi-turn program writing and revision yield the greatest gains, while trajectories show tool creation and use co-evolving within tasks.Source: §5 · p. 8

Scope

• The evaluation covers only STaRK’s three text-rich graphs, with at most 10 entity types, 18 relation types, 1.9M entities, and 39.8M edges.Source: Limitations · p. 9
• WebQSP, CWQ, GrailQA, Freebase- or Wikidata-scale schemas, aggregation questions, and DKGI’s scaling with graph size are left to future work.Source: Limitations · p. 9
• For graphs too large to scan or questions requiring exact aggregation, purpose-built indexes, graph databases, and query languages may be more suitable.Source: Limitations · p. 9

Models, baselines and data

• The main results use proprietary GPT-6-Luna and GPT-6-Astra models accessed by API in September and October 2026; exact reproduction depends on their continued availability, and open-weight models remain untested.Source: Limitations · p. 9
• Baseline numbers come from literature runs with different backbones, budgets, and evaluation code; the evaluation uses STaRK queries, most of which are synthetic.Source: Limitations · p. 9

Protocol and deployment

• Because the agent can execute arbitrary shell commands, deployment requires a sandbox such as the authors’ setup.Source: Limitations · p. 9

Ethical Considerations

• The study uses public STaRK data and released artifacts under their stated licenses, and runs arbitrary-shell agents in a sandbox that protects graph files and blocks network and label access.Reported point estimates come from one evaluation per configuration and query set.Source: Ethical Considerations · p. 9

A Knowledge Graph Export

• The export preserves all nodes, attributes, stored edge rows, and type labels; candidate IDs define valid outputs, while noncandidate nodes can serve as evidence.The export is checked for complete, ordered node files and sampled records matching the source.Source: A Knowledge Graph Export · p. 14

B Prompts and Prompt Validation

• Prompt variants change ranking or tool instructions, including shell-only, one-program, no-edges, and predefined-tool conditions.The final prompt asks agents to check the top candidate against every query requirement and rank further plausible candidates afterward.Source: §B.1 · p. 15

B.1 Prompt Texts

• Agents are instructed to use local node and edge files with shell tools or Python, inspect the format and schema, and return ranked candidate IDs supported by graph evidence.Source: §B.1 · pp. 15–16

B.2 Prompt Validation

• Prompt decisions were made on validation data before test results; on 750 further validation queries, R@20 rose from 61.9 to 71.3 with Pi and from 57.6 to 68.8 with Codex, while H@1 changed by −2.3 and −0.8 points, respectively.Source: §B.2 · pp. 16–17

• Full DKGI runs with Pi or Codex, using the same model, reasoning effort, and context window; programs are written inline and discarded after each query.Codex uses its own tool and system prompt, with web search and other optional features disabled.Source: C Harness Details · p. 17
• Graph-tool variants expose schema, search, node lookup, and neighborhood tools; Retrieval only removes the neighbors tool.Graph tools + code lets the agent call the same tools from Python while retaining read and bash access.Source: C Harness Details · p. 18, Table 5 · p. 18
• Shell only restricts executable programs to 15 listed shell tools and bars Python; One program allows a single Python start per attempt.In the test subset, 573 of 710 One program queries used that single invocation.Source: C Harness Details · p. 18
• H@1 scores whether the first entry is a gold answer, R@20 measures gold-answer recall in the first 20, and MRR uses the first gold answer’s rank among the first 100.Failed attempts count as misses; all attempts for one Amazon One program query reached the time limit.Source: C Harness Details · p. 19
• The full GPT-6-Astra DKGI test run cost about $3,000; reported costs use recorded token usage, which is missing for some failed attempts.Source: C Harness Details · p. 19

D Harness Comparison on the Test Subset

• Table 7 compares DKGI harnesses on the 710-query test subset; Full DKGI results come from full-test runs, while Codex with GPT-6-Astra uses a separate test-subset run.The rows use the same prompt and time limit per attempt; Table 6 reports cost and resource use per query, with cost as a mean and other columns as medians.Source: D Harness Comparison on the Test Subset · p. 19, Table 6 · p. 19, Table 7 · p. 20

E Compared Baselines

• The baseline descriptions specify each system’s KG access, STaRK training or tuning, named backbone models, and reported configuration.The appendix presents the systems in the order and grouping of Table 2.Source: E Compared Baselines · p. 19

E.1 Trained or Tuned on STaRK

• AvaTaR scores candidate nodes using predefined knowledge-base functions, while a comparator LLM writes optimized instructions from contrasting training-query performance.The reported configuration uses Claude 3 Opus.Source: §E.1 · p. 19
• PASemiQA fine-tunes a planner to predict shortest relation paths, then uses a GPT-4 agent to expand relevant neighbors, retrieve by embedding similarity, and return answers.The reported H@1 and MRR are taken from the paper’s Tables 2 and 10.Source: §E.1 · pp. 19–20
• mFAR trains an encoder and query-conditioned weights on STaRK queries; MoR combines a fine-tuned planner, layer-wise traversal, textual matching, and a trained reranker.GraphRAFT uses a Neo4j pipeline with schema-constrained Cypher generation; its two models are fine-tuned on MAG and Prime.Source: §E.1 · p. 20
• GRASP plans and queries Neo4j, fuses rescored candidates with mFAR, then reranks them; its fusion weights are validation-tuned and its Prime prompt includes 19 training-query plans.The reported system is the supervised GRASP configuration.Source: §E.1 · p. 20

E.2 Training-Free

• BM25, ColBERTv2, and GritLM-7B rank candidate-node documents using lexical matching, token-level embedding matches, and document-query embedding similarity, respectively.Their reported results are from the full test sets in the STaRK paper.Source: §E.2 · p. 20
• KAR expands queries from retrieved entity-neighbor triples, while HybGRAG routes queries between text and hybrid retrieval and iteratively refines routing with feedback.KAR uses up to two-hop neighbors; HybGRAG’s reported configuration uses Claude 3 Sonnet and ada-002 embeddings.Source: §E.2 · p. 21
• AF-Retriever grounds parsed Cypher constraints with embedding search and set intersection, fills candidate slots through vector search, and pairwise-reranks 20 candidates.The reported zero-shot configuration uses gpt-oss-120B and text-embedding-3-small.Source: §E.2 · p. 21
• ARK uses parallel agents with global BM25 search and typed one-hop neighborhood search, then fuses selected nodes by vote count.The reported results use GPT-5.2; the paper’s graph tools call ARK’s two tool implementations.Source: §E.2 · p. 21

E.3 KG Reasoning Methods Adapted to STaRK

• Yang et al. adapt three KG reasoning methods to STaRK and report their H@1 and MRR from the cited paper’s Tables 2 and 10.The supplied passage does not name the three methods.Source: §E.3 · p. 21
• ToG alternates LLM-guided relation and entity exploration; RoG plans with KG-grounded relation paths, while GoG proposes and verifies triples during training-free search.ToG stops when paths seem sufficient to the LLM or when search reaches its maximum depth.Source: §E.3 · p. 21

Design

• GPT-6-Luna answers each probe in one closed-book turn; the ID probe runs on all 7,108 queries, while the name probe covers MAG and Prime only.Amazon is excluded from the name probe because its product titles are long and rarely match exactly.Source: F Closed-Book Probes · p. 21

Name mapping

• The name probe maps returned names to node IDs using normalized node-file names, including MAG titles and Prime names, details.name, and aliases.A query counts as answered at rank r when the returned name maps to at least one gold node.Source: F Closed-Book Probes · pp. 21–22

Results

• On probe-missed queries, Full DKGI reaches H@1 of 88.4 on MAG and 56.9 on Prime; relation-access variants score 55.6–63.2 on missed Prime test-subset queries.Retrieval only and No edges score 28.7 and 28.3 on those 223 Prime queries.Source: F Closed-Book Probes · p. 22, Table 8 · p. 22
• The name probe answers 2.3% of MAG queries and 20.9% of Prime queries at H@1, while the ID probe gets only one Prime query correct.The ID probe gets zero of 1,642 Amazon and zero of 2,665 MAG queries correct.Source: F Closed-Book Probes · p. 22

Scope and command intents

• Trajectory measures cover 7,108 full-test queries per model in Pi and use outputs as the model saw them; command-intent comparisons include validation and public DCI-Agent-Lite trajectories.The Codex column of Table 9 uses GPT-6-Luna; paired values are for GPT-6-Luna and GPT-6-Astra, respectively.Source: G Behavior Measures · p. 22

Command intents and trajectory segments

• Python commands account for 47.6% of Full DKGI shell commands overall, and Python calls dominate later trajectory segments while search calls peak in the second segment.Python accounts for 55.6%, 41.5%, and 48.8% of Full DKGI shell commands on Amazon, MAG, and Prime; 41.1% of GPT-6-Luna Python commands refer to edge files.Source: G Behavior Measures · p. 23

Revisions

• Programs count as revisions at similarity of at least 0.5 after boilerplate removal; 99.7% of revisions use a program from an earlier turn whose output was visible.36.5% / 33.7% of programs meet the revision criterion.Source: G Behavior Measures · p. 23

IDs from earlier outputs

• Programs include IDs shown in earlier outputs in 48.4% / 59.9% of cases, and 83.1% / 95.9% of queries with Python have such a program.The measure excludes years, multiples of 1,000, and numbers appearing in the prompt.Source: G Behavior Measures · p. 23

Failures and the next program

• Among programs with a next program, 85.6% / 91.1% of failures are followed by a revision that runs successfully.Interpreter start errors are excluded from the failure and success counts.Source: G Behavior Measures · p. 23

Lineages and gold IDs in outputs

• Gold IDs appear in 40.0% / 49.4% of first lineage versions and 72.0% / 67.9% of last versions; programs provide the first shown gold ID for 68.5% of correct GPT-6-Luna queries.Programs make up 38.2% of all calls, including read calls.Source: G Behavior Measures · p. 23

Truncated outputs and the next turn

• After trajectory-segment weighting, Python runs more often in the next turn after truncated than after untruncated outputs of at least 4 KB for both models.The rates are 66.8% vs. 59.3% for GPT-6-Luna and 76.1% vs. 40.1% for GPT-6-Astra.Source: G Behavior Measures · p. 24

Text-constraint tags

• The LLM annotator marks 96% of Amazon and 57% of Prime queries as having an additional text constraint; Amazon H@1 is 8.1 points higher on constrained queries, while Prime is 2.1 points lower.All MAG queries are treated as text-constrained based on their topic descriptions.Source: G Behavior Measures · p. 24

H Error Analysis

• The analysis classifies incorrect outputs as empty answers, ranking errors, selection errors, or search misses according to the returned list and evidence visible to the model.A gold record counts as visible when its node_id appears in a record header or alone in square brackets; IDs elsewhere, such as edge rows, do not count.Source: H Error Analysis · p. 24
• 55% of GPT-6-Luna errors and 60% of GPT-6-Astra errors are ranking errors; search misses account for 36% and 24%, respectively, while empty answers are 1% for both.These shares divide incorrectly answered queries on the full test sets.Source: Table 10 · p. 24, H Error Analysis · p. 24

I Case Studies

• Across three test requests, Full DKGI ranks a gold node first, whereas Graph tools ranks another node first on each.The cases span MAG, Prime, and Amazon.Source: I Case Studies · p. 24

I.1 Case 1 (MAG): A Program Grows into a Multi-Hop Traversal

• In the MAG case, Full DKGI traverses from an institution through authors to papers, intersects with papers in a field, and finds the gold paper; Graph tools misses it across 38 outputs.The KG links the institution to 66 authors and those authors to 236 papers; 86 papers have the Data quality field, and one is among the 236. An initial newline-related bug is corrected before the traversal succeeds.Source: §I.1 · p. 25

I.2 Case 2 (Prime): A Program as Set Algebra over Two Relations

• In the Prime case, Full DKGI intersects pericardium genes with expression-present and expression-absent neighbors of seven thalamus nodes, then ranks ADH1B first.The program returns four genes; Graph tools tests genes individually and returns the same four with PDLIM3 first.Source: §I.2 · p. 26
• Predefined graph tools expose one-node, one-relation calls, while the Prime program combines relation-based sets in one run and computes their difference.Graph tools spends 25 calls on the 13 genes in this case.Source: §I.2 · p. 27

I.3 Case 3 (Amazon): An Error, Its Repair and Four Narrower Filters

• 18 gold products rank 1–18 with Full DKGI, while Graph tools returns them at ranks 2–19 behind an incorrect Rico Industries tag.The request asks for officially licensed NCAA brand metal tags.Source: §I.3 · p. 27
• After a literal-newline quoting error, Full DKGI reruns the code with corrected shell quoting and narrows its filter four times after truncated outputs.The final program looks up titles for 27 candidate IDs before returning the 18 gold products.Source: §I.3 · p. 27
• Graph tools searches product names and summaries with BM25 and returns complete records one at a time, but reviews precede titles in those records.The Full DKGI program filters titles, features, and descriptions across product records.Source: §I.3 · p. 27