PaperlayerView exact revision on arXiv

Source-linked AI summary

CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence

Yuzhe Zhang, Weijie Zhu, Haolin Yang, Ziyun Zhang, Xianwei Xue, Mengke Chen, Qiutong Pan, Huaqian Cai

arXiv:2609.36987v1cs.CL

TL;DR

Existing Text-to-Cypher benchmarks evaluate isolated queries rather than conversational sessions grounded in prior results. CYPHERTURN provides a human-verified multi-turn benchmark and evaluates models with guided and autonomous protocols. Results show low performance and a consequential leaderboard reordering under autonomy, establishing conversational graph reasoning as an open challenge.

  • Problem

    Existing Text-to-Cypher benchmarks evaluate independent queries, not conversational sessions in which follow-ups depend on earlier results.

    Source: §1 · p. 1

  • Method

    CYPHERTURN is a human-verified multi-turn Text-to-Cypher benchmark evaluated with guided oracle-context and autonomous agentic protocols.

    Source: §1 · p. 2

  • Results

    Evaluation finds low multi-turn performance and autonomous leaderboard reordering; the best model reaches only 64.7% guided EX.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    CYPHERTURN establishes conversational graph reasoning as an open challenge and motivates multi-turn training corpora and finer-grained semantic equivalence metrics.

    Source: §6 · p. 9

    All seven graphs are synthetic, so CYPHERTURN execution accuracy does not extrapolate to deployment accuracy on real enterprise graphs.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the multi-turn sessions through which analysts actually work. We introduce CypherTurn, the first benchmark for conversational Text-to-Cypher evaluation, comprising 721 sessions and 5,927 turns across 7 knowledge graphs and 13 conversational phenomena. We evaluate 15 models under a guided oracle protocol and a fully autonomous agentic protocol, yielding four findings. First, the best model reaches only 64.7% execution accuracy, and session-level correctness remains below 5%. Second, despite strong overall rank correlation, frontier models exhibit a consequential reordering of the top of the leaderboard under autonomous operation, a phenomenon we term the Autonomy Divergence, which reveals error-management as a partially independent capability from raw generation skill. Third, scaling action budgets from x3 to x10 fails to close the autonomy gap, as the strongest frontier models self-limit to approximately two actions per turn regardless of available budget. Fourth, single-turn Cypher fine-tuning degrades multi-turn instruction following, while architecture-appropriate specialization outperforms several frontier models. These results establish CypherTurn as an open challenge for conversational graph database reasoning. Code and data are available at https://github.com/BarryQ/CypherTurn.

1 Introduction

Existing Text-to-Cypher benchmarks evaluate independent queries, while real analysis depends on turns grounded in earlier results. CYPHERTURN addresses this gap with a human-verified multi-turn benchmark and complementary evaluation protocols, and identifies consequential leaderboard reordering under autonomous operation.

1 Introduction

  • Existing Text-to-Cypher benchmarks evaluate independent queries, leaving conversational references and dependencies across turns unmeasured.Follow-up questions may depend on results returned in earlier turns.Source: §1 · pp. 1–2
  • CYPHERTURN introduces a human-verified benchmark in which each turn’s Cypher must reference its predecessor’s result.It comprises 721 sessions, 5,927 turns, seven knowledge graphs, 13 phenomena, and nine chain modes.Source: §1 · p. 2
  • The Guided Protocol isolates query-generation ability with oracle context, while the Agentic Protocol requires models to manage their own multi-turn history.Source: §1 · p. 2
  • Under autonomous operation, frontier models reorder at the top of the leaderboard, revealing error management as partly independent from raw Cypher-generation skill.Source: §1 · p. 2

2 The CYPHERTURN Benchmark

CYPHERTURN combines multi-turn Text-to-Cypher evaluation with contextual anaphora and autonomous agent evaluation. Its benchmark spans purpose-built graphs, conversational phenomena, chain modes, personas, and human-reviewed queries.

2 The CYPHERTURN Benchmark

  • CYPHERTURN is the first Text-to-Cypher benchmark to combine multi-turn evaluation, contextual anaphora annotation, and a fully autonomous agentic protocol.The comparison table covers Text-to-Cypher and graph-query benchmarks; MTGQL targets nGQL rather than Cypher.Source: §2 · p. 2, Table 1 · p. 2

2.1 Knowledge Graphs

  • Seven synthetic knowledge graphs use distinct topologies to pose structural, recursive, hierarchical, and path-traversal challenges.Examples include a directed cycle, diamond convergence, bipartite cycle, citation loop, and bureaucratic hierarchy.Source: §2.1 · pp. 2–3

2.2 Conversational Phenomena

  • The 13 conversational phenomena group chain-dependent challenges into navigation, aggregation, filtering, and discourse categories.They cover traversal, numerical summaries, attribute constraints, complements, and fresh independent queries.Source: §2.2 · p. 3

2.3 Chain Modes

  • Nine chain modes provide a structural annotation layer distinct from the 13 finer-grained linguistic phenomenon labels.The modes include two non-chain-dependent types and seven chain-dependent types; multiple phenomena can map to one mode.Source: §2.3 · pp. 3–4

2.4 Session Personas

  • Each session receives one of ten near-uniformly sampled personas that diversifies utterance register rather than serving as a validated experimental factor.The dataset contains 5,927 turns, averaging 8.2 turns per session.Source: §2.4 · p. 4

2.5 Human Annotation and Quality Control

  • Two Cypher-proficient annotators review intent, anaphora clarity, and persona-appropriate naturalness, with κ = 0.83 accept/reject agreement.Revised turns are corrected and re-executed, and every retained gold query is independently verified to execute on its graph.Source: §2.5 · p. 4

3 Evaluation Framework

CypherTurn evaluates multi-turn Cypher generation under both oracle-provided history and autonomous session management. It reports turn- and session-level measures of execution, structural correctness, and error propagation.

3.1 Two Evaluation Protocols

  • The Guided Protocol supplies gold-annotated history to isolate query generation, while the Agentic Protocol requires autonomous history management under a shared session budget.The agentic setting allows models to allocate actions across turns and exposes them to errors compounding through session dependencies.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Metrics

  • Execution Accuracy (EX) is the primary metric, requiring identical execution results, including rows and column names.It compares predicted and gold queries executed over the same graph.Source: §3.2 · p. 4
  • Provenance Subgraph Jaccard Similarity (PSJS) measures structural correctness and diagnoses correct subgraph retrieval with output-formatting errors.For aggregation queries, the matched nodes are those bound by the MATCH clause.Source: §3.2 · pp. 4–5
  • Chain Error Rate (CER) measures how often a turn fails after its immediately preceding turn failed, indicating whether errors cascade.The first session turn and turns following a successful turn are excluded.Source: §3.2 · p. 5
  • Session Exact Match (SEM) is 1 only when every turn in a session achieves EX = 1.The metric is binary at the session level.Source: §3.2 · p. 5

4 Experiments

The experiments compare 15 models under guided and agentic evaluation, with Claude Opus 4.7 leading guided execution accuracy and Gemini-3.1-Flash-Lite leading under autonomy. The results also show that session-level correctness is rare and that higher action budgets benefit small models more than frontier models.

4.1 Experimental Setup

  • The evaluation covers 10 frontier API models, two small open-source baselines, and three fine-tuned or specialized models.CypherRI-7B is excluded from aggregate statistics because of fill-in-the-middle token contamination.Source: §4.1 · p. 5

4.2 Main Results

  • 0.647 EX makes Claude Opus 4.7 the guided leader, while Gemini-3.1-Flash-Lite moves from fourth to first under autonomy at 0.432.The Guided and Agentic ×3 comparison appears in Table 2; Figure 4 shows agentic EX across three action budgets.Source: §4.2 · pp. 5–6, Table 2 · p. 6, Figure 4 · p. 6
  • Only 33 of 721 sessions are fully correct for Claude Opus 4.7, whose session-level exact match is 0.046; STRuCT-LLM-Novo reaches 0.526 guided EX.STRuCT-LLM-Novo surpasses six frontier models.Source: §4.2 · p. 5
  • Expanding the action budget from ×3 to ×10 yields limited frontier gains but nearly doubles Llama-3.1-8B’s agentic EX to 0.261.Figure 4 plots agentic EX under ×3, ×5, and ×10 budgets.Source: §4.2 · p. 6, Figure 4 · p. 6

5 Analysis

The analysis locates difficulty in conversational phenomena and shows that autonomous history management separates some frontier models despite correlated overall rankings. Larger action budgets do not resolve that gap, while multi-turn-aligned specialization can help where single-turn fine-tuning hurts.

5.1 The Difficulty Landscape

  • REFINE averages 0.244 EX and CONTRAST 0.273, versus 0.650 for AGG_MAX and 0.602 for AGG_AVG; phenomenon type is the strongest difficulty predictor.Persona and graph topology are secondary factors; frontier-model persona spreads are typically 10–16 points.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2 The Autonomy Divergence: Why Rankings Shift

  • Frontier-model rankings correlate less under autonomy: Spearman ρ is 0.73, or 0.63 after excluding MiniMax-M2.7, with 11 of 45 pairwise comparisons reversing significantly.The overall correlation is 0.86 across all 14 models; the authors characterize the effect as differential robustness rather than a systemic reversal.Source: §5.2 · p. 7
  • Median conditional CER rises from 0.47 under guided evaluation to 0.77 under agentic evaluation, and exceeds 0.83 for DeepSeek-V3.2 and Qwen3-235B.The guided figure is computed from each model’s own per-turn EX, not injected oracle context.Source: §5.2 · p. 7
  • INSPECT_SCHEMA allocation correlates with agentic EX at Spearman ρ = 0.93 (p<0.001), while collapsing models spend 62–73% of actions on EXECUTE_CYPHER retries.The authors interpret schema inspection as a behavioral marker of resilience, not its cause; the correlation remains above 0.88 under leave-one-out removal.Source: §5.2 · p. 7, Table 3 · p. 8
  • Guided EX stays stable across T1–T10, but agentic ERNIE-5.0, DeepSeek-V3.2, and Qwen3-235B decline to near zero by T10.Claude and Gemini plateau, while GLM-5 recovers to roughly 0.44 after dipping at T3.Source: §5.2 · pp. 7–8, Figure 6 · p. 8

5.3 The Budget Ceiling: Why Extra Actions Don’t Help

  • Gemini-3.1-Flash-Lite remains near 0.43 from ×3 to ×10, while GLM-5 falls to 0.271 at ×10; Llama-3.1-8B nearly doubles to 0.261.At ×10, Gemini and Claude remain 0.142 and 0.223 points below their guided EX, respectively; frontier models self-limit or spend actions on retries.Source: §5.3 · p. 8
  • Removing budget-exhaustion turns shifts scores by at most 3.3 points for the four main-analysis models and 6.5 points for frontier ERNIE-5.0.Budget exhaustion exceeds 80% for small models and adjusts absolute scores without reordering the frontier.Source: §5.3 · p. 8

5.4 The Specialization Paradox

  • Single-turn Gemma fine-tuning lowers guided EX from 0.281 to 0.261, whereas STRuCT-LLM-Novo raises QwQ-32B’s guided EX from 0.395 to 0.526.The reported gain is 13.1 points; the authors argue the training paradigm must match the task’s multi-turn nature.Source: §5.4 · pp. 8–9, Table 4 · p. 9

6 Conclusion

CYPHERTURN evaluates conversational Text-to-Cypher across guided and agentic protocols, exposing low accuracy and a gap between query generation and autonomous error management. Its findings also show that larger action budgets do not close the gap and single-turn fine-tuning can hurt multi-turn performance.

6 Conclusion

  • The best model reaches only 64.7% guided EX, while session-level correctness barely exceeds 4%.The benchmark comprises 721 sessions and 5,927 gold-verified queries across seven knowledge graphs and 13 phenomena.Source: §6 · p. 9
  • Autonomy Divergence captures a top-leaderboard reordering under autonomous operation, where error management is partly independent of raw generation skill.The reordering is invisible under oracle-context evaluation.Source: §6 · p. 9
  • Extra action budget does not close the autonomy gap, and frontier models self-limit regardless of available resources.The conclusion also reports that single-turn fine-tuning degrades multi-turn performance, while architecture-appropriate specialization outperforms several frontier models.Source: §6 · p. 9

Limitations

The paper bounds its results to synthetic English graphs, a finite model sample, and an execution metric sensitive to output formatting. Related work spans conversational SQL, single-turn graph-query benchmarks, and agentic evaluation, but the cited graph-query landscape lacks CYPHERTURN’s combination of capabilities.

Limitations

  • All seven knowledge graphs are synthetic, so benchmark execution accuracy does not extrapolate to real enterprise deployment.The benchmark measures multi-turn reasoning under controlled topology; a real-world graph pilot remains future work.Source: Limitations · p. 9
  • The evaluation covers 15 models but not all competitive systems, and newer releases may shift the reported absolute scores.All sessions are in English, leaving the persistence of Autonomy Divergence across typologically different languages open.Source: Limitations · p. 9
  • Execution Accuracy requires result-set identity including column names, potentially penalizing semantically equivalent queries with different formatting or column ordering.A finer-grained semantic equivalence metric remains future work.Source: Limitations · p. 9

A Related Work

  • Prior work covers conversational Text-to-SQL and diverse graph-query benchmarks, but existing Cypher datasets lack genuine conversational context and chain-dependent anaphora annotation.MTGQL is a multi-turn graph-query benchmark for nGQL, but does not combine Cypher, contextual anaphora annotation, and an agentic protocol.Source: A Related Work · p. 13
  • Agentic benchmarks expose gaps between oracle and autonomous performance; CYPHERTURN adds graph-traversal-specific dependencies and CER for cross-turn error propagation.The cited prior work includes interactive SQL evaluation, tool-agent environments, and long-context evaluation.Source: A Related Work · p. 14

B Comparison Benchmarks

The benchmark comparison situates CYPHERTURN among single-turn Cypher datasets and a distinct multi-turn nGQL benchmark. Its construction combines varied graph topologies and conversational phenomena with session personas, while the evaluation spans frontier, open, and specialized models.

B Comparison Benchmarks

  • The compared Cypher benchmarks are single-turn; MTGQL is the multi-turn graph-query exception, targeting nGQL rather than Cypher.The cited comparisons include CypherBench, ZOGRASCOPE, SM3-Text-to-Query, Text2GQL-Bench, and MTGQL.Source: B Comparison Benchmarks · p. 14

C Model Descriptions

• Frontier Models. / • Small Open-Source Models.

• Fine-tuned / Specialized Models.

  • CypherRI-7B’s reasoning tokens contaminate generated Cypher payloads, producing near-zero execution accuracy from format-compliance failure rather than generation ability.Its results are excluded from aggregate statistics.Source: • Fine-tuned / Specialized Models · p. 16

D.1 Knowledge Graph Schemas

  • Seven purpose-built graphs use distinct topologies to pose traversal challenges, including cycles, bipartite cycles, diamond convergence, stars, and self-loops.Examples include recursive organizational traversal, lateral alliance edges, and multi-party invoice structures.Source: §D.1 · p. 16, Figure 7 · p. 17

D.2 Conversational Phenomena

  • Every session turn is annotated with one of 13 conversational phenomena grouped into four categories.The supplied passage says the accompanying table lists each phenomenon’s category, chain mode, dependency status, definition, and canonical Cypher pattern.Source: §D.2 · p. 16

D.3 Chain Modes

  • The benchmark defines nine chain modes describing how a current query relates to the prior result.The listed modes include two non-chain-dependent modes; all others require the prior result to formulate the current query.Source: §D.3 · p. 16

D.4 Session Personas

  • Sessions receive one of 10 randomly assigned personas to govern utterance register; persona naturalness is the weakest agreement criterion.The pooled κ = 0.83 is measured, while criterion-level values are a retrospective decomposition rather than an independent remeasurement.Source: §D.4 · pp. 17–18

D.5 Evaluation Protocols

The evaluation contrasts guided query generation with agentic sessions in which models must discover schema and manage shared action budgets. Results report substantial variation by graph and phenomenon, while protocol variants alter ranking conditions.

D.5 Evaluation Protocols

  • The five actions are shared across protocols; agentic models must discover withheld schema through INSPECT_SCHEMA, whereas guided prompts include it.Each action costs one budget unit.Source: §D.5 · pp. 18–19, Table 9 · p. 20
  • Guided turns receive gold prior-turn history and full schema, with an action loop capped at three steps.For CONTRAST turns, the immediately preceding gold result is passed without row truncation.Source: §D.5 · p. 19
  • ASK_USER returns scripted replies in guided evaluation and a dedicated LLM-based simulator response in agentic evaluation.In guided evaluation, non-ambiguous turns receive a fixed discouragement string.Source: §D.5 · p. 19
  • Agentic action budget is shared across a session; exhaustion mid-session assigns zero EX to all remaining turns, penalizing early over-investment.The budget is B = T × m, with m ∈{3, 5, 10}; each action decrements the shared budget.Source: §D.5 · pp. 20–21
  • Three protocol variants separately test persistent schema, disclosed session horizon, and prompts encouraging full-budget use.The budget-encouraging variant runs at tenfold budget.Source: §D.5 · p. 21

E.1 Per-Graph Results

  • Agentic EX drops sharply while graph ranking is broadly preserved; Arcane Archive falls from 0.469 guided to 0.250 agentic ×3.Celestial Court drops from 0.485 guided to 0.279 agentic ×3.Source: §E.1 · p. 21

E.1 Per-Graph Results / E.2 Per-Phenomenon Results

  • Guided REFINE is the hardest phenomenon (mean 0.244); Stellar Colony (0.426) and Merchant Harbor (0.434) are the hardest guided graphs, while Celestial Court is easiest (0.485).The per-phenomenon figures are means; the graph figures are guided EX values.Source: §E.2 · p. 21, §E.1 · p. 21

E.3 Per-Chain-Mode Results

Guided execution is highest for independent topic shifts, while chain-dependent modes vary in difficulty. Removing oracle context causes substantial degradation, especially for contrast, and context-resolution errors compound over long sessions.

E.3 Per-Chain-Mode Results

  • Topic_shift has the highest guided mean EX at 0.642, while None (T1) is 0.359 despite being self-contained because it requires cold-start schema grounding.Among chain-dependent modes, agg_numeric and aggregate are easiest; contrast and narrow are hardest.Source: §E.3 · pp. 21–22
  • Contrast retains only 27% of its guided EX under the agentic protocol, the largest relative drop among chain modes.Aggregate and value_narrow show the largest absolute drops, at 0.29 and 0.28 points.Source: §E.3 · pp. 21–22
  • Context-resolution errors compound over long sessions.Source: §E.3 · p. 22

E.4 Per-Persona Results

Persona effects are moderate under guided oracle context but more consequential under agentic evaluation, where terse requests place greater demands on context resolution.

E.4 Per-Persona Results

  • The verbose user persona has the highest guided per-model mean EX at 0.511, while persona spreads are typically 10–16 percentage points for frontier models.The spread indicates persona does not dramatically inflate or suppress raw generation ability under oracle context.Source: §E.4 · pp. 22–23, Table 16 · p. 24
  • Under agentic evaluation, the impatient executive persona has the lowest per-model mean EX at 0.226, as terse utterances with implicit referents require stronger context resolution.Source: §E.4 · p. 23
  • Under guided oracle context, the impatient executive sits only marginally above the novice user, whose column mean is lowest.Source: §E.4 · p. 23, Table 16 · p. 24

E.5 Agentic Budget Scaling

Increasing agentic action budgets produces varied model responses rather than uniform gains. Agentic chain-error rates can rise sharply across turns, while the guided protocol keeps failures more isolated.

E.5 Agentic Budget Scaling

  • Gemini-3.1-FL execution accuracy is nearly flat across ×3, ×5, and ×10 budgets at 0.432, 0.431, and 0.430, respectively.Other responses range from gains that plateau at ×5 to late gains, overbudget degradation, and volatility.Source: §E.5 · p. 24, Table 18 · p. 25
  • Under agentic evaluation, DeepSeek-V3.2 and Qwen3-235B approach 1.0 chain error rate by T9–T10, while Gemini-3.1-Flash-Lite plateaus near 0.70.Under guided evaluation, CER remains moderate and near-flat; gold oracle context quarantines failures and limits propagation.Source: §E.5 · pp. 24–25, Figure 8 · p. 24
  • Frontier models use ASK_USER for only 1.9%–9.3% of non-SUBMIT actions, mostly on harder turns where conditional accuracy falls below overall accuracy.Generic rejections to most schema probes make ASK_USER a narrow, selectivity-biased channel rather than a reliable recovery mechanism.Source: §E.5 · p. 25

F Protocol Robustness

Protocol perturbations preserve broad model rankings but change who leads, and none closes the guided-to-agentic gap. Budget exhaustion has limited impact on frontier rankings, while case studies and prompt specifications illustrate the evaluation setup.

F.1 Variant Design and Overall Effect

  • All three protocol variants preserve overall ranking at Spearman ρ ≥0.78, yet each displaces Gemini-3.1-Flash-Lite from the top position.The variants alter schema persistence, horizon disclosure, or encouragement to use the full budget; Gemini’s base-agentic lead therefore partly reflects protocol strategy rather than raw capability.Source: §F.1 · p. 26, Table 20 · p. 26

F.2 Gap Persistence and Budget Cliff

  • No protocol variant closes any frontier model’s guided-to-agentic gap; even under Agentic-PS, Gemini drops 0.137, Claude 0.186, and Qwen3 0.292 from guided EX.The cross-model gap spread remains 0.090–0.174 across variants.Source: §F.2 · p. 27, Table 24 · p. 27
  • Excluding budget-exhausted turns leaves the frontier ranking unchanged, with Gemini still first and Qwen3 and DeepSeek at the tail.The budget cliff contributes only about 3 of Qwen3-235B’s 31-point overall drop; it dominates only small models, whose exhaustion rates exceed 80%.Source: §F.2 · p. 27, Table 25 · p. 27

G Case Studies

  • Across the five case studies, examples illustrate perfect guided performance, cascading agentic failure, agentic resilience, fine-tuning backfire, and SQL contamination.The sessions show the user utterance, predicted Cypher, and clean output-token count.Source: G Case Studies · p. 28, Figure 9 · p. 29, Figure 10 · p. 30

H Evaluation Prompts

  • The evaluation prompts are reproduced verbatim, with P1 shared across protocols, P2 guided-only, P3–P4 agentic-only, and P5–P6 used by the agentic user simulator.Source: H Evaluation Prompts · p. 34

P1 — Action-Space System Prompt (shared by both protocols)

  • The shared action-space prompt requires one action per response, charges one budget unit per action, and directs models to build on prior conversation results rather than restart.The agentic prompt adds autonomous mode, a total action budget, and schema discovery through INSPECT_SCHEMA.Source: P1 — Action-Space System Prompt… · p. 34

Related papers

Loading 2609.36987v1…