PaperlayerView exact revision on arXiv

Source-linked AI summary

What Drives Recovery in Agentic Text-to-Cypher? LAST-CQ: An LLM Agent Self-Refinement Framework

Ioannis Prokopiou, Athanasios Aidinis, Panagiotis-Christos Kyrmpatsos, Pantelis Vikatos

arXiv:2609.12746v1cs.AIcs.CLcs.LGcs.MAcs.SE

TL;DR

Agentic Text-to-Cypher loops can improve query generation, but it is unclear which loop components account for the gains. Using LAST-CQ as an instrumented testbed, the paper finds that counterfactual results favor failure detection and routing over more sophisticated feedback or parallel sampling. Its judge calibration is limited to a single annotator and selected corrected-query outputs.

  • Problem

    Agentic-loop gains are often attributed to the overall architecture, leaving the contributions of failure detection, feedback synthesis, and regeneration unclear.

    Source: §1 · p. 1

  • Method

    LAST-CQ is a five-agent, training-free Text-to-Cypher framework used to remove and test refinement mechanisms through counterfactuals.

    Source: §1 · p. 1, §9 · p. 9

  • Results

    Removing correction costs 3.1% against the single-pass system and 12.3% against a no-refinement counterfactual, while the study localizes the gain to detection and orchestration.

    Source: §1 · p. 1, §9 · p. 9

  • Takeaways & Limitations

    The counterfactuals indicate that detecting failure and routing queries to retries matter more here than feedback sophistication or generation volume.

    Source: §1 · p. 1, §9 · p. 9

    Judge calibration used one author and corrected queries on two backbones, so it establishes the direction and rough size of optimism, not a corpus-level correction.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Agentic pipelines for structured-query generation are rapidly expanding, but it is unclear which part of the loop produces the gain. We use LAST-CQ -- a five-agent, training-free, execution-grounded Text-to-Cypher framework -- as an instrumented testbed, running three counterfactuals over 2,471 live-database queries and six backbones spanning three vendor scale tiers. Removing correction is worth between 3.1% aggregate execution-BLEU against the single-pass system and 12.3% against a no-refinement counterfactual (up to 80.7% for the weakest backbone). Replacing schema-grounded, LLM-synthesised feedback with raw database error strings costs almost nothing (20.9% vs. 19.9% naive exact match; <0.2% end-to-end; equivalent within $\pm 0.075$ set-F1 by two one-sided tests). Spending the same call budget on parallel sampling degrades quality by 10-11%. What works is detecting failure and routing it to a retry, not the feedback sophistication or number of samples. LAST-CQ itself recovers 91.7% of queries that fail under single-pass generation, while a query that succeeds first time still costs exactly one LLM call. We also show that n-gram overlap on serialised results is not a bound in either direction: it over-scores against set equivalence on 65.9% of results while under-scoring against judged semantics. Finally, we calibrate our LLM judge against blind human labels and find it optimistic by 9 points.

1 Introduction

The paper asks which parts of an agentic Text-to-Cypher loop account for its gains, then uses LAST-CQ to isolate them. Across three counterfactuals, the results point to failure detection and routing rather than feedback sophistication or generation volume.

1 Introduction

  • The study frames the unresolved issue as whether agentic-loop gains come from failure detection, feedback synthesis, or regeneration.Prior work reports gains for the loop as a whole, leaving the contribution of each mechanism unclear.Source: §1 · p. 1
  • LAST-CQ recovers 91.7% of queries that fail under single-pass generation, while making each loop mechanism independently removable for measurement.It is a five-agent, training-free refinement framework for Text-to-Cypher.Source: §1 · p. 1
  • Removing correction costs 3.1% against the single-pass system and 12.3% against a no-refinement counterfactual, reaching 80.7% for the weakest model.The counterfactuals cover 2,471 queries and six backbones.Source: §1 · p. 1
  • The paper contributes an ablation testbed and an order-invariant evaluation of agentic structured-query generation.It reports judge-based semantic accuracy over 13,563 executed results and compares n-gram overlap with stricter and looser correctness notions.Source: §1 · p. 2

2 Related Work

Prior work studies execution-grounded refinement and related Text-to-Cypher, SPARQL, and graph-retrieval approaches. This paper distinguishes itself by testing which loop components matter and by comparing its framework with the closest published system.

2.1 Execution-Grounded Refinement

  • Execution-guided refinement work often rests on the premise that richer feedback improves correction, but this paper tests that premise and finds it largely unsupported here.The cited line of work includes Self-Refine, Reflexion, prospective reflection, and dynamic agent topologies.Source: §2.1 · p. 2

2.2 Text to Cypher, SPARQL and Graph RAG

  • Related work spans property-graph adaptations, benchmark and training-data releases, commercial retries on raw errors, Graph RAG, and Text-to-SPARQL.The cited Graph RAG approaches bypass query formulation by retrieving subgraph context, trading precision and index utilisation for robustness.Source: §2.2 · p. 2
  • Gusarov et al. (2025) is the closest published system, but it has no ablation or recovery analysis, unlike this paper’s component ablations and recovery analysis.The systems also differ in engine, agent count, validation, correction, and evaluation setup.Source: §2.2 · p. 2
  • The authors do not claim the architectural family is novel; whether their findings transfer to a Memgraph pipeline with fuzzy entity re-grounding remains untested.They claim only that the counterfactual design is directly reusable on that pipeline.Source: §2.2 · p. 2

3 LAST-CQ: An Instrumented Testbed

LAST-CQ separates Text-to-Cypher generation into agents with explicit call costs, schema checks, and execution feedback. Its formal control flow retries failures, while worked examples show the distinct roles of synthesized and fixed hints.

3.1 Agents

  • LAST-CQ uses five agents: deterministic schema parsing, generation, validation, correction, and execution, with model calls concentrated in generation and failure handling.The parser handled all 2,471 benchmark schemas without invoking its LLM fallback; validation can return immediately when checks pass.Source: §3.1 · p. 3

3.2 Formal Framework

  • The framework treats a query as correct when execution returns a non-empty result, while acknowledging that non-empty output does not establish semantic correctness.Its refinement state tracks the candidate query, hints, attempted-query history, and schema.Source: §3.2 · p. 3
  • Validation and correction alternate within an iteration budget; an empty execution result triggers a relaxation pass before the process terminates.The hint sources differ: Hval is synthesized from deterministic schema analysis, whereas Hrelax is a fixed constant.Source: §3.2 · p. 3
  • A first-pass success costs exactly one LLM call, while each failed cycle adds up to three calls for parsing, hint synthesis, and correction.The passage reports that 53–97% of queries need no correction and derived mean LLM calls range from about 1.1 for Claude to 2.5 for Gemini Flash.Source: §3.2 · p. 3

3.3 Worked Example

  • The worked examples contrast a schema-grounded hint replacing invalid YEAR() with a fixed relaxation hint for a valid query that returns nothing.The second trace applies a case-insensitive regex form from Hrelax to address a case- and substring-sensitive filter.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

4 Experimental Setup

The evaluation uses executable Text-to-Cypher questions, six prompted backbones, and several refinement counterfactuals. It reports metrics with different invariances and notes that some result-set analyses use only retained-payload subsets.

4 Experimental Setup

  • The study uses 2,471 executable question–query pairs across 16 domains and six off-the-shelf backbones at three scales, all run in context at T = 0 without fine-tuning.The dataset is the executable subset of the Neo4j Text2Cypher benchmark.Source: §4 · p. 4
  • The comparison includes single-pass generation, LAST-CQ variants, naive retry, and Best-of-3; the no-refinement score reassigns zero to queries entering correction rather than coming from a separate run.The setup uses two validation cycles and one relaxation pass for LAST-CQ+DB.Source: §4 · pp. 4–5
  • The authors exclude fine-tuned models because their reported gains are larger than the inference-time results studied here and require labelled data and training.The paper instead asks which refinement-loop component earns its keep with a frozen model behind an API.Source: §4 · p. 5
  • Execution Google-BLEU is order-sensitive and treated as a comparability metric, whereas exact match, judge semantic accuracy, and set metrics are order-invariant.The judge labels executed results CORRECT, PARTIAL, or INCORRECT; set-based metrics operate on canonicalized rows.Source: §4 · p. 5
  • Per-query result payloads were retained for the 1,917-query correction subset and a 214-query paired subset, so set-based metrics use those subsets rather than the full corpus.“Aggregate” denotes an unweighted mean across six backbones, not pooled queries.Source: §4 · p. 5

5 RQ1: Does the Framework Work?

LAST-CQ improves execution-based query quality and recovers most queries that fail on the first pass, with recovery adding cost mainly when correction is needed. Its judged semantic accuracy describes refined outputs but does not establish a semantic improvement over single-pass generation.

5.1 Execution Quality

  • Judged semantic accuracy characterizes LAST-CQ+DB outputs, but cannot establish semantic improvement over single-pass generation because baseline payloads were not retained.The judge was run on LAST-CQ+DB outputs only.Source: §5.1 · p. 5, Table 2 · p. 6
  • 3.4% average GLEU improvement over Baseline accompanies a more modest +0.5% exact-match improvement.Schema-only LAST-CQ yields about +1% GLEU and slightly decreases it for the two strongest models.Source: §5.1 · p. 5, Table 2 · p. 6

5.2 Recovery and Cost

  • 91.7% of queries failing single-pass generation are recovered, lifting hit retrieval by 11.8% at 3.14–4.19 pipeline stages per query.The pooled recovery rate is 1,799/1,917 = 93.8%, compared with the 91.7% per-model mean.Source: §5.2 · pp. 5–6, Table 2 · p. 6

6 RQ2: What Actually Drives Recovery?

Three counterfactuals separate the value of correction from the value of its feedback: removing correction hurts, while schema-grounded hints add little and equal-budget sampling performs worse. The comparisons show that outcomes depend on the baseline and evaluation convention.

6.1 Removing Correction Entirely

  • Correction’s aggregate contribution is bounded by 12.3% versus the no-refinement counterfactual (corrected queries zeroed) and +3.1% versus deployed single-pass generation (0.4729 vs. 0.4587); Gemini Flash gains 80.7% under the former.Source: §6.1 · p. 6

6.2 Removing Schema Grounding

  • The 1,917-query feedback comparison measures relative correction quality under LAST-CQ’s shared detector, not relative end-to-end system quality.The compared set comprises model-query pairs that LAST-CQ chose to correct.Source: §6.2 · p. 7
  • 19.9% exact match for naive retry versus 20.9% for LAST-CQ, with under 0.2% end-to-end GLEU difference, shows little gain from schema-grounded hints.On paired retained results, set-F1 is 0.200 for LAST-CQ versus 0.219 for naive retry; the difference is equivalent to zero within ±0.075 set-F1, but not ±0.05.Source: §6.2 · p. 7
  • 12.0% of corrected queries have empty references, and excluding them reverses the aggregate edge: naive retry scores 0.365 versus 0.350 GLEU and 0.104 versus 0.070 set-F1.The benchmark awards a perfect GLEU score when the reference is empty and the output is not.Source: §6.2 · p. 7

6.3 Spending the Budget on Sampling

  • Equal-budget sampling with a weak first-non-empty selector was a worse budget use than grounded sequential correction, with 10.8% and 11.4% degradations against LAST-CQ+DB scores of 0.4378 and 0.5421.The selector was deliberately the cheap first-non-empty choice, not majority vote or reranking by EXPLAIN validity.Source: §6.3 · p. 8

7 RQ3: Are We Measuring It Correctly?

The evaluation compares serialized-result overlap with set-based and judged notions of correctness, which can disagree in opposite directions. A blind human check also finds that the LLM judge overstates validity.

7 RQ3: Are We Measuring It Correctly?

  • 65.9% of retained result pairs have BLEU above set-F1, while 3.1% fall below; set-identical results average 0.891 BLEU, exposing both over- and under-scoring.Against judged semantics, BLEU is pessimistic: 33.9% of partial-BLEU queries were judged fully correct.Source: §7 · p. 8
  • 88.2% of 13,563 results are at least partially valid and 52.3% correct; recovered queries score 83.3% versus 88.7% for non-recovered queries.The paper argues that no single scalar can adjudicate these systems and reports three invariance classes.Source: §7 · p. 8
  • On 122 corrected queries, the judge labels 74.6% at least partially correct versus 65.6% for humans, a 9-point optimism concentrated in PARTIAL labels.Human-judge agreement is 64.8% for three-way decisions and 82.8% for valid/invalid decisions.Source: §7 · p. 8

8 Implications for Agent Design

The counterfactuals favor detection and routing over sophisticated feedback or parallel sampling within this study. The authors also distinguish accuracy gains from engineering benefits and stress that evaluation metrics encode different notions of correctness.

8 Implications for Agent Design

  • Correction removal costs 3.1–12.3%, schema grounding about one point of exact match, and equal-budget sampling 10–11%, supporting investment in detection and routing over hint sophistication.For verifiable execution environments, the authors recommend bounded retry budgets; the conclusions are scoped to the tested framework and setting.Source: §8 · p. 8
  • Order-sensitive overlap can over-report against set equivalence and under-report against judged semantics, so evaluation should state which invariances its metrics preserve.Deterministic parsing and pre-execution validation save calls or round-trips, but the authors describe these as engineering properties rather than accuracy gains.Source: §8 · pp. 8–9

9 Conclusion

The conclusion presents LAST-CQ as an instrument for ablating agentic refinement and evaluating structured-query results. Its evaluation and deployment scope leaves several directions for future work.

9 Conclusion

  • LAST-CQ recovers 91.7% of single-pass failures over 2,471 queries and six backbones, while counterfactuals localize the gain to detection and orchestration.The framework combines schema-aware validation with execution-grounded correction.Source: §9 · p. 9
  • Future work includes AST canonicalization, tree matching, SMT-based checks, semantic validation beyond surface results, fine-tuning comparisons, and large-scale latency and cost analysis.The paper also identifies evaluation across multiple graph database engines as a dimension to examine.Source: §9 · p. 9

Limitations

The evidence is bounded by one benchmark and engine, incomplete metric coverage, and comparisons conditioned on corrected queries. Several design and evaluation choices further limit what the results establish.

Limitations

  • Generalisation beyond the executable Neo4j benchmark and its 16 domains, including to other property-graph engines, remains untested.The counterfactual design is described as engine-agnostic, but evaluation on another engine and benchmark is identified as a next experiment.Source: Limitations · p. 9
  • Set-based precision, recall, F1, and Jaccard cover the 1,917-query correction subset and 214-query paired subset, not the full 2,471-query corpus per model.Full-corpus order-invariant evidence relies on exact match and judge accuracy; the paired head-to-head covers two of six backbones.Source: Limitations · p. 9
  • The 1,917-query comparison is conditioned on LAST-CQ failures, so it measures correction quality rather than end-to-end quality against naive retry.Naive retry is not evaluated on queries LAST-CQ answered on the first pass.Source: Limitations · p. 9
  • The no-refinement counterfactual re-scores an existing run and upper-bounds refinement’s contribution; the single-pass baseline provides the +3.1% lower bound.Both comparisons are reported rather than treating the counterfactual as an executed alternative system.Source: Limitations · p. 9
  • Schema-grounding equivalence is bounded to ±0.075 set-F1 on 214 queries across two backbones, while best-of-3 confounds temperature with first-non-empty selection.A smaller schema-grounding advantage cannot be excluded, and the sampling result concerns equal-budget sampling as configured, not sampling in principle.Source: Limitations · p. 9
  • Human calibration uses one annotator and corrected queries on two backbones, establishing judge optimism’s direction and rough size, not corpus-level corrected accuracy.The judge was not run on baseline outputs, so the calibration cannot establish semantic improvement over single-pass generation.Source: Limitations · p. 9
  • For 12.0% of corrected queries, empty references make any non-empty output score 1.0 under the benchmark’s GLEU convention, even when wrong.The authors report an exclusion analysis and note that fewer empty references would test the loop more sharply.Source: Limitations · pp. 9–10
  • The analysis did not systematically sweep N or M, ablate individual relaxation hints, test blind retry without feedback, or record per-call token counts.N=2 came from the observed cycle distribution and M=1 is fixed by implementation; larger M was not measured.Source: Limitations · p. 10

Ethics Statement

The paper reports use of public data and APIs, describes model scale groupings based partly on vendor positioning, and calibrates its execution-outcome judge against a limited blind human annotation.

Ethics Statement

  • The study uses public benchmarks, demonstration databases, and commercial LLM APIs, with no private or sensitive data processed.For deployment against sensitive graphs, the authors advise query-level access controls and result-scope limits.Source: Ethics Statement · p. 10

A Model Configurations

  • Six backbones run at T = 0 without fine-tuning, while scale groupings reflect vendor positioning and relative pricing rather than measured size.Only DeepSeek-V3 has a vendor-published parameter count; third-party estimates are identified as such.Source: A Model Configurations · p. 13, Table 5 · p. 13

B Judge and Annotation Protocol

  • The primary GPT-4o-mini judge evaluates executed results on a three-point rubric, with parseable verdicts for 13,563 of 14,826 model-query pairs.Coverage ranges from 87.6% to 94.3%; semantic-accuracy results therefore describe queries with comparable returned payloads.Source: B Judge and Annotation Protocol · p. 13
  • Table 6 reports 64.8% three-way human-judge agreement (κ = 0.48) and 82.8% agreement after CORRECT and PARTIAL are grouped as valid (κ = 0.59).The comparison uses the 122 blind-labelled queries.Source: Table 6 · p. 14
  • On 122 blind-labelled queries, the judge rated 15 of 42 human-INCORRECT cases as PARTIAL, and only one as CORRECT.The labels came from one author and two backbones; they establish the direction and rough size of judge optimism, not corpus-level corrected accuracy.Source: B Judge and Annotation Protocol · p. 14

C Additional Analyses

Most queries entering validation resolve in one cycle, while results vary across error classes and evaluation metrics. The analyses also report benchmark edge cases and the coverage of the schema parser and set metrics.

C Additional Analyses

  • 80% of the 1,917 queries entering validation resolve in one cycle and 20% require two; fewer than 13% of queries enter a cycle for most models.Gemini Flash is the exception at roughly 50%; mean GLEU for one-cycle corrections is 0.36–0.55, compared with 0.43–0.54 for zero-cycle queries.Source: C Additional Analyses · p. 14, Figure 3 · p. 14
  • Mean validation cycles per query range from 0.03 for Claude to 0.50 for Gemini Flash, with 87.6%–97.5% of queries resolved without correction.The per-model cycle means are 0.19, 0.50, 0.03, 0.07, 0.09, and 0.04 for GPT-4o-mini, Gemini Flash, Claude, Gemini Pro, GPT-4o, and DeepSeek, respectively.Source: C Additional Analyses · p. 14
  • Table 2 values have average 95% bootstrap confidence intervals of ±0.015 GLEU, and per-model baseline-to-LAST-CQ+DB gains fall within overlapping intervals.The authors frame the effect of interest as recovery rather than incremental gain on already-working queries.Source: C Additional Analyses · p. 14
  • Across models, 23% of results reproduce the reference exactly under GLEU, 18% score above 0.5, 58% below, and under 1% have no overlap.Section 7 attributes the middle band largely to metric behaviour rather than semantic failure.Source: C Additional Analyses · p. 14
  • Schema violations are recovered in 88% of 75 classifiable cases and runtime errors in 49% of 119, while syntax errors are never recovered (0/31).Type mismatches are almost never recovered (8% of 13); DeepSeek retains a 93% recovery rate despite a GLEU decrease linked mostly to schema normalisation differences.Source: C Additional Analyses · p. 14
  • Among 230 corrected queries with empty reference results, 54 return non-empty output under naive retry; in the 214-query paired subset, 3 of 33 do so under LAST-CQ.Both are scored 1.0 by the benchmark’s GLEU convention despite being wrong.Source: C Additional Analyses · pp. 14–15
  • Deterministic parsers handle 100% of benchmark schemas using two of eight supported formats, and the LLM fallback never fires.The authors note that non-standard deployment schemas could still require the fallback.Source: C Additional Analyses · p. 15
  • Set metrics canonicalise rows as sorted key-value tuples, discarding row and key order; when both result sets are empty, all measures equal 1.When exactly one set is empty, all measures equal 0.Source: C Additional Analyses · p. 15
Loading 2609.12746v1…