PaperlayerView exact revision on arXiv

Source-linked AI summary

PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systems

Suraj Ranganath, Anish Raghavendra

arXiv:2606.08481v1cs.LGcs.AIcs.DBcs.SE

TL;DR

Enterprise Text2Cypher benchmarks must capture private, changing graph schemas and workloads while ensuring executable, diverse, governed examples. PIPE-Cypher builds such benchmarks locally through profiling, reverse grounding, constrained generation, deterministic validation, and calibrated judging, producing balanced and auditable exports that expose downstream model failures. Its scope is bounded by proxy graphs, template concentration, limited judge calibration, and excluded risky Cypher constructs.

  • Problem

    Enterprise benchmarks need to represent organization-specific schemas, values, governance constraints, and changing operational questions that static public datasets cannot capture.

  • Method

    PIPE-Cypher combines schema profiling, reverse grounding, constrained local generation, deterministic Cypher governance, execution validation, diversity controls, redaction, and local judge review.

  • Results

    PIPE-Cypher produced 3,000 accepted examples from 4,925 candidates, with perfect category balance, near-perfect difficulty balance, and 1,115 unique grounded entity values.

  • Takeaways & Limitations

    The workflow makes Text2Cypher benchmarking private, repeatable, balanced, auditable, refreshable, and capable of revealing failures that syntax-only evaluation would hide.

  • Takeaways & Limitations

    Evaluation uses public enterprise-style proxy graphs, query-signature diagnostics show template concentration, judge calibration has one human annotator, and risky Cypher constructs are excluded.

Abstract

from arXiv · show

Enterprise property graphs vary widely in schema structure, internal terminology, domain assumptions, governance constraints, and user interaction patterns. A deployment-relevant Text2Cypher benchmark therefore reflects the questions users and agents actually ask of that graph. Creating such a benchmark is difficult because schemas and values are unique, and graph structure changes over time. Each NL-query pair must also be executable, use real graph entities, preserve diversity, and remain balanced across query types and difficulty levels. We present PIPE-Cypher, a local benchmark-generation pipeline that turns a live property graph and optional seed queries from customer questions, analyst logs, or agent tool calls into balanced NL-to-Cypher benchmarks. PIPE-Cypher combines schema profiling, reverse-query grounding, constrained generation, deterministic Cypher governance, execution validation, redaction, diversity controls, and a calibrated local LLM judge. Using local Qwen3.5-9B generation and judging, PIPE-Cypher exports 3,000 accepted FinBench/SNB examples, completes three audited ablation suites, calibrates judge behavior with human labels, and evaluates 11 local downstream models. The resulting benchmark is deliberately discriminative: zero-shot transfer is weak, while a few-shot control shows that schema-specific example banks can help compatible model families. Together, PIPE-Cypher makes Text2Cypher benchmarking a repeatable process that evolves with the graph, its users, and its target workloads.

1 Introduction

Enterprise Text2Cypher evaluation must reflect organization-specific schemas, values, governance rules, and operational questions rather than relying only on public datasets. PIPE-Cypher addresses this need with a constrained, repeatable workflow for generating executable, privacy-aware benchmarks from live graphs.

  • Enterprise graph analytics depends on organization-specific labels, relationship directions, values, governance rules, and recurring operational questions.
  • Static public datasets cannot contain private taxonomies and answer-enabling values or evolve with production schema changes.
  • General-agent benchmark generation is less reliable for enterprise refreshes because local models struggle with grounding, safety, balance, ambiguity rejection, and reviewable evidence.
  • PIPE-Cypher uses constrained generation while deterministic graph checks make benchmark creation repeatable, scalable, and auditable.

2 Related Work

Prior Text2Cypher work emphasizes schema grounding, execution checks, verification, and complexity-aware evaluation, while PIPE-Cypher shifts attention from static resources to organization-specific benchmark generation and refresh. It also extends diversity assessment beyond wording to graph and query structure.

  • Recent Text2Cypher resources establish the importance of schema grounding, execution checks, verification, and complexity-aware evaluation.
  • PIPE-Cypher retains validation discipline but studies the process of generating, refreshing, auditing, and redacting benchmarks for an organization’s own graph and local endpoint.
  • Text-to-SQL research motivates execution as semantic feedback rather than a cosmetic scoring check.
  • PIPE-Cypher combines lexical diagnostics with schema coverage, relationship and property coverage, query-signature diversity, structural rates, and normalized entropy.

3 Method

PIPE-Cypher organizes benchmark generation into six stages, using reverse grounding and deterministic validation to ensure accepted examples are answerable and safe. A local judge reviews candidates only after execution evidence is available.

  • PIPE-Cypher has six stages: schema profiling, workload planning, reverse grounding, constrained generation and repair, validation and execution, and LLM-judge review.
  • Accepted examples must pass schema checks, parser-style structure extraction, live execution, and judge review rather than relying on prompt compliance.
  • Schema profiling records labels, relationship types, properties, observed directions, and bounded categorical values for eight operational workload categories.
  • Reverse grounding runs read-only Cypher to find slot values that actually produce rows, making candidate questions answerable.
  • Deterministic checks enforce read-only safety, syntax shape, schema validity, relationship direction, categorical values, and non-empty execution where required.

4 Implementation

The implementation packages graph profiling, local model endpoints, Cypher governance, execution traces, and audit/export tools into a backend-portable workflow. It supports privacy-aware generation, workload templates, prompt profiles, and guarded validation for enterprise onboarding.

  • PIPE-Cypher is a Python package built around a read-only graph client, schema/value profiling, local model endpoints, Cypher governance, and export/audit tools.
  • Generation and evaluation remain inside the organization’s compute boundary without paid generation APIs, while downstream evaluation covers 11 local model families.
  • FinBench and SNB profiles record graph-specific properties, relationship structures, and bounded categorical strings that constrain generation.
  • Reverse-binding queries fill workload-template slots with proven values, while placeholderized few-shot examples expose structure without repeatedly revealing tenant values.
  • Prompt profiles compare schema-only, instructions-only, examples-only, examples-plus-instructions, and fully governed generation under common evidence checks.
  • The judge sees only query-relevant schema elements, while deterministic validators continue checking candidates against the full schema.
  • Cypher governance enforces exact matching, relationship direction, RETURN DISTINCT, categorical values, contextual return columns, and parser-aware rewrite boundaries.
  • A launch guard checks executable slot coverage before scaling, but exported examples must still pass validation, execution, diversity, and judge gates.

5 Experiments

The experiments ask whether a local-model workflow can generate balanced, executable benchmarks, improve reliability through Cypher-specific controls, expose downstream failures, and onboard another graph.

  • Research questions: Four research questions test benchmark generation, governed reliability, downstream diagnostic value, and onboarding beyond FinBench or SNB.The experiments target both pipeline quality and whether the workflow generalizes across graph workloads.
  • Evaluation setup: Table 2 reports full live generation with local Qwen3.5-9B, including initial candidates and category-specific recovery top-ups.The table’s candidate counts include both the sequential run and recovery process.
  • Evaluation setup: 3,000 accepted examples span LDBC FinBench and LDBC SNB across eight workload categories.The evaluation reports 2,000 FinBench and 1,000 SNB examples, with three completed ablation suites.

6 Results

PIPE-Cypher produced a balanced, executable benchmark, enforced strong quality gates, and provided diversity and transfer diagnostics, including third-graph onboarding evidence with an explicit private-tenant boundary.

  • Benchmark generation: 3,000 accepted examples emerged from 4,925 candidates, and every export passed read-only, syntax, schema, execution, non-empty-result, and judge gates.Generation and judging used local Qwen3.5-9B models.
  • Diversity: The full export achieved perfect category balance, near-perfect difficulty balance, 1,115 unique grounded entity values, and 82.6% exact quotation of grounded values.Query-signature diversity remained low, while governed selection improved structural, lexical, signature, and schema coverage after quality gates.
  • Governed generation: Only 2 of 4,925 candidates were schema-invalid, while every non-unconstrained target-100 cell reached all eight category targets.Across three evidence-ready suites, target-normalized coverage was 1.000 for every non-unconstrained cell.
  • Downstream transfer: The no-signature control reached 0.200 mean accuracy, while ordered and random same-category banks reached 0.269/0.267 as operational upper bounds.The no-signature result excludes exact query-signature matches and near-duplicate questions.
  • Third-graph onboarding: ICIJ onboarding produced 800 accepted examples from 983 candidates with 100 examples per category, but this does not establish private-tenant coverage.The third graph exercised relationship-count, anti-join, and top-k templates beyond the two LDBC workloads.

7 Industry Use

PIPE-Cypher supports industry use by grounding benchmarks in enterprise-specific questions, exposing poor transfer to new schemas, and enabling held-out evaluation or private example banks while remaining refreshable and scoped beyond private-tenant proof.

  • Deployment grounding: Seed queries from customer questions, analyst logs, or agent tool calls can bridge deployment workloads and benchmark construction.The pipeline expands these seeds into balanced tests of expected graph operations.
  • Deployment grounding: General Text2Cypher models often do not transfer cleanly to a new graph with its own schema.This pattern appears in the downstream stress test and motivates schema-specific evaluation.
  • Operational uses: Accepted examples support both private held-out testing under organizational schema, values, and safety rules and schema-specific retrieval or adaptation.The paper does not claim a tenant-specific fine-tuning result.
  • Refresh and audit: Recorded schema snapshots, graph profiles, model identifiers, gates, execution samples, judge scores, difficulty features, and source runs support refresh and audit as graphs change.The workflow is intended to be rerun when products, sources, or business logic change.
  • Scope boundary: Validation on FinBench, SNB, and ICIJ keeps the private-tenant gap explicit.The reported validation spans public enterprise-style graphs rather than private customer graphs.

8 Conclusion

PIPE-Cypher presents Text2Cypher benchmarking as a private, repeatable workflow in which executable constraints, grounding, validation, execution, diversity diagnostics, and judging produce auditable benchmarks that reveal downstream failures.

  • Conclusion: PIPE-Cypher reframes Text2Cypher benchmarking as a private, repeatable enterprise workflow.The workflow combines executable Cypher checks, reverse grounding, execution evidence, diversity diagnostics, and a calibrated local judge.
  • Conclusion: The resulting benchmark is balanced, auditable, refreshable, and able to reveal downstream model failures that syntax-only evaluation would hide.The conclusion attributes these properties to the combined pipeline components.

Limitations

PIPE-Cypher’s evidence is bounded by proxy graphs, template concentration, restricted Cypher coverage, and limited judge calibration. The reported judge calibration is promising but remains uncertain beyond its small single-annotator sample.

  • The 80-row calibration used one human annotator; no false accepts were observed, but wider confidence intervals and larger audits may reveal additional failure modes.
  • FinBench, SNB, and ICIJ are public enterprise-style proxies, so onboarding is tested without covering every proprietary-tenant constraint.
  • The benchmark remains vulnerable to template concentration because seeded, execution-grounded generation produces low query-signature diversity.
  • PIPE-Cypher excludes writes, undirected relationships, UNION, CALL, UNWIND, and parser-risky rewrites, limiting coverage of production Cypher idioms.

Ethics Statement

PIPE-Cypher frames benchmark generation as an organization-run process that combines privacy controls, governance, validation, auditing, and refresh support. Its reported artifact and audits emphasize balanced coverage while keeping sensitive values under local policy.

  • Privacy and governance: PIPE-Cypher is designed for private generation, with authorized access, sampled-value review, human-labeling documentation, configurable value policies, and redacted exports.
  • Prior-work positioning: The organization-run factory assembles outcome-aware grounding, deterministic Cypher checks, local judge calibration, refresh support, and audit logs rather than providing only a static dataset.
  • Reliability and portability: Target-normalized coverage stayed complete across governed target sizes and seeds, and the third-graph ICIJ audit reached all eight category targets using schema-derived templates.
  • Validation: Every exported example passed read-only, syntax, schema, execution, non-empty-result, and judge gates, while most rejected candidates were duplicates, diversity failures, or empty results.
  • Distribution: The full export preserves a 2:1 FinBench/SNB mix, exact category balance, and a near-even easy/medium difficulty split.

E.1 Transfer Controls

The transfer study shows that execution-based evaluation exposes semantic failures missed by parse or schema validity, while schema-specific demonstrations help compatible model families but not universally. Diversity and strategy diagnostics further show that balanced categories do not ensure broad structural coverage, motivating novelty-based subset selection and strategy-aware analysis.

  • Downstream transfer: 0.963 parse validity and 0.916 schema validity contrast with 0.189 exact execution accuracy for Qwen3.5-9B on the full held-out split.The gap indicates that executable, schema-plausible queries can still answer the wrong operational question.
  • Downstream transfer: Zero-shot transfer is weak, while schema-specific demonstration banks improve compatible Qwen, Qwen-Coder, and one Gemma Text2Cypher LoRA family.Several public fine-tuned checkpoints remain brittle under enterprise-style prompting.
  • Downstream transfer: Example-bank gains are model-family dependent rather than universal, according to checkpoint-level uncertainty analysis.The uncertainty interval uses model checkpoints, not individual question rows, as the resampling unit.
  • Downstream transfer: The scored no-signature control excludes exact query-signature matches and near-duplicate questions, whereas ordered and random same-category banks serve as operational upper bounds.The leakage-aware control is stricter than banks that may share query signatures with test examples.
  • Diversity and strategy diagnostics: Category and difficulty balance do not guarantee schema coverage, query-signature diversity, or operator coverage across joins, paths, negation, ranking, and bounded-result patterns.The diagnostics therefore examine both benchmark composition and downstream outcomes by gold Cypher strategy.
  • Diversity and strategy diagnostics: An MMR-style selector improves diversity and coverage metrics while preserving a signature-disjoint 640/80/80 split, but residual template concentration remains.When cells have only one or two viable signatures, stronger diversity requires oversampling or more source templates during generation.

G Supplementary Evaluator Metrics

The supplementary evaluator materials define execution accuracy and answer-set F1 as the correctness measures, while treating text-overlap metrics as debugging aids. They also document a governed local workflow with deterministic validation, conservative rewrite auditing, redaction checks, prompt contracts, and post-hoc human calibration.

  • Evaluator metrics: Reference-based text metrics support debugging answer rendering, paraphrase sensitivity, and near-match behavior but do not replace execution accuracy or answer-set F1.BERTScore and FrugalScore are optional integrations requiring additional metric or model packages.
  • Governance and validation: The validator cascade keeps unsafe, schema-invalid, empty, or semantically weak examples out of the exported benchmark before judge review.The deployment workflow uses deterministic and judge gates in execution order.
  • Governance and validation: All reported-run generated queries were already identical to normalized Cypher, so accepted examples did not rely on semantics-changing rewrites.Rewriting remained available for conservative validation and logging rather than silent semantic alteration.
  • Governance and validation: The governance audit separates direction, schema/value, syntax/parser, and read-only failures, finding no direction failures in full generation records but direction errors in downstream predictions.This separation shows which Cypher-specific gates contribute to benchmark governance and which downstream failures remain exposed.
  • Governance and validation: The counterfactual gate audit identifies which failure classes would enter the benchmark if duplicate, diversity, execution, judge, schema, direction, or safety gates were weakened.This provides a failure-oriented complement to aggregate acceptance rates.
  • Privacy and deployment: The redaction audit exact-matches sensitive values across entity bindings, Cypher literals, reverse-grounding literals, and string-valued result samples after applying the configured policy.The audit provides a measurable privacy check but does not replace a tenant’s PII classifier.
  • Privacy and deployment: PIPE-Cypher versions prompt contracts for generation, repair, judging, and downstream evaluation, while human review serves post-hoc calibration rather than a production gate.The contracts specify schema grounding, read-only behavior, strict outputs, validation inputs, and judge scoring fields.
Loading 2606.08481v1…