PaperlayerView exact revision on arXiv

Source-linked AI summary

CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records

Suparno Roy Chowdhury, Manan Roy Choudhury, Dhruv Madhwal, Vivek Gupta

arXiv:2609.07022v1cs.CRcs.AI

TL;DR

Questions over privacy-constrained records require evidence from both structured attributes and free-text narratives, but existing resources only partially evaluate this setting. CIPHER benchmarks these questions with executable supervision and evaluates systems across evidence conditions; systems remain unreliable even when supporting records are provided, with failures centered on selection and predicate interpretation.

  • Problem

    Hybrid questions in privacy-sensitive records require grounding constraints across structured fields and narratives, while existing resources only partially evaluate this setting.

    Source: §1 · p. 1

  • Method

    CIPHER pairs questions from three record domains with executable SQL, reference answers, and supporting-record provenance, then compares system families across redacted and surrogate-restored evidence conditions.

    Source: §1 · p. 2

  • Results

    Systems remain substantially unreliable even with supporting records, with errors arising mainly from record selection and narrative-predicate interpretation rather than arithmetic execution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CIPHER provides a testbed for diagnosing evidence-use failures and assessing how transformations of sensitive text affect reasoning over hybrid records.

    Source: §1 · p. 2, §7 · p. 8

    Redacted-versus-restored results use Llama 3.3 70B only, so whether restoration effects depend on the backbone requires comparisons across models.

    Source: §8 · p. 9

Abstract

from arXiv · show

Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cross-record Inference over Privacy-Hardened Evidence Records), a benchmark of expert-validated questions from consumer-finance, clinical, and law-enforcement records. The questions cover common tabular operations and include executable SQL supervision. We evaluate retrieval, prompting, table-specialist, and hybrid symbolic-neural systems under native redaction and surrogate-based evidence restoration. All system families exhibit substantial failures even when supporting records are provided. Most errors arise from incorrect record selection and predicate interpretation rather than arithmetic execution. Privacy transformations have non-uniform effects, sometimes obscuring necessary evidence and sometimes reducing distraction. CIPHER provides a reproducible testbed for diagnosing these failures and assessing how transformations of sensitive text affect reasoning over hybrid records.

1 Introduction

CIPHER evaluates reasoning over privacy-constrained records that require combining structured attributes with narrative evidence. Its results show persistent failures in record selection and predicate interpretation, even when supporting records are available.

1 Introduction

  • CIPHER contains 3,427 questions from consumer-finance, clinical, and law-enforcement records, covering filtering, aggregation, ranking, trend analysis, and co-occurrence.Every question requires both structured and narrative evidence.Source: §1 · p. 2
  • Each question pairs executable SQL with a reference answer and supporting-record provenance, enabling answer regeneration, controlled evidence contexts, and failure tracing.These annotations are retained for evaluation but withheld from models.Source: §1 · p. 2
  • Aligned native-redaction and surrogate-based surface conditions support within-domain comparisons of how visible text changes affect model behavior; surrogates are not recovered private values or formal privacy guarantees.Transformations may obscure relevant evidence or remove distracting cues, with effects that are not necessarily uniform across domains or systems.Source: §1 · pp. 1–2
  • The evaluation compares retrieval, prompting, table-specialist, and hybrid symbolic–neural systems in support-complete contexts and full-table evidence-selection experiments.It uses answer-level metrics and a diagnostic failure taxonomy.Source: §1 · p. 2
  • Systems remain unreliable even with supporting records; errors primarily reflect irrelevant-record selection and narrative-predicate misinterpretation, while privacy transformations can help some configurations and hurt others.Fixed-depth retrieval is particularly weak on questions requiring evidence from several records.Source: §1 · p. 2

2 Related Work

Prior tabular and hybrid QA resources largely use clean, regular web-derived data, while newer work examines scale and reasoning complexity. CIPHER combines institutionally redacted narratives with structured records and executable supervision that supports tracing errors to evidence-use stages.

2 Related Work

  • Tabular QA benchmarks such as WikiTableQuestions, WikiSQL, and Spider [15, 17, 19] emphasize semantic parsing and text-to-SQL, while hybrid datasets extend reasoning across tables and text [21, 16, 22, 1].These datasets typically use regular schemas and low-noise data, where annotation artifacts or distributional overlap may support performance [20].Source: §2 · p. 2
  • RUST-BENCH [24] and long-context or tool-based studies [25] examine degradation with larger tables, heterogeneous domains, or longer reasoning chains, but not privacy-driven effects.These studies leave the effects of privacy transformations on performance unexamined.Source: §2 · p. 2
  • CIPHER combines noisy, institutionally redacted narratives with structured attributes across three domains, unlike resources centered on cleaner web-derived tables.The resource pairs each question with executable supervision and supporting-record provenance to trace errors to evidence-use stages rather than only scoring answers.Source: §2 · pp. 2–3

3 Task Definition

CIPHER defines hybrid questions through constraints grounded in both structured attributes and narrative text, with executable specifications supporting reference answers and provenance. Its aligned redacted and restored conditions compare reasoning over different visible evidence while holding questions and record identifiers fixed.

3.1 Hybrid Questions

  • A hybrid question requires at least one constraint grounded in structured attributes and one grounded in narrative text, and may involve one or several qualifying records.Questions cover filtering, aggregation, ranking, trend analysis, and co-occurrence; executable SQL regenerates the reference answer and supporting-record identifiers.Source: §3.1 · p. 3
  • Each question pairs with executable SQL and a construction-only annotation view used for reference generation, support-controlled contexts, and evaluation; these annotations are not shown to systems.The construction pipeline pairs source records with question and SQL templates, then reviews, revises, or removes instances.Source: §3.1 · p. 3

3.2 Redacted and Restored Evidence

  • Redacted and restored conditions hold questions and record identifiers fixed, but may differ in visible collections, answers, and supporting records; paired comparisons include only questions supported in both conditions.Surrogates are plausible replacements, not recovered private values, and the comparison measures sensitivity to visible information rather than privacy protection or re-identification risk.Source: §3.2 · p. 3

4 The CIPHER Benchmark

CIPHER draws on law-enforcement, consumer-finance, and clinical records with different schemas, narrative styles, lengths, and de-identification provenance. The benchmark reports varied question and answer distributions, while template-family review retains most templates across domains with high reviewer agreement.

4.1 Domains

  • The benchmark uses 322 law-enforcement reports, 400 consumer-finance complaints, and 500 clinical discharge notes, with domain-specific schemas, narratives, and de-identification provenance.FIR reports are processed using OCR, layout analysis, and field extraction; MIMIC materials remain subject to credentialing, licensing, and redistribution requirements.Source: §4.1 · p. 4

4.2 Benchmark Statistics

  • CIPHER contains 3,427 questions—1,571 FIR, 1,356 CFPB, and 500 MIMIC—with average narrative lengths from 630 words in FIR to 1,018 in MIMIC.FIR and CFPB have substantial proportions of list-valued answers, whereas MIMIC is dominated by numeric and textual outputs.Source: §4.2 · p. 4, Table 1 · p. 4

4.3 Template Audit

  • Across domains, 88.0% to 94.5% of templates are retained through direct approval or revision, and pairwise reviewer agreement before adjudication ranges from Cohen’s κ = 0.93 to 0.96.Reviewers independently inspect generated variants, source records, SQL, reference outputs, and supporting-record identifiers; disagreements are adjudicated against question details and execution traces.Source: §4.3 · p. 4, Figure 3 · p. 4, Table 2 · p. 5

5 Experimental Setup

The experimental setup compares model and system families for hybrid-record reasoning, then evaluates them using controlled evidence contexts, complementary answer metrics, and a validated semantic judge.

5.1 Models

  • Five checkpoints span proprietary and open-weight model families, and evaluation fixes greedy decoding at temperature 0.0 with an 8,192-token output limit.Questions and record serializations stay constant across configurations when interfaces permit; exact checkpoints, API versions, prompts, and parameters are released.Source: §5.1 · p. 5

5.2 Baselines

  • The baseline suite covers direct and few-shot prompting, explicit reasoning, retrieval-augmented generation, hybrid symbolic–neural systems, and table specialists.Reasoning baselines include CoT, PoT, and LtM; retrieval uses embedding or cross-encoder ranking, while WEAVER and BlendSQL combine language-model reasoning with symbolic table operations or SQL-like programs.Source: §5.2 · p. 5
  • TableGPT2 combines a dedicated table encoder with a language model, while TAT-LLM uses a step-wise extract, reason, and execute pipeline for tabular and textual input.Both are complete systems and report one score per dataset rather than scores across backbone models.Source: §5.2 · p. 5

5.3 Evaluation

  • The primary evaluation supplies all specification-identified supporting records with same-domain distractors, separating missing-evidence failures from errors in evidence use.Distractors are checked not to change the reference output; the complementary full-table setting evaluates evidence selection over the complete candidate collection.Source: §5.3 · pp. 5–6
  • M3 combines lexical overlap and semantic acceptance per question, giving credit only when the judge accepts an answer and its content agrees with the reference.M1 measures lexical overlap, M2 is judge-produced semantic acceptance, and the study retains both alongside M3 because they can disagree.Source: §5.3 · p. 6
  • Human decisions on 750 outputs agreed with Gemini 2.5 Flash at Cohen’s κ of 0.91 overall, with domain-level values from 0.88 to 0.93.Two NLP researchers independently judged outputs sampled evenly across CFPB, FIR, and MIMIC; M2 remains complementary because the judge may favor wording or model families.Source: §5.3 · p. 6

6 Results and Analysis

Across three analyses, systems remain unreliable even when required records are provided, surrogate restoration has non-uniform effects, and failures cluster in record selection and predicate interpretation. The section compares support-controlled and full-table settings, evidence conditions, and operation-specific failure patterns.

6.1 RQ1: How well do current systems reason over hybrid records?

  • 29.6 prompting average M3 exceeds 25.2 for table-based methods and 2.8 for retrieval; RAG and RAG+Rerank average 2.9 and 2.7 at k = 5.Reranking shifts each domain average by less than half a point and never overtakes the bi-encoder.Source: §6.1 · pp. 6–7
  • 26.5 is the median M3 across 135 configurations; 85 (63.0%) score below 30, none reaches 50, and the maximum is 48.6.The support-controlled inputs include required records and same-domain distractors.Source: §6.1 · p. 6, Figure 4 · p. 7
  • Gemini 2.5 Pro leads Qwen 3-80B by 34.8 and 37.1 M3 points under BlendSQL on CFPB and FIR, respectively.This comparison uses a 30% sample; semantic acceptance drives the gap.Source: §6.1 · p. 7

6.2 RQ2: Does restoring masked surface information consistently improve reasoning?

  • CFPB mean M1 rises 5.3 while M2 falls 7.6; MIMIC improves in six of seven configurations, while BlendSQL gains 14.5 on FIR but loses 11.6 on MIMIC and 10.4 on CFPB.Plausible replacements may introduce distracting entities into institutionally de-identified complaints; restoration effects depend partly on how methods convert narrative content into record-level decisions.Source: §6.2 · pp. 7–8
  • 12 of 21 configurations improve, eight decline, and one is unchanged; mean M3 changes are +7.6 for FIR, +3.7 for MIMIC, and −5.5 for CFPB.Figure 5 compares native redacted with surrogate-restored evidence for Llama 3.3 70B.Source: §6.2 · p. 7, Figure 5 · p. 8

6.1 RQ1: How well do current systems reason over hybrid records?

  • On complete CFPB and FIR collections, Gemini 2.0 Flash never exceeds 42.0 M3; CoT leads CFPB at 42.0 and LtM leads FIR at 39.1.Failures arise both in locating relevant records and interpreting narrative conditions that determine whether they qualify.Source: §6.1 · p. 7, Table 3 · p. 8

6.3 RQ3: Where do current systems fail?

  • Evidence-selection errors dominate aggregation (72.5%), ranking (73.8%), trend (50.4%), and co-occurrence (48.1%); filtering most often fails at predicate interpretation (40.4%).The labels come from majority-failed questions in a 30% domain-stratified sample, with one dominant category per question.Source: Figure 6 · p. 8, §6.3 · p. 8

7 Conclusion

CIPHER benchmarks language-model reasoning over privacy-constrained records combining structured fields and free-text narratives. Its analyses find that failures center on selecting records and interpreting predicates, while privacy transformations can either obscure evidence or reduce distraction.

7 Conclusion

  • CIPHER is a benchmark for language-model reasoning over privacy-constrained records that combine structured fields and free-text narratives.It covers three real-world domains and five analytic operation types, with executable supervision.Source: §7 · p. 8
  • Systems fail mainly when selecting records and interpreting predicates, even when supporting evidence is present.The analysis identifies these as dominant failure modes rather than failures of final computation.Source: §7 · p. 8
  • Privacy transformations have method- and domain-dependent effects: they may remove necessary cues or suppress misleading ones.CIPHER is presented as a foundation for developing models that reason reliably over sensitive records.Source: §7 · p. 8

8 Limitations

The conclusions are bounded by model, setting, and answer-type coverage. The paper also notes that template-level review does not guarantee every instantiated question is free of residual errors.

8 Limitations

  • The redacted-versus-restored comparison uses Llama 3.3 70B only, leaving possible backbone dependence untested.The authors say establishing this requires repeating the comparison across models.Source: §8 · p. 9
  • Baseline and setting coverage is partial: H-Star, Binder, and large reasoning models are absent, while full-table evaluation covers one model and two domains.Scaling the full-table setting further requires specialized pipelines.Source: §8 · p. 9
  • CIPHER omits open-ended explanation, justification, and long-form summarization, and template-level review cannot ensure every instantiated question is error-free.Future work plans to extend the analysis along these dimensions.Source: §8 · p. 9

9 Ethics Statement

CIPHER is presented as a diagnostic research benchmark, with explicit safeguards around its data, use, and interpretation. The appendix also documents the prompting and judging instructions used in the evaluation.

9 Ethics Statement

  • CIPHER uses public, de-identified, or appropriately licensed sources, and its redacted and unredacted settings are evaluation conditions—not deployment recommendations.The authors state that strong benchmark performance is not certification for real-world use in high-stakes legal, financial, or medical settings.Source: §9 · p. 9
  • The benchmark’s institutional, geographic, and linguistic scope may carry biases from its source data; the authors encourage extensions across jurisdictions, languages, and populations.They plan to examine these dimensions more granularly in future work.Source: §9 · p. 9

Appendix

  • The appendix collects the prompts used for prompting baselines, information extraction, LLM judging, and the complete benchmark results.The supplied prompt passages specify how systems are instructed to process records and answer questions.Source: Appendix · p. 12

A.1 Prompt A: Chain-of-Thought

  • The chain-of-thought prompt treats redaction tokens as opaque strings and asks for a concise one-word or phrase answer.It frames the model as a tabular data-analysis assistant and notes that the narrative field is unstructured.Source: §A.1 · p. 12

A.2 Prompt B: Few-Shot

  • The few-shot prompt instructs models to use narrative and structured columns together to identify relevant rows and return answers in the required format.It supplies examples and allows NONE or an empty list when the table cannot answer the question.Source: §A.2 · p. 12

A.3 Prompt C: Least-to-Most

  • The least-to-most prompt decomposes each question into simpler sub-questions, solves them in sequence using rows and columns, then produces a concise final answer.It specifies NONE or an empty list when the table cannot answer the question.Source: §A.3 · p. 12

A.4 Prompt D: Program-of-Thought

  • The program-of-thought prompt asks for a short Python 3 program over a conceptual table named temp, with column names matching the header row.The prompt identifies narrative interpretation and returning the requested value as common requirements.Source: §A.4 · pp. 12–13

A.5 Prompt E: Information Extraction

  • The information-extraction prompt requires a schema-conforming JSON object and directs models not to guess values that are not grounded in the narrative.It specifies null, false, or an empty list for unavailable scalar, Boolean, or list values.Source: §A.5 · p. 13

B Complete Benchmark Results

Across the complete benchmark, prompting performs best on average, hybrid table reasoning follows, and retrieval performs poorly. Results vary substantially by reasoning strategy and dataset, while the appendix reports a separate sampled model comparison and annotation procedures.

B Complete Benchmark Results

  • Prompting averages 29.6, hybrid table reasoning 25.2, and retrieval 2.8 across datasets and backbones, with retrieval weakest throughout.The complete results are reported in Table 5.Source: B Complete Benchmark Results · p. 13, Table 5 · p. 15
  • The five general-purpose models span barely two and a half points, while method choice matters more than backbone choice.GPT-OSS-20B exceeds GPT-OSS-120B on MIMIC (24.3 against 23.1) but trails it on FIR (19.2 against 20.4) and CFPB (21.7 against 26.4).Source: B Complete Benchmark Results · pp. 13–14
  • The strongest configurations overall are Qwen3-next with least-to-most on FIR (48.6) and CFPB (44.8), while no configuration among the 135 reaches 50.Least-to-most swings 39.0 points on FIR, whereas chain-of-thought has a spread of at most 12.5 points within a dataset.Source: B Complete Benchmark Results · p. 14

B.1 Gemini 2.5 Pro vs. Qwen 3-80B

  • On a random 30% sample of CFPB and FIR, Gemini 2.5 Pro is consistently stronger than Qwen 3-80B under BlendSQL and chain-of-thought; these sampled scores are not directly comparable with Table 5.For BlendSQL, semantic acceptance is 77.0% on CFPB and 87.0% on FIR for Gemini 2.5 Pro, versus 20.7% and 38.4% for Qwen 3-80B.Source: §B.1 · p. 14, Table 4 · p. 14

C Annotation and Validation Guidelines

  • The annotation protocol uses independent review of a subset, Cohen’s κ for categorical decisions, adjudication of disagreements, and template revisions when review identifies systematic issues.The release includes the rubric, validation scripts, agreement statistics, and workflow documentation for reproduction and extension.Source: C Annotation and Validation Guidelines · p. 15

D LLM-as-a-Judge Validation

The paper validates its LLM judge against human decisions and treats judge acceptance as one of several correctness signals. It also assesses surrogate replacements for fluency, consistency, and preservation of meaning.

D LLM-as-a-Judge Validation

  • The judge grants credit for semantically equivalent answers despite differences in wording or formatting, while withholding it for hallucinations, contradictions, or mismatched quantities.M2 is the judge’s semantic acceptance rate.Source: D LLM-as-a-Judge Validation · p. 16
  • Human reviewers agree with the judge at Cohen’s κ = 0.91 overall, with domain values between 0.88 and 0.93.Two NLP researchers reviewed 750 outputs, evenly divided across the three domains; Table 7 reports the agreement.Source: D LLM-as-a-Judge Validation · p. 16, Table 7 · p. 16
  • Although one evaluated system shares the judge’s model family, the paper treats M2 as one signal among three rather than as a standalone correctness measure.The authors note that the checkpoints differ and that agreement is high across all three domains.Source: D LLM-as-a-Judge Validation · p. 16

E Quality of the Surrogate De-redaction

  • Reviewers assess surrogates for fluency, semantic consistency, and matching intent; agreement is highest for matching intent at 94.6%.Agreement is 84.4% for fluency and naturalness and 87.1% for semantic consistency; surrogates are plausible substitutes, not recovered private values.Source: E Quality of the Surrogate De-redaction · pp. 16–17, Table 8 · p. 16

F Redacted vs Unredacted table

Restoration changes performance unevenly across domains and methods. Its effect on semantic acceptance can diverge from lexical overlap, and the combined score follows semantic acceptance in those cases.

F Redacted vs Unredacted table

  • Mean ∆M3 is +7.6 for FIR, +3.7 for MIMIC, and −5.5 for CFPB; among 21 method–domain cells, 12 improve, 8 decline, and 1 is unchanged.The unchanged CoT/FIR cell is +0.0.Source: F Redacted vs Unredacted table · p. 17
  • Lexical overlap rises in 18 of 21 cells, but semantic acceptance moves in both directions and disagrees with M1’s sign in eight; M3 tracks M2 in those cases.On CFPB, CoT, least-to-most, and few-shot gain lexical overlap while losing 2–20 points of judge acceptance, and M3 shows the loss.Source: F Redacted vs Unredacted table · p. 17
  • PoT and WEAVER improve in all three domains, while BlendSQL gains 14.5 points on FIR but loses 11.6 on MIMIC and 10.4 on CFPB.Overall, restoration is net positive for FIR and MIMIC and net negative for CFPB; its effect varies by method.Source: F Redacted vs Unredacted table · p. 17

Related papers

Loading 2609.07022v1…