Data Quality Rule Generation with LLMs

1 Introduction

• Errors absent from an existing rule set pass unnoticed into downstream processes, and manually defining and maintaining rules is complex and time-consuming.Such gaps remain an open practical problem.Source: §1 · p. 2
• LeDQeR generates candidate rules from an observed dirty tuple and rule format, then filters them for executability, correctness, generalizability, and redundancy.The framework is intended to produce a minimal, precise rule set for operational deployment.Source: §1 · p. 2
• The paper formalizes a generalizable generate-filter framework, introduces LeDQeR, and evaluates it on five datasets spanning diverse domains and up to nine error types.Source: §1 · p. 2
• LeDQeR generates executable rules that generalize to unseen data without false positives; results also vary with error type and the amount of prompt context.Source: §1 · p. 2

2 Data Errors and Rule-based DQ Measurement

• The Austrian Post personal contact information dataset supplies a running example for explaining the generate-filter framework and its evaluation.Source: §2 · p. 2, §2.1 · p. 2

2.1 Data Quality and Error Types

• Data errors depend on context and use; the evaluation selects nine error types spanning issues in single values or tuples.Source: §2.1 · p. 2

2.2 Rule-based Data Quality Measurement

• Rule-based DQ measurement evaluates explicit rules that encode expected data properties, but stewards may struggle to anticipate evolving domains and maintain comprehensive rules.Encoding semantic errors is also harder than encoding syntactic ones.Source: §2.2 · pp. 3–4
• Hypothetical GX rules illustrate mistaken expectations that flag valid values, accept errors, or fail to execute; the section notes that filters address such problems.Source: §2.2 · p. 4

3.1 Problem Formalization

• The generate-filter framework creates candidate rules from a dataset, then removes erroneous, redundant, or overlapping rules to improve the resulting set’s correctness and coverage.The framework formalizes rule generation and filtering as two steps.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 LeDQeR: An LLM-based Generate-Filter Framework to Generate DQ Rules

• LeDQeR instantiates this framework with an LLM candidate-generation stage and four complementary filters that progressively refine candidates into the final rule set.The rule sets are successively refined as C ⊇ C′ ⊇ C′′ ⊇ C′′′ ⊇ R.Source: §3.2 · p. 4, Figure 3 · p. 5

3.2.1 LLM-based DQ Rule Generation.

• For each dirty tuple, the LLM receives the record, target syntax, prompt instructions, and dataset description, then generates candidate rules that may vary in number from the record’s errors.The prompt also guides the LLM to detect errors, generate and check rules, and return them in a specified format.Source: §3.2.1 · pp. 4–5
• The input variants range from dirty data alone, the most realistic setting, to dirty data plus error-type information or clean examples, the latter serving as an evaluation upper bound.Dirty & Type applies when an error detector provides type information; Just Dirty requires the LLM to detect errors without additional information.Source: §3.2.1 · p. 5

3.2.2 Filtering pipeline.

• The syntactic filter retains executable rules, while the correctness filter rechecks each rule on its triggering tuple and removes rules that fail to detect its error.The correctness check distinguishes true positives, false negatives, and false positives.Source: §3.2.2 · p. 5
• The coverage filter tests recall across same-error-type training tuples, while the redundancy filter uses a separate test set to select high-recall rules at precision 1 and remove overlap.LeDQeR sets no minimum recall threshold, prioritizing maximum coverage despite the risk that low-recall specific rules may be removed depending on the threshold.Source: §3.2.2 · pp. 5–6

4 Experiments

• The evaluation analyzes rule generation and the syntactic, correctness, coverage, and redundancy filters, with datasets, metrics, and rule counts varying across experiments.Experiments ran on a high-performance computing platform with four NVIDIA A100 GPUs and 1TB RAM.Source: §4 · p. 6, §4.1 · p. 6

4.1 Experimental Setup

• Rules were generated for Great Expectations version 1.9.1, selected as a freely available Python-based tool with an active open-source community and industry partnerships.Great Expectations calls its rules “expectations.”Source: §4.1.1 · p. 6

4.1.2 Datasets.

• The evaluation used five datasets: beers, flights, hospital, and tax were adapted from prior work, while PCI was synthetically generated from real Austrian addresses and augmented with error types.The PCI dataset is for personal contact information.Source: §4.1.2 · p. 6, Table 2 · p. 6

4.1.3 LLMs.

• Three local LLMs span a lightweight general-purpose model, a code-specialized model, and a high-performance general-purpose model.The models were selected based on leaderboard rankings and compatibility with the computing infrastructure.Source: §4.1.3 · p. 6

• The models typically generated more than one rule per tuple, increasing the likelihood of overlapping or duplicated rules and motivating filtering.Source: §4.2 · p. 7
• 87.13%–100% of rules were executable with parameter information across models and datasets.The reported range depends on the model and dataset.Source: §4.2 · p. 7

• 0.33 higher overall performance with Dirty & Type than Just Dirty indicates that error-type and column information improves correctness-filter results.Results for Dirty & Type and Dirty & Clean also show successful rule creation with those input types.Source: §4.3 · p. 8
• Gemma-4 outperformed Qwen3-Coder on correctness, while GLM-4.7-Flash had the lowest performance among the three models.The authors suggest Gemma-4’s general reasoning capabilities may explain the difference.Source: §4.3 · p. 8
• Mean F1 was 0.66 overall, ranging from 0.4 for con to 0.87 for ebv.The authors attribute ebv’s success likely to errors detectable by regular expressions that affect a single value, while con requires cross-column rules.Source: §4.3 · p. 8

4.4 Evaluation of the Coverage Filter

• No minimum recall threshold is enforced, so all candidate rules proceed to the final redundancy-filtering stage.The choice prioritizes maximum coverage and avoids discarding highly specific rules.Source: §4.4 · p. 8
• Mean recall is 0.88 (std = 0.16), with similar generalizability across the three LLMs and input variants.Qwen3-Coder often performs worst; for con errors, Gemma-4 has the lowest generalization, which the authors link possibly to semantic complexity and overly specific rules.Source: §4.4 · p. 8, Figure 5 · p. 9
• Recall is lower for multi-variant error types: dov reaches 0.58, con 0.67, and spm 0.84.The Austrian-city example misses names containing umlauts, hyphens, or multiple words.Source: §4.4 · pp. 8–9

4.5 Evaluation of the Redundancy Filter

• On unseen data, most individual rules have high precision or recall, and a subset has both.These individual-rule results align with the coverage-filter findings, while indicating that some dirty tuples may be missed.Source: §4.5 · p. 9
• At precision 1, the final rules achieve at least 50% coverage except on tax; the tax dataset reaches 99.58% coverage with up to 6 false positives (0.007%).The final rule sets contain between 2 and 136 rules.Source: §4.5 · p. 9
• Relaxing the threshold raises coverage at the cost of false positives: flights gains 9% coverage with +6.7% false positives.The trade-off is consistent across LLMs on beers, flights, and tax; Gemma-4 has the most favorable trade-off on hospital and PCI, followed closely by GLM-4.7-Flash and Qwen3-Coder.Source: §4.5 · p. 9

4.6 Summary of Results

• Syntactically correct rules have executability rates from 87.13% to 100% when prompts include rule names and valid parameters.Purposeful filtering reduces the rules to a small set that generalizes to unseen data.Source: §4.6 · p. 9
• Providing the error type and column rather than only dirty values increases average F1 by 0.33; mean F1 is 0.66, while contradictions reach 0.4.Figure 6 presents per-rule precision and recall on unseen datasets, ordered by F1 score.Source: §4.6 · p. 9, Figure 6 · p. 10
• Mean recall is 0.87 across error types, but is lower for multi-variant dmv (0.58) and con (0.67).Figure 7 presents the redundancy-filter coverage of clean and dirty tuples across datasets, input variants, and LLMs.Source: §4.6 · p. 10, Figure 7 · p. 10
• At precision 1, reduced rules cover at least 50% of dirty tuples on four of five unseen datasets; in tax, 6 false positives raise coverage by about 70 percentage points to near-complete.The tax result illustrates the adjustable precision–recall trade-off for data stewards.Source: §4.6 · p. 10, Figure 7 · p. 10

5 Related Work

• Unlike prior ML rule-learning work on correcting classifiers or characterizing black-box errors, LeDQeR targets rules executable by DQ tools on structured data.The cited foundations include boosting, Patching, and rule learning for NLP systems.Source: §5 · p. 11
• Database approaches to DQ-rule discovery focus largely on combinatorial integrity constraints and functional dependencies, which suit structural constraints better than semantic validation.The text cites Chiang and Miller and Yeh and Puri; newer work explores LLMs for higher-level semantic constraints.Source: §5 · p. 11
• Other LLM work includes generating tabulated features and database curation with Chat2Data.The closest compared frameworks generate complete rule sets from anticipated errors, unlike LeDQeR's observed-error-driven approach.Source: §5 · p. 11, Table 4 · p. 11
• Xie et al. and Schneider require manually specified requirements, whereas LeDQeR can use dirty tuples alone; differing inputs and rule syntax make direct quantitative comparison not meaningful.LeDQeR iteratively refines an existing rule base from a small number of examples.Source: §5 · p. 11

6 Conclusion

• LeDQeR generates candidate rules from dirty tuples for GX, then filters them for executability, correctness, generalizability, and redundancy.The framework targets cases where a dirty tuple passed validation unnoticed.Source: §6 · p. 11
• Across five datasets, LeDQeR generated correct, generalizable rules from dirty tuples alone.Additional information about the error or a clean tuple further improved rule quality.Source: §6 · p. 11
• Error type and location alone significantly improved performance, reducing the need for clean data; dirty tuples alone could suffice for less complex errors.Providing a clean tuple or other error information further improved rule quality.Source: §6 · p. 11
• LeDQeR performed consistently across LLMs, while higher-capacity general-purpose models outperformed lightweight or coding-specialized models as error complexity increased.The authors suggest general reasoning matters more for rule correctness than specialized coding proficiency.Source: §6 · p. 11
• Initial rule correctness and generalizability occasionally varied inversely, underscoring the role of filtering a wide range of candidate rules.The filters remove low-quality rules before they enter the final rule set.Source: §6 · p. 11
• Future work will assess natural-language rule descriptions and methods for automatically detecting a dirty tuple’s error type.Manual reviews found occasional useful descriptions without explicit prompting; providing error type already improved generation performance.Source: §6 · p. 12

Artifacts

• LeDQeR’s implementation code is available in a public GitHub repository.Source: Artifacts · p. 12
• Code to generate the PCI dataset and the other four polluted, labeled datasets are publicly available.Source: Artifacts · p. 12
• The LLMs used in the experiments are available via HuggingFace.Source: Artifacts · p. 12