PaperlayerView exact revision on arXiv

Source-linked AI summary

RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules

Ádám Kovács, Nadia Verdha, Gábor Recski

arXiv:2607.01293v1cs.CL

TL;DR

RuleChef addresses the labor-intensive hand-authoring of transparent NLP rules by using LLMs to synthesize and iteratively refine executable rules from supervision and feedback. Its preliminary evaluations show transparent systems can be competitive with black-box approaches on regular entities, while producing high-precision rules with substantial recall on harder entities.

  • Problem

    Writing and maintaining transparent, deterministic NLP rules remains labor-intensive and requires substantial domain expertise.

  • Method

    RuleChef uses LLMs at learning time to synthesize executable rules from supervision and iteratively refine them using held-out failures and human feedback.

  • Results

    RuleChef produces transparent rule-based systems competitive with black-box approaches for regular entities and high-precision rules with substantial recall for harder entities.

  • Takeaways & Limitations

    RuleChef can support explainable, lower-cost NLP systems, including hybrid systems that use its rules as a basis for subsequent improvements.

  • Takeaways & Limitations

    The quantitative results are based on single runs and fixed splits, with reported F1 variation of approximately ±3 points across repeated runs.

Abstract

from arXiv · show

We present RuleChef, a framework that uses large language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules. RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2.0

1 Introduction

RuleChef uses LLMs at learning time to synthesize and refine executable symbolic rules from supervision, while inference remains deterministic and inspectable. It addresses the labor and expertise required to hand-write rules through held-out evaluation, iterative patching, and human-editable rule systems.

  • Motivation: Explicit rules remain attractive for recurring NLP patterns because they are cheap, deterministic, versionable, and easy to inspect.
  • Problem: Hand-writing and maintaining rules is labor-intensive and requires substantial domain expertise.
  • Approach: RuleChef translates task supervision into executable symbolic rules, using LLMs during learning but not at inference time.Supervision can include examples, corrections, free-text feedback, or observed model behavior.
  • Approach: Its refinement loop evaluates rules on held-out data, clusters failures, accepts quality-improving patches, and resolves conflicts using held-out precision.
  • Evaluation: The paper evaluates learned rules on two tasks against prompting the same LLM and a dedicated neural extractor, including an ablation over one-shot rule prompting.Additional experiments examine human-in-the-loop repair and learning rules from an external model’s behavior.

2 Related Work

RuleChef relates to prior work on automatic rule and regex synthesis, interactive rule systems, weak supervision, and LLM-based code and rule generation. It distinguishes itself by making synthesized rules the final, explainable model rather than requiring substantial manual authoring or using rules only to generate noisy labels.

  • Automatic rule and regex synthesis: Prior automatic regex synthesis uses evolutionary methods, neural sequence-to-sequence models, or natural-language descriptions with positive and negative examples.The reviewed methods include work by Bartoli et al., Locascio et al., Zhong et al., and Chen et al.
  • Interactive rule systems: Interactive systems such as HEIDL, GrASP, Odinson, and POTATO support pattern construction over text and semantic or syntactic graphs but still require substantial manual authoring.Representations include Abstract Meaning Representations and Universal Dependencies.
  • Weak supervision: Weak supervision systems such as Snorkel and Snuba use labeling functions to create training labels, while RuleChef makes rules the final model rather than noisy-label sources for a downstream classifier.Recent work also uses LLM prompts as labeling functions.
  • LLM-based code and rule generation: Related LLM-based approaches synthesize executable programs from examples, survey LLM-based rule and hypothesis generation, and frame RuleChef as a symbolic alternative for LLM knowledge distillation.The paper characterizes this alternative as maximizing explainability and minimizing inference cost.

3 The RuleChef Framework

RuleChef synthesizes executable rules from task specifications and examples, then refines and validates them using held-out development performance and feedback. At inference, deterministic rule execution can replace or complement an existing model.

  • Task definition and synthesis: RuleChef supports classification, NER, untyped-span extraction, and structured-output transformation, with regex, spaCy, or Python rule formats, though evaluation focuses on regexes.Its synthesis prompt combines schemas, examples and corrections, data evidence, and response-format instructions; multiclass tasks use separate prompts per label with counter-examples.
  • Task definition and synthesis: Generated rules are augmented by optional grex pattern suggestions and validated for compilation and task-schema compatibility before acceptance.Grex patterns provide structural hints rather than constraints, helping the LLM identify regularities without overfitting.
  • Refinement loop: The refinement loop clusters failures, samples instances from each cluster, and lets the LLM modify, add, or delete rules using training data while held-out performance controls patch acceptance.Explicit user corrections remain in training as high-value patching signals, while the development split is withheld for evaluation.
  • Execution and conflict resolution: Rule conflicts are resolved by priority and development-set precision, with Wilson lower bounds discounting low-support estimates and leave-one-out analysis removing rules with insufficient marginal contribution.On TAB, case_and_echr_numbers achieved 0.86 precision with 22 true-positive development matches, whereas QUANTITY had precision 1.00 from only two matches and was discounted.
  • Observation mode: Observation mode treats an existing LLM’s production calls as training examples, periodically synthesizes rules, and routes matched queries to rules instead of the LLM.This enables rule bootstrapping without upfront labels and can restrict deployment to rules exceeding a measured-precision threshold.

4 Experimental Setup

The experimental setup evaluates RuleChef on NER and text classification using TAB and Banking77. Experiments use chunked and full-document TAB evaluation, while Banking77 includes Observation mode and established accuracy context.

  • Task coverage: Experiments cover two common NLP tasks: Named Entity Recognition and text classification.The paper describes these as preliminary evaluations of RuleChef.
  • Named Entity Recognition: TAB contains 1,268 European Court of Human Rights decisions annotated for personal-information spans across eight official entity types.The types are PERSON, CODE, DATETIME, QUANTITY, ORG, LOC, DEM, and MISC.
  • Named Entity Recognition: TAB entities are additionally grouped into format types and semantic types according to whether surface form or meaning governs recognition.Format types are CODE, DATETIME, and QUANTITY; semantic types are PERSON, ORG, LOC, DEM, and MISC.
  • Named Entity Recognition: Because full documents exceed LLM context windows, TAB documents are segmented into chunks of at most 600 characters, with 1,000 training and 600 test chunks sampled.RuleChef is evaluated on this chunked test set and the official split of 127 full, unchunked documents, using Kimi-K2.6, an agentic coordinator, and three refinement iterations.
  • Text classification: Banking77 contains over 13k customer-service queries classified by user intent into 77 categories, including keyword-detectable and challenging classes.RuleChef is also evaluated in Observation mode, where system-generated labels provide the only supervision for rule generation; coverage, precision, and replaced LLM calls are measured each iteration.
  • Text classification: 86–87% accuracy is reported for fine-tuned dual-encoder and contrastive models on the full 77-class task with 10 shots per class.This provides context for the Banking77 experiments.

5 Results

RuleChef’s iterative refinement produces readable, deterministic rule systems that outperform baselines on surface-form entity types while remaining weaker on challenging semantic types. Results also show benefits from holdout validation, human feedback, and observation-mode learning, though recall remains a limitation.

  • 5.1–5.3 TAB results: Learned rules outperform direct prompting and GLiNER2 on surface-form entity classes, while trailing direct prompting on challenging types but achieving the highest precision overall.The evaluation covers 600 test chunks from the TAB dataset and compares RuleChef with direct LLM prompting and two GLiNER2 baselines.
  • 5.1–5.3 TAB results: Fewer than 20 LLM calls and approximately 12 minutes were required for the rule-learning process.
  • 5.3 Ablation: Iterative feedback and additional examples substantially improve semantic-type rules, while holdout acceptance increases performance for both format and semantic types.One-shot prompting is nearly useless for semantic types but produces format rules within 5–10 F-score points of the best systems; refinement can degrade format performance before holdout filtering.
  • 5.3 Human-in-the-loop repair: 92 seconds and two LLM calls raised QUANTITY F1 from 5.7 to 35.6 in one human-in-the-loop repair round, with modest gains on two other criticized classes.The repair targeted fraction-shaped case numbers, missed bare application numbers, and PERSON matches on ordinary abbreviations.
  • 5.4 Qualitative analysis: Rule examples encode formatting conventions, entity lists, and contextual patterns, but list- and format-based rules vary substantially in precision.Examples include date formats, honorifics, court and government names, and location-context patterns.
  • 5.5 Banking77: With 10 to 50 observed Banking77 examples, precision rises from .93 to .96 and recall from .19 to .50, supporting use as a first tier in hybrid systems.RuleChef is competitive in precision with prompting and supervised methods but not in recall, a typical rule-based limitation.

6 Discussion and Future Work

RuleChef constructs transparent, iteratively refined rule-based systems for NLP using annotated examples and human feedback, achieving competitive performance for regular entity patterns and high-precision rules for harder entities. The authors characterize the evaluation as preliminary and identify domain variation, limited experimental validation, and prompt dependence as future-work concerns.

  • Contributions: RuleChef learns transparent rule-based systems through iterative refinement from annotated examples and human feedback across NLP tasks.The framework was tested primarily on generic Named Entity Recognition.
  • Findings: For entities with highly regular surface patterns, RuleChef produces fully transparent systems competitive with standard black-box approaches.For more challenging entity groups, it can instead produce high-precision rules.
  • Evaluation: RuleChef evaluates iterative learning strategies including gradual example introduction, performance-improving refinements, and agentic coordination by an external model.The coordinating model selects learning steps using observed rule performance and perceived rule quality.
  • Release: RuleChef is released as open-source software under an Apache 2.0 license.The release includes the framework’s rule-learning approaches for NLP.
  • Limitations and Future Work: The empirical results are strictly preliminary, and feature utility is expected to vary substantially across domains, genres, and datasets.The authors call for in-depth experimental evaluation of the main approaches and note that LLM rule-synthesis quality depends on prompts and separating training signals from failures.

Limitations

The evaluation has methodological and scope limitations: results use single runs and fixed splits, while experiments are restricted to English texts and one LLM family for rule generation.

  • Evaluation design: Results are based on single runs and fixed data splits rather than repeated-run averages.The paper reports metrics from representative experiments instead of means over multiple runs.
  • Evaluation design: ±3 F1 points of variation appears across repeated runs, even for the more regular FORMAT entity group.This variation is reported for selected configurations.
  • Evaluation scope: All experiments use English texts and a single LLM family for rule generation.

A Prompt Templates

RuleChef uses eleven human-editable prompt templates exclusively during learning, organized across rule synthesis and refinement, agentic coordination, observation mode, and auxiliary functions. Only the LLM fallback prompt operates at inference time, and only when no rule fires and delegation is configured.

  • A Prompt Templates: Eleven prompt templates span rule synthesis, refinement, agentic coordination, observation, and auxiliary functions, with LLM calls restricted to learning time except for fallback execution.The templates form four functional groups.
  • A.1 Learning prompts: The synthesis prompt generates an initial rule set once from the full training dataset, while multi-class tasks instead issue one per-class call using positives and counter-examples.Per-class synthesis prevents cross-class interference and is not used for binary or untyped extraction tasks.
  • A.1 Learning prompts: The patch prompt updates rules once per refinement iteration, selecting among six variants that vary failure snippets, data evidence, and non-relevant-rule formatting.RuleChef selects the longest variant that fits the context window.
  • A.2 Agentic coordinator prompts: Agentic coordination adds refinement guidance, holistic rule critique, redundancy auditing, and learning-trigger decisions for retraining on accumulated streaming or batch data.Auditor-proposed structural changes are re-evaluated on a held-out split, while critic feedback re-enters through the standard feedback interface.
  • A.3 Observation mode prompts: In observation mode, task discovery infers a schema from raw API logs, and observation mapping converts batches of up to ten logs into filtered structured input-output pairs.Mapped examples enter the training buffer for subsequent synthesis or patch calls.
  • A.4 Utility prompts: Synthetic example generation augments sparse training sets by producing one realistic input on demand, with a seed parameter driving diversity across calls.This prompt serves an auxiliary learning function.
  • A.4 Utility prompts: The LLM fallback prompt is the only inference-time prompt and runs only when no rule fires and the executor delegates instead of abstaining.It is issued as a last resort during execution.
Loading 2607.01293v1…