PaperlayerView exact revision on arXiv

Source-linked AI summary

A Task-Centric Ontology and Deterministic Domain Rules as a Verifiable Core for AI-Assisted Chemistry Problem Solving

Ibrokhimsho Abduchaborov

arXiv:2608.26164v1cs.AI

TL;DR

LLMs are useful for interpreting chemistry questions but difficult to inspect and validate as sole domain reasoners. This paper builds a task-centric ontology and deterministic symbolic core for bounded school-level chemistry tasks, with fallbacks for uncovered families. The complete system matched 296 of 300 references, while the ontology-rule subset matched 266 of 269; these results establish internal coverage and consistency, not independent generalization.

  • Problem

    LLMs can interpret chemistry questions but their internal reasoning may be inconsistent, overconfident, and difficult to inspect, motivating a verifiable alternative for domain reasoning.

  • Method

    The paper constructs a task-centric ontology and ontology-linked deterministic procedures, separating reusable rules from task-specific expert-coded fallbacks for a bounded chemistry problem collection.

  • Results

    The complete system matched 296 of 300 references, while the ontology-rule subset matched 266 of 269 problems.

  • Takeaways & Limitations

    A bounded task-focused ontology with deterministic procedures can route most collection problems through reusable rules while making remaining errors local and inspectable.

  • Takeaways & Limitations

    Because the problem collection informed both ontology construction and evaluation, the reported rates measure internal coverage and consistency rather than independent generalization.

Abstract

from arXiv · show

Large language models can interpret natural-language chemistry questions, but their internal reasoning is difficult to inspect, constrain, and validate. This paper presents ChemOntoRule, a proof-of-concept symbolic core for AI-assisted school-level chemistry problem solving. The central design choice is task-centric ontology engineering: the ontology is constructed around the concepts, properties, relations, and executable procedures required by a defined collection of chemistry problems, rather than as a universal representation of chemistry. The implemented artifact combines a lightweight ontology serialized in JSON and RDF/Turtle with deterministic Python rules for electronic structure, periodic trends, oxidation states, oxide and hydride behavior, and related school-level reasoning patterns. A separate expert-coded fallback handles problem families not yet represented by general rules. The system was examined on 300 human-authored and manually validated chemistry problems. The complete system matched 296 of 300 reference answers (98.67%). The ontology-driven rule subset covered 269 problems and matched 266 references (98.88%); 31 problems were handled by task-specific expert-coded fallbacks, with 30 matches. Because the same collection informed ontology construction and evaluation, these results measure implemented coverage and internal consistency, not independent generalization. We analyze the four mismatches, distinguish structural validation from chemical correctness, and define a future architecture in which a language model acts primarily as a translator from user language into a normalized ontological task frame. Token efficiency is presented as a testable hypothesis for future controlled studies, not as a result of the current work.

1 Introduction

The paper proposes a task-centric ontology and deterministic symbolic core for inspectable school-level chemistry reasoning, reserving LLMs for translation and verbalization. Its proof-of-concept contribution is scoped to selected problem families and evaluated through coverage, reference matching, and error analysis.

  • Motivation: LLMs provide flexible chemistry language interfaces but can be inconsistent, overconfident, and difficult to inspect as sole domain reasoners.The paper motivates explicit symbolic reasoning where intermediate steps, rule applicability, and final answers must be inspectable.
  • Architecture: The proposed division of labor translates free-form user requests into constrained task representations while ontology-linked procedures perform chemistry reasoning.The intended architecture later uses the language model to verbalize results, but the current study evaluates only the symbolic core.
  • Design: Task-centric ontology engineering derives ontology content from competency requirements in a bounded problem collection rather than attempting a complete chemistry ontology.The approach identifies required entities, properties, relations, and executable rules for the target task space.
  • Implementation: The symbolic artifact combines ontology-backed deterministic rules for selected school-level chemistry families with separate task-specific expert-coded fallbacks.This separation makes reusable general procedures distinct from solutions used when the general rule base lacks coverage.
  • Evaluation: The proof-of-concept evaluates 300 human-authored and manually validated problems using coverage, reference matching, and mismatch analysis rather than primarily benchmarking an LLM.The study differs from LLM benchmark work because it examines a deterministic core intended to receive structured task representations.

3 Research Scope and Questions

The study defines a bounded evaluation around routing chemistry problems to general rules or fallbacks, matching validated references, explaining mismatches, and specifying a future translator interface. Its scope excludes claims about generalization, controlled LLM superiority, inference cost, or token savings.

  • Research questions: The evaluation asks what proportion of 300 problems can use general ontology-driven rules rather than task-specific fallbacks.This routing question is paired with separate assessment of complete-system, rule-subset, and fallback-subset reference matching.
  • Scope boundaries: The collection’s answer-type imbalance limits conclusions about open-ended derivations and numeric problem solving.Table 1 is described as dominated by selection-sequence questions.
  • Research questions: The study examines which modeling or implementation assumptions explain observed mismatches.This frames errors as an object of analysis rather than treating aggregate matching as sufficient evidence.
  • Research questions: The study also asks what interface a future LLM translator should expose so natural-language requests can be resolved by the deterministic symbolic core.The interface question concerns translation into a representation usable by the symbolic system.
  • Scope boundaries: The paper does not claim out-of-distribution generalization, superiority over a controlled LLM baseline, reduced inference cost, or measured token savings.These exclusions define the evidentiary boundaries of the research scope.

4 Problem Collection and Validation

The study uses a bounded, competency-driven problem collection to define a lightweight ontology and evaluate its implementation. Because the same 300 problems informed both design and evaluation, results indicate implemented coverage and internal consistency rather than independent generalization.

  • 300 human-authored school-level chemistry problems were manually reviewed and validated, with reference answers and solutions.
  • The collection both supplied requirements for the task-centric ontology and served as the proof-of-concept evaluation set.
  • The collection is dominated by selection-sequence questions, limiting conclusions about open-ended derivations and numeric problem solving.
  • Ontology development identifies entities, target relations, required properties, rule conditions, output formats, and structural checks for each recurring problem family.
  • Competency questions include outer-level electron counts, oxidation states from oxide formulas, periodic-trend ordering, and stable-shell ion transformations.
  • The implementation uses a lightweight ontology backed by external Python rules rather than a pure OWL reasoner.

6 Deterministic Domain-Rule Engine

The deterministic rule engine applies explicit school-level procedures to electronic structure, periodic trends, oxidation states, oxide behavior, hydrides, and recognized task templates. It records whether each result came from a reusable ontology rule or a task-specific expert fallback.

  • The engine covers electron configurations, subshell counts, outer-level and valence electrons, unpaired electrons, and stable-ion electron counts.
  • It evaluates oxidation-state ranges, common oxide formula feasibility, and heuristic metallic, nonmetallic, acidic, basic, and hydride trends.
  • Routing and ordering rules assign recognized task templates to the relevant deterministic procedures.
  • Each solution record labels its method as ontology_rule for reusable ontology-linked procedures or manual_expert for uncovered task families.
  • Separating fallback answers from ontology inference creates an expansion path in which fallback families become general only after their entities, conditions, transformations, and validation rules are independently represented.

7 Processing and Validation

The processing pipeline instruments solutions as logical stages, validates output structure, and separately measures agreement with human-validated reference answers. All 300 outputs passed structural checks, while chemical performance was assessed through normalized reference matching.

  • The instrumentation produces 1,800 stage records and 1,800 event logs across 300 tasks.
  • Dashboard stage durations are generated for visualization and are not wall-clock latency measurements.
  • Structural validation checks answer presence, non-empty selections, plausible selection length, and option-index range.
  • Reference matching tests whether normalized output equals the human-validated answer, distinguishing chemical performance from structural conformity.
  • All 300 outputs passed implemented structural checks, and chemical performance was reported using reference matching.
  • Table 2 presents proof-of-concept reference matching with Wilson 95% confidence intervals for observed proportions.

8 Experimental Protocol

The evaluation ran the solver on 300 manually validated problems, recording routing and solution metadata and comparing normalized answers with references. Results distinguish reusable ontology rules from expert-coded fallbacks and report a separate uncontrolled LLM comparison.

  • The available LLM answer file matched all 300 references, but missing model and protocol details prevented controlled-baseline or superiority claims.
  • 269 of 300 problems received ontology-driven rule solutions, while 31 used task-specific expert-coded fallbacks.
  • 296 of 300 reference answers matched for the complete system (98.67%).
  • All oxidation-state, expert-reaction, organic, and extended-equation task families matched their references; three mismatches occurred in general rule families and one in an expert-coded calculation.

10 Mismatch Analysis

The four mismatches expose distinct weaknesses in semantic scope, applicability conditions, domain heuristics, and task-specific arithmetic. The analysis uses these errors to identify abstractions for the reusable rule layer.

  • Four outputs failed to match references, and the mismatch analysis targets abstractions for reusable rule-layer repair.
  • The four weakness classes were missing semantic scope, incomplete applicability conditions, oversimplified transition-metal heuristics, and task-specific arithmetic.
  • Semantic scope can make a numerically available property wrong for a question when its intended scope is not modeled.
  • Relations such as belongsToPrincipalLevel and same-outer-level constraints would reduce ambiguity in outer-level reasoning.

11 AI Translation Layer

The proposed architecture assigns language models the role of translating natural-language requests into normalized task frames, while a symbolic core performs inspectable chemistry reasoning. This architecture remains prospective because the current experiment did not evaluate the translator or token efficiency.

  • The proposed translator emits a normalized task frame containing a task family, entities, requested relation, constraints, and output schema.
  • The symbolic core selects an applicable rule, executes it, records a trace, and validates the output before optional language verbalization.
  • The design keeps domain reasoning inspectable and amendable by experts while accepting varied natural-language wording.
  • The experiment did not implement or evaluate the proposed LLM translator.
  • The bounded collection showed 269-rule coverage, while four mismatches remained local and inspectable rather than being eliminated.
  • Token efficiency is a hypothesis for controlled future experiments, not a present result or claim.

13 Threats to Validity and Limitations

The evaluation is constrained by overlap between ontology development and testing, narrow task coverage, fallback reliance, incomplete formal validation, and limited reproducibility evidence. Future work prioritizes independent testing, reusable modules, measured AI interfaces, robustness evaluation, and open benchmarking.

  • The problem collection served both ontology development and evaluation, so reported rates measure internal coverage and consistency rather than independent generalization.
  • 291 of 300 cases are selection-sequence questions, limiting evidence for open-ended equations, calculations, and explanatory reasoning.
  • 31 tasks use expert-coded fallbacks, whose performance must remain separate from general ontology-driven inference.
  • Most rules remain procedural Python logic outside the RDF/OWL graph, without full use of OWL and SHACL validation capabilities.
  • Normalized exact matching does not verify every intermediate chemical step, and mismatches may reflect rounding conventions or ambiguity.
  • Future studies should use independent paraphrase, entity, and compositional tests while measuring latency, memory, tokens, and cost under controlled AI baselines.The planned work also includes an LLM translator, reusable chemistry modules, SHACL constraints, robustness tests, and an openly licensed benchmark.

15 Conclusion

The paper presents a task-centric chemistry ontology and deterministic rule engine as a verifiable core, with language models translating requests into explicit task frames. The bounded evaluation shows strong implemented coverage and internal consistency, while independent generalization and measured LLM-interface benefits remain future work.

  • The prototype routes 269 of 300 school-level problems to reusable ontology-linked rules and uses expert-coded fallbacks for 31.
  • 296 of 300 reference answers are matched by the complete system, while the ontology-rule subset matches 266 of 269.
  • The reported results establish coverage and internal consistency within a bounded task space, not independent generalization.
  • A language model can translate user requests into explicit task frames before a symbolic core applies inspectable rules and validators.
  • Independent evaluation, formalized constraints, a measured LLM interface, and controlled token-efficiency experiments remain future work.

A Implementation Summary

The implementation combines lightweight ontology artifacts, deterministic Python solving, task records, traces, logs, and dashboard outputs. Its contextual LLM comparison is excluded from central claims because the available file lacks metadata for controlled benchmarking.

  • The implementation includes JSON and RDF/Turtle ontology artifacts, a deterministic Python solver, solved-task records, process metrics, stage traces, event logs, and a static dashboard.
  • The ontology covers atomic structure, periodic properties, oxidation states, oxides and hydrides, and manual rules for mixed assignments.
  • The stored LLM solution file matches normalized references for all 300 tasks, but it is treated only as contextual material rather than evidence of superiority.
  • The LLM comparison lacks model, provider, prompt, decoding, attempt, token, and post-processing metadata required for controlled evaluation.
  • Future evaluation should freeze the ontology and rules before testing paraphrase, entity, and compositional transfer.
  • Human adjudicators should classify disagreements across parsing, ontology mapping, rule applicability, execution, validation, and explanation.
Loading 2608.26164v1…