PaperlayerView exact revision on arXiv

Source-linked AI summary

From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options

Obed Junias, Maria Leonor Pacheco

arXiv:2608.12836v1cs.CLcs.AI

TL;DR

LLMs can judge atomic answers correctly yet fail to compose them under AND, OR, and NEITHER/NOR operators. The paper separates atomic evidence from logical composition using contrastive scoring and operator-constrained ILP inference. Across LOGICAL-COMMONSENSEQA and LOGICAL-SATA, structured inference substantially outperforms direct prompting, with the largest improvements on NEITHER/NOR.

  • Problem

    LLMs often fail to combine otherwise correct atomic judgments when answer options use explicit logical operators.

  • Method

    The framework decomposes compound options into atomic answers, elicits contrastive evidence, calibrates scores, and composes them with operator-constrained integer linear programming.

  • Results

    Structured inference substantially outperforms direct prompting across LOGICAL-COMMONSENSEQA and LOGICAL-SATA, with the largest improvements on NEITHER/NOR.

  • Takeaways & Limitations

    Separating atomic evaluation from logical composition reduces the compositionality gap observed under direct compound-answer prediction.

  • Takeaways & Limitations

    The evaluation uses one model and two benchmarks with binary operators over pairs of atomic answers, so generalization to other models, longer expressions, and other operators is unestablished.

Abstract

from arXiv · show

Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.

1 Introduction

The paper argues that logical composition is a distinct challenge for LLMs: models may assess atomic propositions correctly yet fail when combining them under explicit operators. It proposes separating atomic evidence from constrained logical inference and evaluates this approach on two benchmarks.

  • LLMs show operator-dependent reasoning failures, with performance strongest on conjunction, weaker on disjunction, and collapsing on negated compositions.
  • Standard prompting fuses atomic assessment and logical composition in one step, creating a compositionality gap that is difficult to diagnose or constrain.
  • The framework decomposes each compound option into atomic answers, elicits paired positive and negative hypotheses, and scores each unique atom once.
  • An operator-constrained integer linear program calibrates atomic scores, jointly infers atomic assignments, and selects exactly one compound option.
  • LOGICAL-SATA extends operator-based compound-answer reasoning to paragraph-based reading comprehension using data constructed from SATA-Bench.
  • The evaluation covers LOGICAL-COMMONSENSEQA and LOGICAL-SATA, with the largest reported gains on NEITHER/NOR.

2 Related Work

Prior work studies logical reasoning, decomposition, multi-answer question answering, confidence elicitation, and neuro-symbolic inference. This paper connects these strands by applying structured inference to compound answer options with explicit operators.

  • Logical-reasoning benchmarks test deriving conclusions from facts, rules, premises, or constraints, while specialized datasets isolate conjunction, disjunction, and negation.
  • Decomposition methods make intermediate reasoning explicit or divide complex problems into simpler subproblems, including tree-structured entailment explanations.
  • Many multiple-choice benchmarks score each candidate as a whole and therefore do not test combining several atomic answers under an explicit operator.
  • Multi-answer benchmarks allow several correct responses, with SATA-Bench requiring models to identify the complete set of independently correct choices.
  • LOGICAL-SATA transfers operator-based compound options to paragraph-based reading comprehension by combining independently annotated SATA-Bench answers.
  • Confidence and contrastive-judgment research motivates collecting evidence for competing or opposing interpretations rather than relying only on isolated scores.
  • Neuro-symbolic methods translate language problems into formal representations processed by deterministic solvers, while earlier systems combine uncertain predictions with symbolic constraints.

3 Framework Overview

The framework separates atomic judgment from logical composition: an LLM supplies contrastive evidence for each unique atom, while calibrated scores are composed by an operator-constrained ILP.

  • Task formulation: Each compound option is parsed into two atomic answers joined by AND, OR, or NEITHER/NOR, with validity determined jointly by atom statuses and operator semantics.The task contains four compound options, and exactly one is valid by construction.
  • Atomic decomposition: Shared atomic answers are represented once, so one inferred status applies consistently wherever an atom appears across options.This prevents repeated occurrences of the same proposition from receiving different judgments.
  • Contrastive evidence: For each unique atom, the model compares paired positive and negative hypotheses rather than judging a compound option directly.The hypotheses state that the atom satisfies or does not satisfy the context, and the model produces evidence without selecting the correct hypothesis at this stage.
  • Score calibration: Raw scores express relative preference between opposing hypotheses, so calibration makes atomic evidence comparable before global composition.The paper evaluates Platt and isotonic calibration, while relative calibration also uses within-instance standing and the maximum positive score.
  • Globally constrained inference: An ILP assigns binary statuses to atoms and validity variables to options, enforcing the exact AND, OR, and NEITHER/NOR composition rules.The objective selects the feasible assignment with the strongest total evidence, and the unique option with x_i = 1 becomes the prediction.

4 Benchmark Datasets

The paper evaluates compound-answer reasoning on two benchmarks with different atomic evidence sources: commonsense plausibility and reading comprehension. LOGICAL-SATA is newly constructed from SATA-Bench, while LOGICAL-COMMONSENSEQA combines commonsense questions with logical operators.

  • The benchmarks contain compound answer options whose two atomic answers are joined by AND, OR, or NEITHER/NOR.
  • LOGICAL-COMMONSENSEQA: LOGICAL-COMMONSENSEQA contains 19,996 instances across AND, OR, NEITHER/NOR, and MIXED operator settings.Its test set has equal human-validated and non-validated subsets of 1,000 instances each.
  • LOGICAL-SATA: LOGICAL-SATA pairs annotated correct and incorrect reading-comprehension answers into compound options after filtering source questions for valid distractor construction.The construction begins from the human-labeled training partition of SATA-Bench.
  • LOGICAL-SATA: LOGICAL-SATA constructs operator-specific and MIXED instances by sampling one valid option and three distractors from operator-compatible pools.
  • LOGICAL-SATA: The resulting LOGICAL-SATA dataset contains 5,400 instances divided into 2,400 training, 1,000 development, and 2,000 test examples.

5 Experiments

Experiments compare structured inference with direct prompting across two benchmarks and examine operator-specific performance, calibration, and error propagation. Structured inference substantially improves Macro-F1, especially for NEITHER/NOR, while relative calibration provides additional gains in mixed-operator settings.

  • Experimental setup: Experiments use Llama-3.1-8B-Instruct at temperature 0.7, averaging results over five runs and reporting Macro-F1, Brier score, and log loss.
  • Main results: 27.5 points: on LOGICAL-COMMONSENSEQA-HV, Macro-F1 rises from 48.3 with strongest direct prompting to 75.8 with structured inference and 77.0 with relative calibration.
  • Main results: On LOGICAL-SATA, Macro-F1 increases from 47.0 with strongest direct prompting to 72.2 with paired multiple-choice structured inference.
  • Performance across logical operators: The largest operator-specific gains occur on NEITHER/NOR, rising from 14.0 to 76.8 on LOGICAL-COMMONSENSEQA and from 12.6 to 73.4 on LOGICAL-SATA.
  • Calibration: Relative calibration reduces atomic Brier score and log loss on both benchmarks, with the strongest downstream gains in MIXED settings.Macro-F1 increases by 4.9 on LOGICAL-COMMONSENSEQA and 11.2 points on LOGICAL-SATA in MIXED settings.
  • Error analysis: Gold atomic statuses yield 1.00 inference accuracy on both benchmarks, while observed compound accuracy is 0.758 on LOGICAL-COMMONSENSEQA-HV and 0.723 on LOGICAL-SATA.This isolates surviving atomic errors as the informative source of compound mistakes.

6 Conclusions and Future Work

The paper separates atomic evaluation from logical composition, using contrastive evidence and globally constrained inference for compound answer options. Across two benchmarks, this approach outperforms direct prompting, with the largest improvements on NEITHER/NOR and additional benefits from relative calibration in MIXED settings.

  • The framework decomposes compound options into atomic answers, elicits evidence for opposing hypotheses, and combines scores through globally constrained inference.
  • Relative calibration compares each atomic score with other scores in the same instance and performs best overall, with its largest gains in MIXED settings.
  • Future work could extend the framework to longer options, nested expressions, additional operators, less structured text, probabilistic inference, and more model families and benchmarks.

Limitations

The evaluation is limited to one model and two benchmarks with binary operators over pairs of atomic answers, and its results depend on the quality of atomic evidence.

  • The evaluation does not establish whether the gains generalize to other models, longer logical expressions, or operators such as implication and exclusive disjunction.
  • The framework depends on atomic evidence quality, so commonsense interpretation, passage-grounding, and source-annotation errors can propagate to final predictions.
  • The benchmarks enforce exactly one valid compound option, whereas real tasks may permit multiple valid answers or no valid answer.

Ethical Considerations

The work uses publicly available data without human-subject data collection, but acknowledges dataset and model-output risks and limits its claims for high-stakes use.

  • The study uses publicly available data and does not involve human-subject data collection.
  • Benchmarks and model outputs may reflect biases, ambiguities, or annotation errors inherited from their source datasets.
  • The framework improves consistency under explicit logical constraints but does not guarantee correct underlying atomic judgments.
  • The method should not be interpreted as providing reliable logical guarantees for medical, legal, or financial decision-making.

A Implementation Details

Experiments use Llama-3.1-8B-Instruct with fixed sampling, calibration, inference, hardware, and evaluation settings.

  • All experiments use Llama-3.1-8B-Instruct, with direct baselines spanning zero- through three-shot prompting and zero-shot chain-of-thought.
  • Atomic confidence scores use temperature 0.7, five generations per atomic answer, five-run averaging, and random seed 42.
  • Calibration uses 2,400 training instances for LOGICAL-SATA and an equally sized, logically balanced sample for LOGICAL-COMMONSENSEQA.
  • Global inference uses Gurobi Optimizer 13.0.2 on an NVIDIA A100 GPU.

B.1 Full Accuracy Results

The appendix reports full accuracy and Macro-F1 results, confidence-elicitation comparisons, and error analyses across LOGICAL-COMMONSENSEQA and LOGICAL-SATA.

  • Full accuracy results: Tables 5 and 6 report accuracy on LOGICAL-COMMONSENSEQA’s NV/HV subsets and the LOGICAL-SATA test set.
  • Confidence elicitation: Paired multiple-choice confidence achieves 75.8 Macro-F1 on LOGICAL-COMMONSENSEQA-HV and 72.2 on LOGICAL-SATA, outperforming independent true–false confidence.
  • Confidence elicitation: Generation sampling performs below paired multiple-choice confidence overall and has greater variability, while verbalized confidence performs substantially worse.
  • Error analysis: Separating semantic atomic errors from logical-composition errors distinguishes whether failures arise during understanding or propagation.
  • Error analysis: Inspected errors include incorrect atomic confidence, missed supported labels, and insufficient attention to question modifiers or relations.

C.5.2 Logical Propagation of Atomic Errors

Logical operators determine how atomic scoring errors propagate through globally constrained inference, with different operators tolerating or amplifying the same mistake.

  • Operator effects: The same atomic error can have different consequences under OR, AND, and MIXED constructions.
  • Operator effects: OR can tolerate some false-positive atomics because the gold option remains supported by its disjuncts.
  • Operator effects: AND errors arise when required conjuncts are rejected or competing conjunctions gain support from false positives.
  • Operator effects: NEITHER/NOR errors arise when at least one member of the gold pair is incorrectly accepted.
  • Operator effects: MIXED instances are more complex because one atomic judgment can influence options governed by different operators.
  • Global inference: Exact-one global inference may adjust atomic assignments when local evidence makes no option or multiple options logically valid.

D Benchmark Structure and Examples

The paper illustrates benchmark instances across four logical settings and shows how atomic-answer judgments are elicited through paired hypotheses and confidence estimates. LOGICAL-SATA examples span multiple source domains, while LOGICAL-COMMONSENSEQA examples omit the passage field.

  • Atomic evaluation: Each atomic answer is evaluated through positive and negative hypotheses stating whether it satisfies the relevant constraint.The paired prompt presents both interpretations, and the model selects the one correct with respect to the context and constraint.
  • Confidence estimation: Independent true–false confidence evaluates positive and negative hypotheses in separate model calls.Verbalized confidence additionally asks the model to report a numerical confidence value.
  • Prompt structure: The atomic-evaluation prompt includes the atomic statement, two competing hypotheses, constraint-focused instructions, and a single-letter output requirement.The model must judge the atomic answer independently of other answer options and return only A or B.
  • Confidence estimation: Generation sampling uses benchmark-specific templates, while sampled A/B frequencies provide positive and negative evidence scores instead of answer-token probabilities.The procedure samples five responses and uses the same templates as paired multiple-choice confidence.
  • Benchmark examples: Figures 5 and 6 show representative instances covering AND, OR, NEITHER/NOR, and MIXED settings, with atomic labels and a unique highlighted gold option.LOGICAL-SATA examples come from four source domains, and excerpts are shortened for readability.
Loading 2608.12836v1…