Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models

1. Introduction

• Logical inconsistencies can produce unreliable conclusions and paradoxes, making coherent predictions important for structured reasoning and decision-making.The introduction identifies temporal or spatial reasoning, optimization, and automated decision systems as relevant domains.Source: §1 · p. 1
• The proposed framework measures logical preference consistency through transitivity, commutativity, and negation invariance.The paper presents these as mathematical properties for quantifying consistency.Source: §1 · p. 1
• REPAIR refines noisy pairwise comparisons with rank aggregation and extrapolates additional comparisons logically, improving consistency without sacrificing alignment with human preferences.The method is proposed for instruction-tuning data.Source: §1 · p. 1
• The authors report that logical consistency correlates strongly with judgment robustness and that more consistent models outperform less consistent ones in logic-dependent tasks.They also report improved efficiency in sorting-based ranking algorithms reliant on logical coherence (Liu et al., 2024).Source: §1 · p. 1

2.1. Measuring Transitivity

• stran(K) tests whether sampled K-item relation graphs are acyclic, with 1 indicating nearly perfect transitivity.The metric is applicable to partial graphs, and estimates for larger item sets use at most 1,000 sampled subgraphs; scores typically decrease as K grows, enabling comparisons at a fixed subset size.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2. Measuring Commutativity

• scomm measures whether pairwise judgments remain unchanged when the order of the compared items is reversed.It averages across item pairs and ranges from 0 to 1, with 1 indicating perfect commutativity.Source: §2.2 · p. 4

2.3. Measuring Negation Invariance

• sneg checks whether judgments match the expected reversal when the model is prompted with a negated relation.The score averages across pairwise permutations and ranges from 0 to 1, with 1 indicating perfect negation invariance.Source: §2.3 · p. 4

3.1. Evaluation Setup

• The evaluation covers summary coherence, document relevance, and temporal or causal event ordering, and reports consistency averages alongside human agreement and sampled-output self-agreement.Self-agreement is measured from multiple chain-of-thought outputs at temperature 0.7; higher values indicate greater judgment stability.Source: §3.1 · p. 4

3.2. Results and Analysis

• Gemma2-9B, Phi-3-medium, and Deepseek-chat-v2 perform well across all three consistency dimensions, but strength in one dimension does not guarantee strength in others.The authors hypothesize that Phi-3’s high consistency may reflect cleaner, less self-contradictory synthesized training data; they also report more consistent performance on CaTeRS, possibly because its relations are more objective.Source: §3.2 · pp. 4–5
• Chain-of-thought prompting did not generally improve logical consistency and sometimes reduced transitivity, which the authors hypothesize may reflect variations in judgment standards across comparisons.Source: §3.2 · p. 5

3.3. Consistency and Reliability

• Commutativity strongly correlates with human agreement across nearly all tested datasets and models, consistent with the link between noncommutativity and positional bias.The comparison prompts were paraphrased ten times to test sensitivity to input variation.Source: §3.3 · pp. 5–6, Figure 5 · p. 6
• Transitivity strongly correlates with self-agreement across all three datasets, supporting its use as a proxy for local judgment robustness.Self-agreement is the share of majority choices across 10 chain-of-thought samples at temperature 0.7.Source: §3.1 · p. 4, §3.3 · p. 5, Figure 4 · p. 6
• The commutativity–human-preference correlation varies by model, with Llama-3-8B showing a higher correlation than Gemma-2-9B.The authors hypothesize that the difference may relate to training designed for prompt robustness versus fine-tuning to specific prompting styles.Source: §3.3 · p. 6

4.1. Estimating Rankings from Noisy Pairwise Data

• REPAIR estimates a full or partial ranking from win-loss rates, then extrapolates it into self-consistent pairwise comparisons that can be augmented with negated relations.Each item’s rate is wins minus losses divided by its number of comparisons; win-loss ranking handles sparse, partial comparisons and is invariant to comparison order.Source: §4.1 · p. 6
• The illustrated refinement turns sparse, potentially non-transitive annotations into more numerous consistent comparisons, leaving items unpaired when their preference relation is unknown.Source: Figure 6 · p. 7

4.2. Experiments

• Experiments simulate noise by flipping 10% of Summarize From Feedback training labels, then compare perturbed data with refined and augmented variants on 200 sampled test instances.The variants include REPAIR-ed data with and without additional negated-relation comparisons.Source: §4.2 · p. 7
• Training on REPAIR-ed data significantly improves transitivity and commutativity while maintaining strong human alignment; adding negated relations improves negation invariance but may slightly reduce other properties.The authors attribute the possible reduction to distraction and forgetting effects.Source: Table 3 · p. 7, §4.2 · p. 7

• PairS uses an LLM as a pairwise evaluator in a merge-sort ranking method, with performance measured by Spearman correlation against human rankings.Source: §5 · p. 8
• PairS assumes near-perfect transitivity and commutativity for optimal ranking, motivating the examination of consistency in logically grounded LLM applications.Source: §5 · p. 8
• Phi-3-mini achieves better PairS ranking performance than GPT-3.5-turbo despite slightly lower human-judgment accuracy, and more commutative models gain less from calibration and require fewer computations.Source: §5 · p. 8

• The reviewed consistency literature has predominantly addressed factual knowledge and entailment across limited statements.Source: §6 · p. 8
• Factual-consistency research includes paraphrased-relation benchmarks, knowledge-graph studies, and the reverse-curse phenomenon, where learned knowledge does not imply its reverse.Source: §6 · p. 8
• Entailment-consistency work has examined transitivity and symmetry in natural-language inference, tested LLM consistency, and used adversarial first-order logic examples to improve prediction consistency.Source: §6 · p. 8
• Most prior work focuses on first-order relations between two or three statements, leaving consistency in broader multi-item decision-making comparatively underexplored.The paper frames this gap as relevant to deploying LLMs in practical applications.Source: §6 · p. 8

7. Conclusion

• Models trained on REPAIR-ed data achieve higher logical preference consistency without compromising human alignment.Source: §7 · p. 8
• Integrating logically consistent LLMs into logic-driven algorithms enhanced global performance and computational efficiency.Source: §7 · p. 8
• The framework quantifies transitivity, commutativity, and negation invariance as proxies for assessing overall reliability.Source: §7 · p. 8

Impact Statement

• The authors hope logical coherence will contribute to more reliable AI in healthcare, finance, and education, where decision-making and reasoning matter.Source: Impact Statement · p. 8
• The PairS results are summarized as showing that better transitivity corresponds to better rankings, while improved commutativity means less calibration is needed.The caption reports average Spearman correlations over 100 runs for both PairS methods.Source: Table 4 · p. 9
• Logical consistency does not automatically guarantee fairness, transparency, or accountability.Source: Impact Statement · p. 9

A. Transitivity Calculation

• The transitivity procedure builds a directed preference graph from pairwise LLM judgments, then checks that graph for cycles.Source: A. Transitivity Calculation · p. 14

B. Dataset

• The evaluation datasets cover summarization coherence, document reranking relevance, event ordering, summary comparisons, and MS MARCO reranking.SummEval has 100 source texts with 16 summary candidates; NovelEval has 21 questions; the CaTeRS subset contains 70 instances; MS MARCO uses 800 training and 200 test examples.Source: B. Dataset · pp. 14–15

C. Large Language Models in Evaluation

• The evaluation includes open-source Llama, Mistral, Zephyr, Phi, and Gemma models as well as GPT-3.5-turbo-0125 and Deepseek-chat-v2; instruction tuning used a learning rate of 5 × 10−5 for two epochs.Source: C. Large Language Models in Evaluation · p. 15, F. Instruction-Tuning: Training Details · p. 16

D. Additional Related Work

• The related work describes consistency-improvement approaches including augmentation, training regularization, adversarial training, and consistency regularization.Source: D. Additional Related Work · p. 15

E. Distinguishing Logical Transitivity from Social Network Analysis (SNA) Transitivity

• The paper’s logical transitivity measures acyclicity in directed relation graphs and circulations spanning 3 to K nodes, unlike SNA transitivity’s closed-triangle clustering.Source: E. Distinguishing Logical Transitivity… · p. 16

G. Metric Computation details and Prompt Templates

• Metric computation uses pairwise comparisons of items in task-specific contexts, including summaries, retrieved documents, and unordered events; standard and reversed or negated prompts produce preference matrices for the metrics.Figures 7 and 8 show prompt templates for dataset comparisons and instruction tuning.Source: G. Metric Computation details and… · pp. 16, 18, Figure 7 · p. 17

H. The Choice of K

• Stran(K) is robust across K, while larger K expands its value range and can make model comparisons more distinct; there is no gold-standard K, and choice depends on models and tasks.The reported model trend runs from earlier Llama-2 and Zephyr-7B-beta models to Gemma-2 and Llama-3; objective tasks tend to have higher consistency.Source: H. The Choice of K · pp. 18–19

I. Theoretical Evidence for the Choice of transitivity Sub-graph sampling size M

• The authors justify M = 1000 sampled sub-graphs by a worst-case estimate requiring at least 394 samples for 95% confidence, and assume comparisons are irreflexive by omitting diagonal matrix entries.The paper notes that comparing an item against itself is uncommon in the cited preference, causal-order, and temporal-order settings.Source: §I · p. 19, J. The Irreflexive Assumption · p. 19

K. Motivations for our Data Refinement and Augmentation Framework

• The data-refinement motivation is that preference annotations are often noisy and self-contradictory; the authors propose estimating a coherent ranking, then augmenting it with conflict-free pairwise comparisons.The authors expect this to be more efficient than aligning directly to noisy, inconsistent annotations.Source: K. Motivations for our Data Refinement… · p. 19

L. Performance of REPAIR on the MS-MARCO Dataset

• REPAIR enhances consistency while maintaining alignment with human preferences on MS MARCO.Source: L. Performance of REPAIR on the… · p. 19
• Higher transitivity is associated with more accurate overall rankings predicted by PairS, while stronger commutativity is associated with less calibration needed for optimal performance.Table 5 reports average Spearman correlations over 100 runs for both PairS methods and states that REPAIR-ed training improves PairS performance with reduced calibration need.Source: L. Performance of REPAIR on the… · p. 20, Table 5 · p. 20

M. Ablation Study: REPAIR Down-Sampling Performance

• The ablation randomly samples REPAIR-ed data to match the perturbed dataset’s size, holding other experimental settings constant.This controls for dataset-size differences, though random sampling may lose information.Source: M. Ablation Study: REPAIR… · p. 20
• Random sampling from REPAIR-ed data can discard some of the information added by augmentation, so the comparison may not show its optimal benefit.The authors note that targeted sampling could mitigate this information loss.Source: M. Ablation Study: REPAIR… · p. 20
• All three consistency metrics improve over perturbed data despite down-sampling, while alignment with human preferences declines slightly.The authors attribute the decline to information loss from random sampling and describe it as relatively minor.Source: M. Ablation Study: REPAIR… · p. 20

N. Ablation Study: Alternative Ranking Estimation Methods for REPAIR

• Elo and Bradley–Terry achieve slightly better human accuracy and consistency metrics than win-loss rate, with relatively small performance differences.The comparison uses REPAIR-ed Summarize from Feedback data; the table caption reports improvements across the methods.Source: Table 7 · p. 21, N. Ablation Study: Alternative Ranking… · p. 21
• Win-loss rate treats comparisons independently, preserves ties when evidence does not rank items against each other, but misses indirect comparisons in transitive chains.It is a straightforward baseline suited to settings where ties are acceptable or efficiency is a priority.Source: N. Ablation Study: Alternative Ranking… · pp. 21–22
• Elo incorporates transitive information but is sensitive to comparison order, which can add noise because this dataset is static and order carries no inherent meaning.With sparse comparisons, reversing the order can change the inferred ranking.Source: N. Ablation Study: Alternative Ranking… · p. 21
• Bradley–Terry estimates rankings by maximum likelihood and handles transitive relationships, but may force different scores for items that should tie.That forced ranking can introduce noise when ties should be preserved.Source: N. Ablation Study: Alternative Ranking… · p. 21
• Estimator effectiveness depends on comparison sparsity and computational constraints; Elo and Bradley–Terry may struggle with sparse data, while win-loss rate is simpler and more efficient.Elo and Bradley–Terry capture transitive relationships but can be limited by noise and computational complexity.Source: N. Ablation Study: Alternative Ranking… · pp. 21–22