PaperlayerView exact revision on arXiv

Source-linked AI summary

Integrating the Analytic Hierarchy Process with Large Language Models for Transparent Multi-Criteria Decision-Making

Han Zhiguang, Farah Benamara, Pascale Zaraté

arXiv:2609.16779v1cs.AI

TL;DR

Opaque LLM reasoning makes multi-criteria decisions difficult to validate, especially in high-stakes settings. The paper combines LLMs with AHP, introduces an expert-annotated legal benchmark, and operationalizes the complete AHP workflow end to end. Across experiments, structured AHP reasoning improves interpretability and alignment with expert judgments, while effectiveness varies with model capability and reasoning coordination.

  • Problem

    LLMs lack structured, verifiable reasoning for multi-criteria decisions, while existing LLM-AHP studies have inconsistent results and limited evaluation resources.

  • Method

    The paper builds an expert-annotated LEGAL-AHP benchmark and an end-to-end LLM framework that decomposes AHP into criterion generation, weighting, and aggregation.

  • Results

    AHP-based prompting improves interpretability and alignment with expert judgments, with stronger models benefiting more consistently and the pipeline transferring to non-legal ranking tasks.

  • Takeaways & Limitations

    Reliable automation depends on both structured decomposition and the underlying model’s reasoning and instruction-following capacity.

  • Takeaways & Limitations

    Effectiveness depends on model reasoning capability and multi-agent coordination, while expert-defined criteria and pairwise comparisons introduce subjectivity.

Abstract

from arXiv · show

LLMs are increasingly employed in a wide range of decision-making tasks. However, the opacity of their internal reasoning makes it difficult to validate or interpret their outputs, and the need for interpretability becomes especially critical in high-stakes settings. This study examines the decision-making capabilities of LLMs through the Analytic Hierarchy Process (AHP), a classical and widely used multicriteria decision-making framework. We construct a new annotated benchmark based on AHP and propose the first end-to-end approach that enables LLMs to perform the complete AHP workflow. Experiments in real-world decision problems in the legal and higher-education ranking domains show that our method significantly improves alignment with expert judgments.

1 Introduction

AHP offers an explicit, auditable workflow for multi-criteria decisions, motivating its integration with LLMs to address opaque reasoning and limited automation. The paper proposes an end-to-end LLM-based AHP approach, evaluates its robustness and portability, and reports improved alignment with expert judgment.

  • AHP decomposes goals into criteria and alternatives, derives priority weights from pairwise comparisons, checks consistency, and synthesizes a final decision.The consistency ratio criterion is ratio < 0.1.
  • Existing AHP-enhancement methods improve scalability or uncertainty handling but often rely on predefined knowledge bases or manual inputs, limiting flexibility.
  • Prior LLM-AHP studies use simple prompting, lack standardized datasets and systematic consistency verification, and report inconsistent findings.
  • The paper introduces an expert-annotated legal benchmark and the first end-to-end LLM approach operationalizing the complete AHP workflow.
  • The proposed strategies include explicit AHP instructions, a single-agent framework, and a multi-agent architecture designed for interpretable decision-making.
  • The evaluation examines internal pipeline stages, final decisions, methodological robustness, transparency, and portability beyond legal decision problems.

2 Related Work

Related work positions LLMs as increasingly common decision-support and autonomous decision-makers while identifying a gap in structured, verifiable multi-criteria reasoning. This paper addresses that gap by applying LLMs to the full AHP pipeline and introducing a legal benchmark with expert annotations.

  • LLMs support decision-making across domains, but their application to multi-criteria decision-making remains limited by opaque reasoning and insufficient structured logic.
  • Feature-level and post-hoc XAI explanations generally do not capture how models weigh multiple criteria to reach final outcomes.
  • Prior LLM-AHP studies report mixed performance, consistency violations, or only partial workflow integration.
  • The paper contributes a first benchmark manually annotated according to AHP for legal reasoning and studies LLMs mimicking the full AHP pipeline.

3 The LEGAL-AHP Dataset

LEGAL-AHP reframes selected LEGALBENCH questions as multi-criteria decisions and augments them with expert-defined criteria, weights, and final judgments. Consistency filtering and agreement analyses produce a benchmark with strong overall alignment to LEGALBENCH gold annotations, though one multiple-choice subset shows lower agreement.

  • 3.1 Dataset Selection: The dataset samples five LEGALBENCH subsets totaling 525 QA pairs across diverse legal areas and binary or multiple-choice decision formats.The subsets are Abercrombie, Judicial_Ethics, Common_Law, Decision_Section, and Privacy_Policy.
  • Annotators defined 3 to 7 criteria per QA pair, constructed pairwise comparison matrices, and computed criterion weights under AHP guidelines.
  • The annotation process trained groups on 450 QA pairs and collected separate consensus annotations for 75 held-out pairs.
  • Consistency filtering removed 12 of 20 group annotations for invalid or inconsistent criteria, retaining annotations meeting CR < 0.1 or having more than two criteria.
  • Gold criteria were created by semantically grouping similar criteria across annotator groups and selecting encompassing labels when definitions differed.For Judicial_Ethics, conflict of interest encompassed transparency and professional neutrality.
  • Average agreement between LEGALBENCH gold and LEGAL-AHP reached 80%, while the Decision section had lower agreement, likely because its multiple-choice format impeded consensus.

4 Methodology

The paper operationalizes the complete AHP workflow with LLMs through instruction-based, single-agent, and multi-agent strategies that produce weighted, interpretable decisions. It evaluates these strategies across multiple language models against a Non-AHP baseline using identical decision inputs and few-shot examples.

  • 4.1 AHP-based Strategies: The decision objective scores each alternative by summing criterion weights multiplied by criterion-specific scores, then selects the highest-scoring alternative.The formulation uses s(ai, cj) for an alternative’s score on criterion cj and supports a natural-language explanation of criterion contributions.
  • 4.1 AHP-based Strategies: The framework evaluates three AHP-based strategies: full AHP instructions, a structured single-agent pipeline, and a multi-agent architecture.The strategies cover criteria generation, pairwise weighting, and decision aggregation, with the multi-agent design decomposing the workflow into collaborative components.
  • 4.1 AHP-based Strategies: The single-agent pipeline sequentially generates criteria, derives their relative weights through pairwise comparisons, and ranks alternatives using the resulting weighted scores.Each stage is conditioned on previous outputs, preserving a dialogue history that supports internal consistency and interpretability.
  • 4.1 AHP-based Strategies: The multi-agent framework combines an Information Block that summarizes alternative–decision relations with an AHP Block that derives weights, ranks alternatives, and explains criterion contributions.The AHP Block uses specialized agents for criteria generation, pairwise weighting, and decision making; the Information Block performs in-situ inference from the provided inputs.
  • 4.1 AHP-based Strategies: The study evaluates four LLMs against a Non-AHP LegalBench baseline under identical inputs, few-shot examples, prompts, and generation parameters.The models include Llama-3-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.2, and GPT-4o-mini; accuracy is averaged over three runs.
  • 4.1 AHP-based Strategies: The automatically generated multi-agent setting is examined with Qwen2.5-7B-Instruct and GPT-4o-mini, while manual and automatic agents share the same objective, weighting scheme, and aggregation formulation.A centrally coordinated meta-agent dynamically decomposes tasks and instantiates specialized agents in the automatic setting.

5 Results

Experiments compare non-AHP and AHP-based strategies on legal decisions, human-annotated criteria and decisions, and university-ranking tasks. Results vary by model, with stronger models benefiting most from structured AHP reasoning and portability extending beyond law.

  • RQ1: LEGALBENCH Decisions: 81.7% accuracy is achieved by GPT-4o-mini with the automatically generated Multi-Agent setting, exceeding its AHP-Instruction result of 78.3%.Non-AHP accuracies range from 68.3% to 73.3%, while Mistral and LLAMA3 degrade under several AHP strategies.
  • RQ2: Generated Criteria: Single-Agent criteria are judged most aligned with human reasoning in many cases, including 47 for Mistral, 53 for LLaMA3, and 40 for GPT-4o-mini.AHP-Instruction is selected less often, including 13 cases for GPT-4o-mini, while Multi-Agent reaches 38 for Qwen2.5.
  • RQ3: Human Decisions: 80.0% accuracy is reached by GPT-4o-mini under AHP-Instruction against human-annotated LEGAL-AHP decisions, improving over its automatic-evaluation result of 76.0%.Qwen2.5 remains relatively stable, whereas Mistral and LLAMA3 show limited gains; Multi-Agent Auto is therefore tested on Qwen2.5 and GPT-4o-mini.
  • Cross-result comparison: Criteria-level human alignment and final decision accuracy favor different strategies, with Single-Agent closer at the criterion level and AHP aggregation more stable in final accuracy.The results indicate that model capability and prompt-following stability affect how well models use structured decision procedures.
  • RQ4: Portability: AHP-based strategies improve ranking correlation for some model–benchmark pairs but not all, with Multi-Agent Auto helping GPT-4o-mini on QS and Qwen2.5-7B-Instruct on U.S. News.AHP-based rankings also produce more stable top-tier ordering, while zero-shot rankings fluctuate more.

6 Discussion

The discussion finds that structured AHP prompting improves interpretability and expert alignment, but benefits depend on model reasoning capacity and the decomposition of decision steps. The framework transfers to ranking tasks, while fully robust automation remains open.

  • Model dependence: Structured decision frameworks improve interpretability and alignment with expert judgments, especially for stronger models such as GPT-4o-mini and Qwen2.5.Weaker models struggle to follow multi-step analytical procedures reliably.
  • Automation and decomposition: LLMs can partially automate AHP by generating criteria and interpretable decisions, but reliability decreases as reasoning depth and coordination requirements increase.Single-agent formulations align more closely with human criteria, whereas AHP-guided aggregation yields more stable final accuracy.
  • Cross-domain portability: The same modular pipeline transfers to non-legal ranking problems without domain-specific knowledge by reusing criterion generation, weighting, and aggregation.Its portability is attributed to separating these reasoning stages.
  • Task formats: Binary decisions are generally easier for LLMs, while multiple-choice and comparative tasks benefit most from explicit structure and multi-criteria aggregation.Fully robust and interpretable automation in complex, high-stakes, cross-domain settings remains unresolved.

7 Conclusion

The paper presents an agent-based AHP framework that structures LLM reasoning into auditable criteria generation, weighting, and aggregation. It reports improved reflection of expert decision logic and generalization beyond legal tasks, while identifying further coordination and verification work.

  • Contribution: The framework transforms opaque LLM outputs into auditable decision processes through criterion generation, weighting, and aggregation.It integrates AHP with LLMs for transparent and interpretable multi-criteria decision-making.
  • Scope and future work: The framework generalizes beyond the legal domain to real-world ranking tasks without relying on domain-specific knowledge.Future work targets improved agent coordination, symbolic consistency verification, and broader high-stakes applications.

Ethical Considerations

The ethical discussion describes the dataset sources, annotation context, privacy handling, and the limits of automating context-dependent legal decisions. The intended role is assistance to legal experts rather than replacement.

  • Data and annotation: LEGAL-AHP uses question–answer pairs from publicly available datasets, and the source data are described as free of abusive content and privacy issues.The annotation campaign involved Master’s students in law as part of their academic training.
  • Privacy: The dataset is intended to be anonymized before release to protect privacy.
  • Scope of automation: Legal decisions are highly context dependent, making full automation impractical in complex settings; the framework is intended to assist experts by reducing cognitive load and supporting fair, transparent decisions.

Limitations

The study’s limitations concern dependence on model reasoning and coordination, subjectivity in expert annotations, and the small, labor-intensive LEGAL-AHP dataset.

  • Limitations: Effectiveness depends on the underlying models’ reasoning abilities and the stability of multi-agent coordination, with weaker models struggling on multi-step reasoning.The authors propose adaptive coordination and communication as future work to improve robustness across model capacities.
  • Limitations: Expert-defined criteria and pairwise comparisons introduce subjectivity that can affect the resulting criteria and weights despite consistency checks.Annotator expertise and interpretation may still influence the annotations.
  • Limitations: The LEGAL-AHP dataset is relatively small because AHP-based annotation requires multiple expert annotators and detailed reasoning justifications.This process limits scalability while producing a high-quality benchmark for structured decision-making and interpretability.
  • AHP framework: AHP organizes decisions into goals, criteria, and alternatives, then derives weights through pairwise comparisons and checks consistency using CR ≤0.1.This is the structured workflow instantiated by the paper’s multi-agent framework.
  • Multi-agent framework: The multi-agent pipeline separates contextual information processing from criterion generation, pairwise weighting, and final aggregation.Dedicated agents provide a clear separation of responsibilities in the representative legal classification workflow.

B.2 Training Phase

The training phase selected question subsets from LEGALBENCH and used standardized AHP guidelines to annotate criteria, comparisons, weights, and best answers.

  • Training Phase: Training used the first questions from each series, totaling 32 Abercrombie, 75 Common_Law, 105 Decision_Section, 120 Judicial_Ethics, and 120 Privacy_Policy questions.These counts define the training selection for each subset.
  • AHP-Based Annotation Guidelines: Annotators treated each question as the goal, answer options as alternatives, and the correct answer as the best alternative.The procedure was provided to student annotators to promote consistency and rigor.
  • AHP-Based Annotation Guidelines: Annotators identified relevant criteria, constructed pairwise comparison matrices using the AHP 1–9 scale, normalized them, and averaged rows to derive priority weights.Weights were computed for both criteria and alternatives under each criterion.
  • AHP-Based Annotation Guidelines: Overall alternative scores combined criterion and alternative weights, with the highest-scoring alternative designated the best answer.The resulting annotations were intended to provide a structured decision target.
  • Training Phase: The example annotation process was applied to Decision_Section question–answer pairs from the selected LEGALBENCH subsets.Table 10 illustrates a criteria comparison matrix supplied by annotators.

C Detailed Results in Terms of F1-scores

F1-score results show that AHP and agent-based strategies can improve strong models but are less stable across models and datasets than the Non-AHP baseline.

  • F1-score results: 78.8% F1 for GPT-4o-mini under AHP and 74.5% under Single-Agent exceed its 69.5% Non-AHP baseline.Qwen2.5 reaches 100.0% F1 on Common Law, while Mistral and LLAMA3 show instability under AHP.
  • F1-score results: Non-AHP remains the most stable baseline across models, ranging from 62.9% to 70.1%, whereas AHP-Instruction and Single-Agent results vary more.The structured strategies often help stronger models while degrading weaker ones.
  • F1-score results: GPT-4o-mini leads Abercrombie at 71.3% and Judicial Ethics at 86.6%, while GPT-4o-mini and Qwen2.5 both reach 100.0% on Common Law.On Decision Section, GPT-4o-mini peaks at 76.2% with Single-Agent and Qwen2.5 at 73.3% with Multi-Agent.
  • Portability to Ranking Decision Problems: On ranking tasks, Auto Multi-Agent AHP improves GPT-4o-mini on QS and substantially boosts Qwen2.5 on U.S. News, while matching the baseline for Qwen2.5 on QS.The comparison uses identical top-10 candidate settings for the zero-shot Non-AHP baseline and Auto Multi-Agent AHP.
  • Portability to Ranking Decision Problems: AHP-based rankings produce more stable top-tier university ordering, particularly among highly ranked universities.This result is reported in the position-level comparisons.
Loading 2609.16779v1…