Source-linked AI summary
Overcoming Challenges of Interpretive Structural Modeling with Large Language Models
Everett Rush, David J. Icove, Ari Kim, Byung H. Park, Michael A. Langston
TL;DR
ISM relies on repeated expert interactions to construct factor relationships, and prior work had not comprehensively studied LLMs in the ISM process. This paper integrates LLM-based graph discovery into ISM and benchmarks several prompting approaches. Rowwise and full-graph methods perform best across the tested models.
Problem
ISM’s repeated SME interactions are labor-intensive and limit scale, while comprehensive study of LLM use in ISM has been lacking.
Source: §1 · pp. 1–2Method
The paper integrates LLMs into ISM by discovering the transitive closure of the SSIM with iterative queries of varying size, optionally using RAG.
Source: §3 · pp. 4–5Results
Across the tested models, rowwise and full-graph methods performed best; GPT-5.2 rowwise achieved F1-score 0.77 using RAG.
Source: §4.2 · p. 8, §5 · p. 9Takeaways & Limitations
The results support the suitability of LLMs to aid SSIM discovery in ISM.
Source: §5 · pp. 8–9The study is inference-only: the dataset is treated as test-only, and SHD and F1-score are reported directly.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Interpretive Structural Modeling (ISM) is a well-known process for multi-criteria decision making. The success of ISM over other methodologies is its ability to model causal relationships, the binary scale of factors, and resulting hierarchical representation. Traditionally, the modeling process is performed by repeated interactions with subject matter experts until consensus is reached. This process is tedious, labor-intense, and most importantly limits the ability of ISM to scale to studies with hundreds of variables. Drawing on existing work of causal graph discovery with large language models (LLM) as imperfect experts, this work explores an integrated LLM-ISM approach for ISM. Pairwise, k-wise, rowwise, and full graph discovery methodologies are compared and evaluated. It is shown that causal graph discovery methods for ISM perform best using rowwise (SHD=160, F1-score=0.77) and full graph methods (SHD=135, F1-score=0.73).
1 Introduction
ISM models causal relationships among factors in a hierarchy, but its expert-driven construction is labor-intensive and limits scale. This paper proposes an integrated LLM-ISM approach, addressing a gap in comprehensive studies of LLMs in ISM.
1 Introduction
- ISM organizes mental models into hierarchical models and represents causal relationships among factors on a binary scale.It has been applied to problems across supply chain, logistics, sustainability, business research, and engineering.Source: §1 · p. 1
- Repeated SME interactions to reach consensus make ISM modeling labor-intensive, error-prone, and difficult to scale beyond a small number of factors.Most ISM studies use 5–32 factors, with SMEs annotating all pairwise interactions and sometimes completing multiple rounds to resolve biases.Source: §1 · pp. 1–2
- Although LLMs have guided causal inference frameworks, no comprehensive study had examined their use in the ISM process.ISM is used when little knowledge or data are available about pairwise factor connections, so the process relies heavily on SMEs rather than statistical causal inference and numerical observations.Source: §1 · p. 2
- The paper describes an integrated LLM-ISM approach drawing on LLMs’ causal graph discovery and general knowledge.The authors also state that the tool is publicly available on GitHub.Source: §1 · p. 2
2 Background
The background presents ISM as a staged process for organizing factors and their relationships, then introduces LLM-based causal graph discovery methods and a challenge involving indirect relationships.
2.1 Interpretive Structural Modeling
- ISM supports theory development by addressing what factors are involved and how they relate; its process identifies factors, forms an SSIM, computes transitive closure, partitions levels, and extracts a model.The framework has preparation, operation, and post-operation phases; SSIM construction may rely on literature reviews, SMEs, or surveys, which can be costly and time-consuming.Source: §2.1 · pp. 2–3
2.2 Causal Graph Discovery
- ISM uses a general causal-graph definition that allows cycles, unlike definitions that require directed acyclic graphs.The passage gives a cyclic temperature and sea-ice relationship as an example.Source: §2.2 · p. 3
- Causal graph prompting ranges from pairwise queries to k-wise and rowwise prompts, while full-graph prompting asks for the entire graph in one inference.The figure depicts these prompt scopes for a 12×12 adjacency matrix.Source: §2.2 · p. 3, Figure 1 · p. 4
- An indirect causal relationship can be missed when an intermediate factor is unavailable, motivating the paper’s definition of a perfect expert using auxiliary nodes.The perfect expert answers a pairwise query using direct edges or paths that avoid the supplied auxiliary nodes.Source: §2.2 · pp. 3–4
- Prior work reports that querying a perfect expert with no auxiliary nodes can fail to recover the true causal graph.The supplied proposition passage identifies the result in terms of structural Hamming distance, but its displayed expression is incomplete.Source: §2.2 · p. 4
3 Methods
The method discovers the transitive closure of the SSIM through iterative LLM queries, with prompt size controlling how many relationships are considered per query. It also describes an optional retrieval-augmented extension and evaluates the approach using three models and a benchmark drawn from nine publications.
3 Methods
- The approach discovers SSIM* rather than SSIM directly, uses a general causal-graph definition that includes cycles, and optionally adds retrieval-augmented generation.The authors argue that SSIM is an intermediate artifact because subsequent ISM analysis uses its transitive closure, avoiding an inherent auxiliary-factor penalty associated with discovering SSIM directly.Source: §3 · p. 4
- The iterative algorithm varies query size k: pairwise mode makes O(n^2) queries, while k = n handles n edges per request and makes O(n) queries.For each variable, the LLM predicts outgoing causal edges to the variables in the current list, and the predicted graph is then closed transitively.Source: §3 · p. 5, §3.1 · p. 6
3.1 Incorporating SME Knowledge
- The optional in-context learning extension passes documents to the algorithm and retrieves relevant text chunks using the research question and factors under consideration.The retrieved chunks are included in the LLM query; documents are converted to text using Docling [14].Source: §3.1 · pp. 5–6
3.2 Experimental Setup
- The benchmark is evaluated with GPT-5.2, GPT-OSS 120B, and Nemotron Super 3 on nine peer-reviewed ISM publications that did not rely solely on interviews or surveys.Factors tables and final reachability matrices were extracted for evaluation.Source: §3.2 · p. 6, §3.3 · p. 6
4 Results
The evaluation compares LLM prompting methods for ISM graph discovery across nine benchmark studies. GPT-5.2 rowwise achieved the highest F1-score, while GPT-OSS-120B full graph achieved the lowest SHD, both with RAG.
4 Results
- With RAG, the method retrieves text chunks for factor pairs, uses them in LLM edge prediction, and returns the predicted graph's transitive closure.Source: §4 · p. 7
4.2 Methodology Evaluation
- The evaluation treats the benchmark as test-only, sums per-study SHD, and aggregates true positives, false positives, and false negatives for F1-score.Source: §4 · p. 7, §4.2 · pp. 7–8
4.1 Graph Statistics
- The nine benchmark graphs ranged from 7 to 17 nodes and from 0.31 to 0.57 edge density, averaging 11.7 nodes and 55.4 edges.Source: §4.1 · p. 7, Table 1 · p. 8
4.2 Methodology Evaluation
- All models performed better as the number of comparisons in a single prompt increased.Source: §4.2 · p. 8
- 0.77 F1-score for GPT-5.2 rowwise and 135 SHD for GPT-OSS-120B full graph were the best respective results, both using RAG.The comparison appears in Table 2.Source: Table 2 · p. 8, §4.2 · p. 8
5 Conclusion
The paper concludes that LLMs are suitable for aiding SSIM discovery, with GPT-5.2 outperforming the other evaluated models and rowwise and full graph methods performing best across models. It identifies larger-variable studies, PTSD care, and agentic workflows as future directions.
5 Conclusion
- LLMs are suitable for aiding SSIM discovery; GPT-5.2 outperformed the other evaluated models, and rowwise and full graph methods performed best across models.Source: §5 · pp. 8–9
5.1 Future Work
- The methodology is presented as able to scale to studies with more variables than traditional ISM studies and examine variables at a detailed, granular level.Source: §5.1 · p. 9
- The authors identify applying the methodology to PTSD care as future work, to model hierarchical relationships between treatment and recovery barriers and enablers.Source: §5.1 · p. 9
- Future agentic workflows could use ISM graphs for in-context planning or execution, or incorporate them into LLM training to support learned causal inference.Source: §5.1 · p. 9