Determinants of LLM-assisted Decision-Making

1 Introduction

• LLMs can summarize information, generate ideas and alternatives, identify patterns, analyze data, and simulate perspectives, but may also produce false, sycophantic, or deceptive outputs and amplify harmful content.The paper frames human-AI teams as combining human monitoring and contextual knowledge with AI's capacity to process complex patterns and larger amounts of data.Source: §1 · pp. 1–2
• Research has largely examined isolated factors, leaving no comprehensive overview of the determinants and their interplay in AI- or LLM-assisted decision-making.The paper notes prior work on transparency and explanations but says broader analysis of factor characteristics and interplay is lacking.Source: §1 · p. 2
• An integrative literature review provides a structural overview of technological, psychological, and decision-specific determinants, a dependency framework, and illustrations across multiple scenarios.The framework systematizes potential interactions and interdependencies among determinants.Source: §1 · p. 2
• The analysis identifies trust or reliance, users’ mental models, and information processing as key aspects because of their interactions with other determinants.The paper presents awareness of these factors as relevant to assessing LLM outputs and mitigating risks such as over-reliance.Source: §1 · p. 3

2.1 Human Decision-Making

• Human decision-making evaluates options against goals using a person’s skills, values, preferences, and beliefs, while situational factors such as time pressure also influence it.Simon’s process has intelligence, design, and choice phases, which can overlap or be revisited.Source: §2.1 · p. 3

2.2 Decision Support Systems and AI-assisted Decision-Making

• Decision-support systems help manage information and clarify judgments, while AI-assisted decisions typically leave the final judgment to the human decision-maker.The described cycle includes AI recommendations, human decisions, and environmental feedback that informs when to trust recommendations.Source: §2.2 · p. 4

2.3 Large Language Models

• LLMs perform language tasks and may show emergent abilities such as step-by-step reasoning and instruction following, alongside risks including misleading information and bias.The background defines an emergent ability as one present in larger models but absent in smaller ones.Source: §2.3 · p. 4

2.4 LLM-assisted Decision-Making Process

• During intelligence, LLMs can clarify problems and gather or summarize information; during design, they can structure problems, define criteria, and generate alternatives.In the choice phase, they can compare options against criteria, simulate scenarios, and assess potential risks.Source: §2.4 · pp. 4–5

3 Methodological Approach

• An integrative review identifies relevant literature, synthesizes findings, and derives assumptions about determinants and their possible interactions in LLM-assisted decision-making.The search starts with LLM-focused work, expands to AI and decision-support systems, and uses broader literature when domain-specific interaction research is absent.Source: §3 · pp. 5–7

3.1 Determinants Structure

• The analysis groups determinants as psychological, technological, and decision-specific, excludes environmental factors, and uses feature diagrams to represent determinant structure and interdependencies.The paper also organizes each determinant’s discussion around theoretical background, interactions, implications, and scenario illustrations.Source: §3.1 · pp. 7–8

3.2 Application Scenarios of LLM-assisted Decision-Making

• Six illustrative scenarios provide recurring contexts for examining how psychological, technological, and decision-specific determinants may shape LLM-assisted decisions.The scenarios span medical advice and diagnosis, weight-loss recommendations, vaccination, advertising strategy, and sales forecasting.Source: §3.2 · p. 9

4 Technological Determinants of LLM-assisted Decision Making

• The technological framework groups determinants into capabilities, transparency and explainability, trustworthiness, prompt engineering, application fields, and challenges, limitations, and risks.It also indicates that prompt elements and methods can influence decisions, while an application process uses a single field.Source: Figure 4 · p. 9, §4 · p. 10

Deriving Implications for LLM-assisted Decision-Making

• LLMs can support decisions by summarizing information, generating alternatives, simulating perspectives, analyzing data, and producing reasoned outputs.They also perform tasks such as question answering, translation, code generation, and information retrieval.Source: Deriving Implications for LLM-assisted… · p. 10, §4.1.1 · p. 10

4.2 Transparency and Explainability of LLMs

• Explanations may improve understanding and informed decisions, but LLM self-explanations may not accurately represent their underlying reasoning.In the medical-advice scenario, source transparency and explanations could help users assess whether recommendations rely on credible information.Source: §4.2.1 · p. 11, §4.2.3 · p. 11

4.3 Trustworthiness of LLMs

• Trustworthiness includes reliability, safety, misuse resistance, explainability, social norms, robustness, and fairness, while transparency is described as a potential determinant of trustworthiness.In the weight-loss scenario, disclosing sources, limitations, and risks could support a more informed decision.Source: §4.3.1 · p. 12, §4.3.2 · p. 12, §4.3.4 · p. 12

4.4.1 Theoretical Background

• Prompts establish task context and desired outputs; their elements include instructions, context, input data, and output indicators, while prompt styles vary across several dimensions.Methods include single-turn prompting, such as Chain-of-Thought, and multi-turn approaches such as Tree-of-Thoughts.Source: §4.4.1 · pp. 12–13

4.4.2 Interactions with other Determinants

• Prompting can expose steps or make responses more understandable, and some strategies aim to improve reasoning and compositional capabilities.The scenarios contrast an explanation-seeking sales forecast prompt with iterative medical prompting that permits added details and clarification.Source: §4.4.2 · p. 13, §4.4.3 · pp. 13–14

4.5.1 Theoretical Background

• LLM application fields range from general-purpose chatbots to biology, programming, creative work, law, medicine, reasoning, robotics, and social sciences.The section presents these as potential application domains for decision-making support.Source: §4.5.1 · p. 14

4.6.1 Theoretical Background

• LLM limitations include difficult-to-validate training data, prompt brittleness, outdated knowledge, hallucinations, misaligned behavior, and social and ethical risks.The section also notes explainability challenges, risks of over-reliance, and research concerns including limited experimental design and reproducibility.Source: §4.6.1 · pp. 14–15, §4.6.2 · p. 15

5 Psychological Determinants of LLM-assisted Decision-Making

• The framework identifies trust or reliance, the user’s mental model, information processing, emotions and mood, metacognitions, and decision-making styles as psychological determinants.Figure 5 presents these as determinants in LLM-assisted decision-making.Source: §5 · p. 15, Figure 5 · p. 15
• For trust and reliance, only one of adequate, under-, or over-reliance can affect a decision; the user’s mental model and the decision problem also influence it..Source: §5 · p. 15
• Intuitive thinking always affects the process, whereas deliberate thinking may not; either positive or negative feelings can matter, and monitoring and/or controlling may occur.Only one of optimizing, satisficing, or minimizing can influence a decision process.Source: §5 · p. 16

5.1.1 Theoretical Background

• Appropriate reliance means accepting correct AI predictions and rejecting incorrect ones; over-reliance fails to correct wrong predictions, while under-reliance rejects correct ones.Trust is an attitude that influences reliance, which is behavior following system advice.Source: §5.1.1 · p. 16

5.1.2 Interactions with other Determinants

• Transparency may increase trust by improving perceived effectiveness, but it may also reduce trust by increasing discomfort.Insufficient trust can become a barrier to delegating tasks or decisions to intelligent systems.Source: §5.1.2 · pp. 16–17

5.1.4 Scenario-based Demonstration

• A physician’s uncritical acceptance of a rare-disease diagnosis and a user’s adoption of an extreme diet illustrate over-reliance on LLM advice.Both scenarios describe users accepting recommendations without sufficient scrutiny or verification.Source: §5.1.4 · p. 17
• A marketing manager’s rejection of an LLM’s suggestion illustrates under-reliance, while a sales manager’s scenario links transparency about assumptions and rationale with trust.The sales scenario says transparency might influence trust in projections.Source: §5.1.4 · p. 17

5.2 Mental Model

• Mental models are cognitive representations that shape how people filter and interpret information and understand a decision problem, process, outcome, and feedback.They reflect knowledge, experience, values, beliefs, expectations, and aspirations.Source: §5.2.1 · p. 18
• Understanding an AI system’s capabilities and errors helps users decide when to accept or reject its recommendations.Users develop mental models through interaction, and an accurate understanding of system strengths and weaknesses supports better decisions about following advice.Source: §5.2.1 · p. 18

5.2.2 Interactions with other Determinants

• Explanations can support mental-model maintenance or building, while incomplete or inaccurate mental models are associated with inappropriate trust and reliance.Research cited in the section connects sound, complete explanations with more accurate understanding and novice users’ limited error recognition with over-reliance.Source: §5.2.2 · p. 18
• An accurate LLM mental model can help users set prompts, interpret outputs, recognize errors, and calibrate trust and reliance.Inaccurate models may lead to either over-reliance or under-reliance; users may revise their expectations through repeated interaction.Source: §5.2.2 · pp. 18–19

5.2.4 Scenario-based Demonstration

• For sales projections, awareness of historical-data bias and missing market context could prompt manual review using expertise, recent developments, or colleagues’ insights.The scenario notes that past trends may not reflect future conditions and that LLMs may miss market disruptions.Source: §5.2.4 · pp. 19–20
• More detailed prompts could have asked for common and rare diagnoses with probabilities, while vaccination advice should be cross-checked against reputable medical sources and professionals.The scenarios connect prompt specificity with more targeted responses and recommend verification of LLM health information.Source: §5.2.4 · p. 20

5.3.1 Theoretical Background

• Intuitive processing is fast and resource-efficient but may reduce precision, while deliberate thinking is slower and cognitively demanding.Heuristics can simplify decisions but may also contribute to cognitive biases and sub-optimal choices.Source: §5.3.1 · p. 20

5.3.2 Interactions with other Determinants

• Mental models guide information search but can exclude valuable information, contributing to confirmation bias and neglect of alternative possibilities.Confirmation bias involves seeking or favoring belief-consistent information while discounting contradictory evidence.Source: §5.3.2 · p. 21
• Analytical reasoning and cognitive forcing strategies may reduce over-reliance on AI, while reliance may also reflect the benefits and costs of checking an answer.The section notes that people may be less likely to over-rely when correct answers are valuable or when determining them is relatively inexpensive.Source: §5.3.2 · p. 21
• An LLM’s first output can act as an anchor, limiting users’ adjustment of its recommendation and exploration of alternatives.The section also proposes analytical reasoning as a possible check on over-reliance and describes belief-consistent attention as a risk in LLM use.Source: §5.3.2 · p. 21

5.3.4 Scenario-based Demonstration

• In the scenarios, an initial platform recommendation may narrow Jenna’s search, while David may focus on information supporting a suggested rare illness.Both examples describe attention to confirming information and reduced consideration of alternatives.Source: §5.3.4 · p. 22

5.4.1 Theoretical Background

• Emotions are stimulus-linked and relatively intense but brief, whereas moods are less intense, longer-lasting, and not usually tied to a specific stimulus.The section also describes emotions as capable of both beneficial and harmful effects on decisions.Source: §5.4.1 · p. 22

5.4.2 Interactions with other Determinants

• Positive moods are associated with more global, heuristic processing, whereas negative moods are associated with more analytical and systematic processing.The section presents these patterns through the Affect-as-Information Hypothesis and Affect Infusion Model.Source: §5.4.2 · p. 22
• Positive feelings may encourage intuitive acceptance of LLM suggestions, while negative emotional states may prompt more deliberate evaluation.These are proposed implications drawn from the cited affect models.Source: §5.4.2 · p. 23
• Emotional security and satisfaction may increase trust and reliance on LLMs, while frustration or anger may reduce trust and willingness to use them.Incidental emotions may also affect trust: happiness or gratitude may increase it, while anxiety or sadness may reduce it.Source: §5.4.2 · p. 23

5.4.4 Scenario-based Demonstration

• In the scenarios, positive affect may lead Alex to accept projections with less scrutiny, while negative affect may prompt Anna to seek validation and weigh risks.Positive affect is also linked to confirmation bias in Alex’s scenario; negative feelings are linked to skepticism toward vaccination advice in Paula’s scenario.Source: §5.4.4 · p. 23
• Jenna’s positive feelings may increase trust in the LLM’s marketing recommendation, while Paula’s concern may predispose her to distrust the forum’s advice.The scenarios connect these different emotional states with trust in LLM recommendations.Source: §5.4.4 · pp. 23–24

5.5.1 Theoretical Background

• Meta-reasoning monitors and controls reasoning, problem solving, and decisions; confidence may favor a choice, while uncertainty may prompt more information or another approach [177, p. 275].Monitoring assesses performance; control adjusts the effort devoted to a task.Source: §5.5.1 · p. 24

5.5.2 Interactions with other Determinants

• A strong Feeling of Rightness may discourage reconsideration and risk over-reliance on LLM answers, whereas a Feeling of Error may prompt users to question them and check for inaccuracies.A weak Feeling of Rightness can prompt analytical problem-solving; a Feeling of Error can encourage strategy changes or review of a mental action.Source: §5.5.2 · p. 24

5.5.4 Scenario-based Demonstration

• In the medical-advice scenario, a strong Feeling of Rightness may lead David to accept a rare-illness suggestion without seeking further consultation.The scenario describes this as a risk of over-reliance that could leave other explanations unexplored.Source: §5.5.4 · pp. 24–25
• In the sales-forecast scenario, a Feeling of Error may lead Alex to analyze and verify LLM forecasts, serving as a safeguard against over-reliance.Source: §5.5.4 · p. 25

5.6.1 Theoretical Background

• Minimizers seek a minimum result with little effort, satisficers stop at options meeting criteria, and maximizers search broadly for the best outcome.Decision styles tend to remain relatively consistent across domains, and maximizers typically invest more effort searching for alternatives.Source: §5.6.1 · p. 25

5.6.2 Interactions with other Determinants

• Minimizers are associated with fast processing, maximizers with slow processing, and satisficers may use either depending on decision complexity.Source: §5.6.2 · p. 25
• Minimizers may accept the first adequate LLM suggestion, satisficers may filter options by criteria, and maximizers may use LLMs for extensive comparison and research.The paper suggests minimizers and satisficers may be more prone to over-reliance, while maximizers may be less likely to over-rely because they critically evaluate suggestions and consider multiple sources.Source: §5.6.2 · pp. 25–26

5.6.4 Scenario-based Demonstration

• In the weight-loss scenario, Anna’s minimizing style may lead her to accept an extreme diet suggestion, while a maximizing style may prompt broader research into its effects and alternatives.A satisficing approach would use criteria such as dietary preferences and potential health risks to filter options.Source: §5.6.4 · p. 26

6 Decision-specific Determinants of LLM-assisted Decision Making

• The decision-specific determinants considered are task difficulty, decision reversibility, accountability, and personal significance.The section presents these factors in Figure 6.Source: §6 · p. 26, Figure 6 · p. 26

6.1 Theoretical Background

• Decision type and environment can shift strategy from heuristic to analytical, while complex situations may demand more information than human cognitive abilities can handle.Source: §6.1 · p. 26

6.2 Interactions with other Determinants

• Difficult tasks may increase over-reliance on decision aids, although findings vary and user expertise may reduce task difficulty’s influence on reliance [18].Source: §6.2 · p. 26
• For LLM use, perceived difficulty may increase reliance when users lack expertise, while growing expertise may reduce reliance even on complex tasks.Perceived irreversibility, accountability, and personal significance may promote deliberate processing and potentially mitigate over-reliance.Source: §6.2 · p. 27

6.4 Scenario-based Demonstration

• In the sales-projection scenario, perceived irreversible consequences and accountability may lead Alex to scrutinize forecasts, ask for reasoning, and compare projections with market knowledge and historical data.Source: §6.4 · p. 27

• The dependency framework systematizes potential reciprocal interdependencies among technological, psychological, and decision-specific determinants, using a feature diagram and dependency matrix.The paper draws it from a multidisciplinary literature review and determinant-specific analyses.Source: §7 · p. 27
• Trust in or reliance on LLMs is influenced by the greatest number of factors, including emotions, mood, mental models, information processing, transparency, and trustworthiness.The analysis reports the greatest number of interactions among psychological factors and the fewest at the decision-specific level.Source: §7 · p. 28
• The user’s mental model both shapes and is shaped by other factors, including information processing, prompting techniques, and perceived decision difficulty.Source: §7 · p. 28
• Information processing is linked to emotions, decision significance and reversibility, and the extent of reliance on LLMs.Source: §7 · p. 28
• The dependency matrix shows prompt engineering can affect LLM capabilities, transparency, explainability, and the user’s mental model, while the mental model also influences prompting.Source: §7 · p. 28

8.1 Implications for Decision-Makers and Organizations

• Training should build users’ understanding of LLM capabilities and constraints, critical evaluation of outputs, and prompt-formulation skills.The discussion also recommends explaining how models analyze and generate information.Source: §8.1 · p. 29
• Addressing over- and under-reliance calls for individual training and organizational practices, including monitoring decisions, feedback mechanisms, and guidance on LLM recommendations.The proposed feedback repository records employee observations and organizational responses.Source: §8.1 · p. 29
• Training can address emotional awareness, metacognitive cues such as FOR and FOE, and decision styles, while organizations should encourage critical review of high-stakes or irreversible choices.A strong Feeling of Rightness may accompany uncritical acceptance; tailored training addresses minimizers, satisficers, and maximizers.Source: §8.1 · pp. 29–30

8.2 Limitations

• The review may inherit selection and publication bias, did not systematically collect and synthesize prior research, and faces relevance challenges as LLMs rapidly develop.The authors expect some psychological and decision-specific influences to remain relatively stable despite technological change.Source: §8.2 · p. 30
• Scope is bounded by omitted organizational and environmental factors, other determinants such as time pressure, predominantly theoretical interactions, and exclusion of multimodal models.Organizational factors were excluded as beyond individuals’ direct control; environmental factors were considered difficult for decision-makers or organizations to influence.Source: §8.2 · pp. 30–31

8.3 Future Work

• Further research could use systematic reviews and experiments to assess determinants and test whether variations in them systematically affect decision behavior.The authors suggest studying how emotions relate to recommendation acceptance and reliance on LLM suggestions.Source: §8.3 · p. 31
• Other directions include studying organizational norms, evaluating training, synthesizing real-world cases, and identifying determinants of decisions supported by multimodal models.Case analyses could compare decision complexity, application domains, LLM involvement, psychological factors, and outcomes.Source: §8.3 · pp. 31–32
• Future work should examine multi-agent settings and investigate interactions involving trust or reliance, users’ mental models, and information processing.The proposed multi-agent direction includes interactions among human users, LLM agents, and system-embedded rules.Source: §8.3 · pp. 31–32

• The analysis covers technological, psychological, and decision-specific determinants and illustrates their potential influence through scenarios.Source: §9 · p. 32
• The dependency framework systematizes and visualizes reciprocal interdependencies, with trust or reliance, users’ mental models, and information processing identified as likely key aspects.Source: §9 · p. 33
• The overview and framework can facilitate explanation and prediction of user behavior and provide a foundation for future empirical studies.Source: §9 · p. 33