PaperlayerView exact revision on arXiv

Source-linked AI summary

Behavioral Indicators of Overreliance During Interaction with Conversational Language Models

Chang Liu, Qinyi Zhou, Xinjie Shen, Xingyu Bruce Liu, Tongshuang Wu, Xiang 'Anthony' Chen

arXiv:2602.11567v1cs.HC

TL;DR

Outcome-based measures make it difficult to characterize overreliance during complex conversational interactions. The authors analyzed logs from 77 participants across three misinformation-injected tasks and identified five behavioral patterns associated with overreliance, with implications for adaptive detection and mitigation.

  • Problem

    Outcome-oriented measures can miss interaction processes that distinguish forms of reliance, while conversational LLMs may present fluent misinformation that users accept.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    The authors collected interaction logs from 77 participants completing three tasks with plausible misinformation injected into LLM responses and analyzed behavior patterns against misinformation in final answers.

    Source: §1 · p. 2

  • Results

    Five behavioral patterns correlated with overreliance: low-overreliance users read and navigated carefully, while high-overreliance users often copied extensively and accepted responses despite hesitation.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    The behavioral patterns and analysis framework provide a foundation for research on adaptive strategies to detect and mitigate overreliance in real time.

    Source: §6 · p. 12, §7 · p. 14

    The same behaviors may imply different risks for different users or scenarios, so contextual factors are needed to calibrate detection thresholds and risk levels.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

LLMs are now embedded in a wide range of everyday scenarios. However, their inherent hallucinations risk hiding misinformation in fluent responses, raising concerns about overreliance on AI. Detecting overreliance is challenging, as it often arises in complex, dynamic contexts and cannot be easily captured by post-hoc task outcomes. In this work, we aim to investigate how users' behavioral patterns correlate with overreliance. We collected interaction logs from 77 participants working with an LLM injected plausible misinformation across three real-world tasks and we assessed overreliance by whether participants detected and corrected these errors. By semantically encoding and clustering segments of user interactions, we identified five behavioral patterns linked to overreliance: users with low overreliance show careful task comprehension and fine-grained navigation; users with high overreliance show frequent copy-paste, skipping initial comprehension, repeated LLM references, coarse locating, and accepting misinformation despite hesitation. We discuss design implications for mitigation.

1 Introduction

The introduction motivates studying interaction behavior because task outcomes alone can miss how overreliance develops across conversational turns. It reports five behavioral patterns associated with overreliance and describes a study and analysis framework for identifying them.

1 Introduction

  • Overreliance is defined as accepting incorrect LLM recommendations, whereas reliance is an observable action distinct from trust as an internal attitude.The conventional approach compares task outcomes with and without AI assistance.Source: §1 · p. 1
  • Task outcomes alone can overlook intermediate conversational interactions, making it difficult to trace overreliance-related errors to particular turns or causes.The paper therefore examines mouse- and keyboard-based behaviors as correlates of overreliance.Source: §1 · p. 1
  • The study collected interaction logs from 77 participants completing three tasks with plausible misinformation injected into LLM responses, then measured its influence on final answers.The misinformation simulated errors such as hallucinations, outdated information, and prompt injection while allowing control over content and placement.Source: §1 · p. 2
  • Users with low overreliance carefully read task details, verify LLM responses, and navigate selectively, while high-overreliance users often copy whole sections and adopt responses despite hesitation.High-overreliance users also repeatedly refer to LLM responses and navigate coarsely; the authors interpret patterns using cognitive theories and participants’ strategy reports.Source: §1 · p. 2
  • The authors contribute a dataset, a refined cluster-based framework for relating behavior to overreliance, and five behavioral patterns with empirical evidence, cognitive interpretation, and mitigation implications.The framework is intended to strengthen behavioral-to-reliance inference.Source: §1 · p. 2

2 Related Work

Prior work describes harms and mitigation challenges associated with overreliance, while research on interaction behaviors has not yet systematically organized behavioral correlates for conversational LLMs. This paper positions its approach as filling that process-level gap.

2.1 Overreliance on AI: Existing Frameworks and Unique Challenges of Conversational LLMs

  • Conversational LLMs can produce fluent misinformation, and existing mitigations—including explanations, uncertainty cues, and cognitive forcing—have trade-offs that motivate adaptive, real-time signals.Reported concerns include explanations backfiring, poor uncertainty calibration, and cognitive burden or under-reliance from forcing functions.Source: §2.1 · pp. 2–3

2.2 From Outcome to Process-Oriented Understandings of overreliance on LLMs

  • Outcome-oriented metrics infer overreliance from final task results but miss interaction processes and struggle to provide a unified measure across diverse real-world tasks.They may also fail to distinguish active rational reliance from passive blind compliance.Source: §2.2 · p. 3
  • Prior process-oriented studies propose separate behavioral or contextual signals, but the literature remains fragmented without a systematic quantified framework of behavioral patterns linked to overreliance.Examples include interaction inefficiencies, reading and editing behaviors, contextual features, and accuracy–time tradeoffs.Source: §2.2 · p. 3

2.3 Interaction Behaviors as Performance Correlates

  • Earlier work used interaction traces to characterize task processes, evaluate crowdwork, and predict task-related outcomes; generative-AI studies also linked interaction measures with productivity or collaboration.Examples include task fingerprinting, behavioral traces for worker-result selection, and AI-suggestion acceptance rates associated with perceived productivity.Source: §2.3 · p. 3
  • Although prior studies establish the feasibility of behavioral signals for performance-related measures, the cited work had not applied this approach to understanding LLM overreliance.This study characterizes behavior correlates as a basis for future real-time detection and adaptive mitigation.Source: §2.3 · p. 3

3 Data Collection Study

The study logged behavior from participants completing three tasks with plausible misinformation in LLM responses, measuring overreliance through the influence of that misinformation on final submissions. The tasks use different information sources and task-specific evaluation methods.

3 Data Collection Study

  • The experiment collected detailed action logs while participants completed tasks with plausible misinformation, defining overreliance by whether final submissions incorporated or were influenced by it.The controlled design aimed to measure overreliance while capturing fine-grained accompanying behavior.Source: §3 · pp. 3–4

3.1 Study Design

  • The online study recruited 77 participants, used a browser extension to collect task and LLM-page actions, and presented the task, LLM, and search pages in a split-screen interface.Participants completed surveys and tutorials, submitted answers and logs after each task, and completed a final questionnaire.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Task Design and Misinformation Injections

  • Task selection covered common LLM uses and three information sources for judging outputs—personal knowledge, limited context, and external information—with measurable outcomes.The selected tasks were quiz solving, article summarization, and trip planning; misinformation was injected to support systematic evaluation.Source: §3.2 · p. 4

3.2.1 Quiz Solving.

  • Quiz solving compared survival-item rankings with and without LLM assistance, injecting plausible incorrect rankings and measuring overreliance from the normalized difference in performance scores.The NASA standard scoring algorithm produced deviation scores, with lower scores indicating closer alignment to ground truth.Source: §3.2.1 · pp. 4–5

3.2.2 Article Summarization.

  • Article summarization tested whether participants assessed LLM interpretations against supplied content, with 11 context-dependent misinformation instances and retained misinformation counted in summaries.The LLM was instructed to persist with the injected errors even when challenged.Source: §3.2.2 · p. 5

3.2.3 Trip Planning.

  • Trip planning simulated time-sensitive information seeking; 20 verifiable misinformation instances were injected, and retained itinerary errors were assessed with GPT-4 using named-entity recognition and logical inference.The task asked participants to plan a Copenhagen trip for a specified date and normalized retained misinformation by the total injected instances.Source: §3.2.3 · pp. 5–6

4 Analysis

The analysis converts interaction logs into temporal segments, embeds variable-length action sequences, and clusters the resulting representations. It retains robust clusters for interpreting behavioral patterns associated with overreliance.

4 Analysis

  • The analysis pipeline preprocesses and vectorizes interaction logs, segments sessions at multiple temporal scales, embeds sequences, and clusters them to identify recurring patterns.Post-clustering selection retains robust, informative clusters for interpretation of cognitive and behavioral strategies.Source: §4 · p. 6, Figure 2 · p. 7
  • Sessions were divided into 10–60-second windows with a 1-second stride, balancing isolated events in shorter windows against repetitive sequences in longer ones.Each segment inherited the participant-task overreliance score, enabling localized behavior to be related to overall task performance.Source: §4 · p. 6
  • A transformer-based autoencoder represented variable-length action sequences in a low-dimensional latent space, with the encoder’s [CLS] output used as the clustering embedding.The decoder sought to reconstruct the input sequence.Source: §4 · p. 6
  • DBSCAN clustering explored multiple thresholds and neighborhood sizes, retaining clusters that recurred across parameterizations.Test-set cluster labels were assigned using five-nearest-neighbor classification in embedding space.Source: §4 · p. 7
  • Clusters were filtered for similar training/test overreliance distributions and predictive agreement, then representative sequences near each centroid were manually analyzed.The two criteria were a nonsignificant two-tailed t-test (p > 0.05) and the ability of training members’ average score to predict test members’ average score.Source: §4 · p. 7

5 Findings

Across 54 core clusters, five behavioral patterns distinguished levels of overreliance: careful task comprehension and fine-grained editing contrasted with repeated LLM consultation, coarse copying, and hesitation without verification.

5 Findings

  • 54 core clusters met the predictive-capability and overreliance-similarity criteria; 18 showed significantly high or low overreliance, while 36 showed no significant difference.The 18 salient clusters were the focus of the pattern analysis.Source: §5 · p. 7

5.1 Frequency of Copying-Pasting

  • In trip planning, high-overreliance users typically made 5–6 copy-paste actions per 60-second window, versus 1–3 for low-overreliance users, who sometimes revised pasted text.The authors interpret frequent copying as rapid, low-effort uptake, contrasting it with more deliberate review.Source: §5.1 · p. 9, Figure 4 · p. 10

5.2 Focused Task Comprehension at the Start

  • In article summarization and trip planning, low-overreliance users began by reading the task page; this pattern never appeared in high-overreliance clusters.The authors interpret this early task-focused phase as planning that can establish initial ideas or goals before consulting the LLM.Source: §5.2 · p. 9, Figure 4 · p. 10

5.3 Frequency of Referring LLM’s responses

  • In quiz solving, high-overreliance users switched between LLM and task pages 3–4 times per 10-second window, while low-overreliance users consulted once and ranked independently.The authors characterize repeated on-demand consultation as reactive cognitive offloading, contrasting it with first validating and internalizing responses.Source: §5.3 · pp. 9, 11, Figure 4 · p. 10

5.4 Coarse- vs. Fine-Grained Locating & Editing

  • In article summarization, high-overreliance users used rough, chunk-level copying, while low-overreliance users made precise edits and selectively incorporated LLM text.The authors interpret the contrast as large-block cognitive offloading versus effortful adjustment grounded in users’ own mental models.Source: Figure 4 · p. 10, §5.4 · p. 11

5.5 Pausing and Hesitation Before LLM Prompting

  • In quiz solving, a long task-page pause followed by LLM-page edits appeared in high-overreliance clusters but never in low-overreliance clusters.The authors interpret this as monitoring without control: users may sense inaccuracies yet continue without external verification; participants also described being convinced through discussion.Source: Figure 4 · p. 10, §5.5 · p. 11

6 Discussions

The discussion shifts attention from final task outcomes to interaction behaviors associated with overreliance, while emphasizing that task, model, and user factors can also shape those behaviors. It proposes context-sensitive interface interventions and identifies limits on the study’s ecological validity, generalizability, and cognitive validation.

6 Discussions

  • Outcome-only measures can miss the causes of errors across conversational turns; the authors characterize interaction behaviors and note that final-outcome scoring cannot track changes in overreliance during a task.They suggest real-time monitoring, including scrutiny time, acceptance latency, or physiological indicators.Source: §6 · pp. 11–12, §6.3.2 · p. 13

6.1.1 Task-related factors.

  • Task difficulty and the 15-minute limit may prompt shallow processing or rapid acceptance even without overreliance.Minimal verification can also reflect time-management strategies.Source: §6.1.1 · p. 12

6.1.2 LLM-related factors.

  • Fluent, authoritative, single-answer LLM responses may reduce scrutiny and anchor users even when they are cautious.Source: §6.1.2 · p. 12

6.1.3 User-related factors.

  • Differences in metacognitive awareness, motivation, and expertise can make similar behaviors reflect limited awareness, minimal engagement, or difficulty evaluating an answer rather than misplaced trust.The authors call for experimentally varying these factors to distinguish overreliance from rational strategies and user- or interface-driven effects.Source: §6.1.3 · p. 12

6.2 Design Recommendations for Conversational LLMs’ Interfaces

  • For repeated copy-pasting, shallow task entry, and frequent LLM references, the authors propose factual checks, task-goal confirmation, and question chunking, respectively.Source: §6.2 · p. 12
  • For coarse navigation and repeated similar prompts, they propose granular review and query-trail or Socratic interventions; thresholds should account for context because identical behaviors may carry different risks.Source: §6.2 · pp. 12–13

6.3.1 Simulating LLM-Generated Misinformation.

  • Injected misinformation was designed to appear natural, but it does not fully reproduce naturally occurring hallucinations’ stochastic nuances; prompting and fine-tuning have not been comprehensively compared for misinformation generation with human evaluation.The authors note that prior evidence finds comparable results on many general tasks [107].Source: §6.3.1 · p. 13

6.3 Limitations & Future Work

  • The study’s constrained editing tasks and low-risk setting leave generalization to open-ended, generative, or high-stakes tasks unverified; post-task strategy reports can also be vague, and the collected data are insufficient for a predictive model.In broader tasks, overreliance may involve diminished creativity, homogenized content, or logical inconsistencies, while cognitive validation could use real-time assessment methods.Source: §6.3.2 · p. 13, §6.3.3 · p. 13, §6.3.4 · p. 13

7 Conclusion

The study uses interaction data and cluster-based analyses to examine behavioral correlates of overreliance on conversational LLMs. It reports significant correlations between overreliance and specific interaction patterns, and presents the dataset as a foundation for future adaptive detection and mitigation research.

7 Conclusion

  • Cluster-based analyses found significant correlations between overreliance and interaction patterns, including increased copy-paste actions and varying interaction frequencies on task and LLM pages.Source: §7 · p. 13
  • The work provides a dataset of user interaction behaviors related to overreliance on LLMs.Source: §7 · p. 14
  • The dataset and findings are presented as a foundation for future research on adaptive strategies to detect and mitigate overreliance in real time.Source: §7 · p. 14

A.1 Task Details

The task is framed as a scenario in which users may rely on AI misinformation despite having knowledge that could serve them better.

A.1 Task Details

  • The task operationalizes a scenario where users rely on AI misinformation even when their own knowledge could serve them better.Source: §A.1.1 · p. 17
  • The scenario focuses on users’ reliance on misinformation from AI.Source: §A.1.1 · p. 17
  • The task contrasts AI misinformation with the possibility that users’ own knowledge would be more useful.Source: §A.1.1 · p. 17

A.1.1 Quiz Solving (15 minutes for each trials).

The quiz tasks adapt NASA survival exercises: participants rank items for survival in a harsh environment, and two comparable environments are used to mitigate learning and order effects.

A.1.1 Quiz Solving (15 minutes for each trials).

  • Participants imagine a harsh environment and rank 15 items by importance for survival, adapting NASA survival exercises.Source: §A.1.1 · p. 17
  • The quiz relies on common sense, and participants complete it without external information because the original exercise prohibits external tools.Source: §A.1.1 · p. 17
  • Two comparably difficult tasks—desert and moon survival—were used to mitigate learning effects, with AI-assistance condition and environment order counterbalanced.Source: §A.1.1 · p. 17

A.1.2 Article Summarization (15 minutes).

The article-summarization task asked participants to summarize a science article with LLM assistance, using only that article as their source. It was designed to reflect AI processing of source information that may contain hallucinations.

A.1.2 Article Summarization (15 minutes).

  • The task modeled using AI to process information from a specific source, such as summarizing, extracting details, or answering questions.The source-based AI response could be inaccurate because of hallucination.Source: §A.1.2 · p. 17
  • Participants read and summarized a science article about sneezing, with no search engines allowed.The article was chosen for its familiar topic and because about 3% of its words involved complex vocabulary or concepts.Source: §A.1.2 · p. 17

A.1.3 Trip Planning (15 minutes).

The trip-planning task asked participants to prepare a Copenhagen trip by listing destinations and relevant details, which they should verify online because the information is time-sensitive.

A.1.3 Trip Planning (15 minutes).

  • Participants prepared for a Copenhagen trip by listing specific destinations and relevant details.Source: §A.1.3 · p. 17
  • Participants were expected to search the internet for correct information because trip details are time-sensitive.Source: §A.1.3 · p. 17
  • The task centered on preparing travel plans with current destination information.Source: §A.1.3 · p. 17

A.2 Adjusting the Quiz Solving

This section covers quiz calibration and scoring, participant-log processing and analysis, the clustering-method foundation, and pilot studies used to develop and calibrate the study methods and tasks. It also reports that quiz scores were similar across the two tasks without an LLM and higher with an LLM in each task.

A.2 Adjusting the Quiz Solving; A.3 Details on Computing Scores for Quiz Solving

  • Pilot calibration removed three items from each exercise (N=12), eliminating the score difference (p=0.08); scores sum absolute item-rank deviations from expert rankings.The ranking adjustment fixed items with minimal pilot ranking differences and moved other items farther from their original positions.Source: §A.2 · p. 17, §A.3 · p. 18

A.4 Informed Consent Statement; A.12 Questionaires after Each Task

  • Participants were told browser mouse and keyboard inputs and LLM communications would be collected and kept anonymous; post-task questions asked about deception and how they reached and used their final answer.The survey used 1-to-7 agreement ratings for system-related questions.Source: §A.4 · p. 18, §A.12 · pp. 20, 22

A.6 Distributions of Participants’ Scores in Tasks

  • Desert and moon scores without LLM did not significantly differ (p>0.05), while scores with LLM were significantly higher than each corresponding no-LLM score (p<0.05).The authors state these results align with their hypotheses that participants would perform similarly across subtasks and worse with conversational LLMs that generate misinformation.Source: Figure 5 · p. 18, §A.6 · p. 18

A.7 The Filtering Metrics Values for Different Tasks and Windows; A.8 Raw Data from Behavior Clustering

  • Figure 6 reports filtering-metric values by task and window; raw-cluster visualizations order action-type blocks over time and distinguish Task-page from LLM-page actions.The same visualization format was used for typical sequences across selected clusters, windows, and tasks.Source: §A.7 · p. 18, Figure 6 · p. 19, §A.8.1 · p. 19

A.10 Computing Predictive Score for Test Action Sequence; A.11 Overreliance Levels

  • For each test sequence, predicted overreliance averaged the actual scores of its five nearest same-cluster training neighbors; cluster levels were high, low, or neutral relative to the remaining data.The analysis compared each cluster’s score with the rest of the data within each task-window combination.Source: §A.10 · p. 20, §A.11 · p. 20

A.5 Preprocessing Details; A.13 Analysis Pipeline Details

  • Raw logs were cleaned for incomplete or corrupted records, aligned chronologically across task and LLM pages, and aggregated into higher-level actions; event preprocessing merged consecutive movements, scrolling, keypresses, and deletes.The retained logs numbered 62 for quiz solving, 70 for article summarization, and 60 for trip planning.Source: §A.5 · p. 18, §A.13.1 · p. 22

A.9 Explanation of Each Dimension for Action Vector; A.13.2 Action Vector Encoding.

  • Action sequences were encoded as 37-dimensional vectors comprising action type, normalized timestamp, page context, and 19 action-specific features.Continuous attributes were log-transformed, while categorical attributes used one-hot or binary encoding.Source: §A.9 · p. 19, §A.13.2 · p. 22

A.13.2 Action Vector Encoding.; A.13.4 Clustering Procedure.

  • A Transformer autoencoder compressed sequences into 64-dimensional embeddings for DBSCAN clustering, retaining clusters recurring in at least three parameterizations.Cluster membership for unseen points used 5-nearest-neighbor classification; clusters were retained if train/test scores did not differ significantly (p> .05) and their mean-score gap was below δ=0.15.Source: §A.13.3 · p. 22, §A.13.4 · p. 23, §A.13.5 · p. 23

A.13.4 Clustering Procedure.

  • The clustering methods provide the technical foundation for Section 5’s qualitative interpretation of representative clusters and their alignment with LLM overreliance dynamics.Source: §A.13.6 · p. 23

A.14 Pilot Study

  • Pilot studies with 21 participants (9F, 12M) tested methods for capturing cognitive ground truth, calibrating task difficulty and constraints, and evaluating task categories.Source: §A.14 · p. 23
Loading 2602.11567v1…