PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents

Jiatong Li, Changdae Oh, Hyeong Kyu Choi, Jindong Wang, Sharon Li

arXiv:2602.07796v2cs.CL

TL;DR

User-service agents must communicate decision-relevant information because users and agents hold different parts of the task state, yet private reasoning may not become timely user-facing disclosure. The paper defines and measures information disclosure, evaluates the think tool across models and scenarios, and uses probing and counterfactual intervention to test its role. It finds that golden disclosure most clearly improves immediate environment-edit behavior, while disclosure-aware prompting mitigates enforced-thinking degradation.

  • Problem

    User-service task success depends on agent-user coordination under asymmetric information, but internal reasoning does not reliably translate into timely disclosure of information users need for decisions.

  • Method

    The paper evaluates think-tool modes across five models and three scenarios, defines five disclosure types, and combines evidence-grounded probing with turn-level counterfactual intervention.

  • Results

    Injecting golden decision-relevant information produces the clearest improvement in First Edit Correction Rate, while disclosure-aware prompting mitigates enforced-thinking degradation.

  • Takeaways & Limitations

    Information disclosure is both a diagnostic lens and a practical design target for converting environment knowledge into timely, user-facing communication.

  • Takeaways & Limitations

    Gemini-2.5-Pro was excluded because an unresolved Vertex AI issue introduced confounding failures.

Abstract

from arXiv · show

User-engaged LLM agents increasingly operate in service scenarios where task success depends on coordination between the agent, the user, and a stateful environment. In such interactions, the agent often has access to task policies, tool results, and environment states that the user does not observe. This makes agent-user communication a central component of task completion. In this work, we study a communication-level failure mode of user-service agents: user-invisible reasoning does not necessarily translate into timely user-facing information disclosure. We introduce information disclosure as a measurable communication mechanism, covering state corrections, available options, constraints, consequences, and proactively retrieved environment information. We use this perspective to analyze the widely used user-invisible ``think'' tool, which allows agents to reason internally before responding or acting. Across five models and three user-service scenarios, we first show that enforced thinking has unstable utility. We then analyze response scaling under different think efforts and find that more internal thinking does not reliably translate into richer user-facing communication. Using evidence-grounded probing and turn-level counterfactual intervention, we show that injecting golden decision-relevant information produces the clearest improvement in First Edit Correction Rate, outperforming verbosity and direct think-content controls. This indicates that missing disclosure can causally affect the agent's immediate downstream environment-edit behavior. We further find that a lightweight disclosure-aware prompt mitigates degradation from enforced thinking. These results suggest that the key challenge is not simply making agents think more, but ensuring that private reasoning and environment knowledge are externalized as timely, user-facing information disclosure.

1. Introduction

User-service agents must coordinate with users and stateful environments under asymmetric information, so internal reasoning alone may not ensure timely disclosure of decision-relevant facts. The paper studies information disclosure and tests whether it improves downstream interaction behavior.

  • User-service task success depends on coordination among agents, users, and stateful environments under asymmetric information.
  • Enforced internal thinking does not reliably improve task success or reliability across five models, three scenarios, and two think modes.
  • Information disclosure is defined as proactive communication of decision-relevant state corrections, options, constraints, consequences, and retrieved environment information.
  • The study validates think-tool effects, analyzes think-effort scaling, probes disclosure, and applies turn-level counterfactual intervention.
  • Injecting golden decision-relevant information improves First Edit Correction Rate more than verbosity or think-content controls, indicating missing disclosure affects immediate environment-edit behavior.
  • A disclosure-aware prompt mitigates enforced-thinking degradation, supporting optimization of both private reasoning and timely user-facing communication.

2. Related Work

Prior work studies agent-user communication, internal reasoning, and tool use, but the paper focuses on a communication-level explanation for unstable think-tool utility. It frames externalization of decision-relevant information as distinct from private reasoning.

  • User-engaged agents communicate with users through natural language while interacting with external environments through formatted actions or tool calls.
  • Reasoning research includes chain-of-thought prompting, structured search, planning, reflection, and ReAct-style interleaving of reasoning with actions.
  • The think tool appends private reasoning to agent context without exposing it to users or modifying the environment.
  • This paper examines whether unstable think-tool utility reflects failures to externalize decision-relevant information during user-engaged tasks.

3. An Empirical Study of Think on Response Scaling

The empirical study evaluates enforced thinking across models and scenarios, then examines whether think effort scales user-visible responses and task reliability. A case study illustrates how private reasoning can omit a constraint before a consequential user decision.

  • 3.1. Preliminary Evaluation of the “Think” Tool: The agent compares original tools with a free-form internal think tool that neither queries nor modifies the environment.
  • 3.1. Preliminary Evaluation of the “Think” Tool: The evaluation covers GPT-4o, GPT-4.1, GPT-5, DeepSeek-V3.1, and gpt-oss-120b across Retail, Airline, and TS-Phone scenarios.
  • 3.1. Preliminary Evaluation of the “Think” Tool: Gemini-2.5-Pro was excluded because an unresolved Vertex AI issue introduced confounding failures.
  • 3.1. Preliminary Evaluation of the “Think” Tool: pass^2 drops from 63.48% to 52.46% in one reported comparison, while GPT-4o on Airline drops from 30.00% to 19.33%.
  • 3. An Empirical Study of Think on Response Scaling: Response scaling compares no, low, medium, and high think settings, using response length as a coarse proxy for user-visible communication.
  • 3.3. Case Study on Agent Output: In a retail case, explicit disclosure that only an entire pending order could be cancelled changed the user’s next decision, whereas enforced think kept relevant context private.

4. Counterfactual Intervention Pipeline of Information Disclosure

The pipeline makes natural-language information disclosure observable through evidence-grounded turn annotations, then injects missing decision-relevant content from matched successful trajectories into failed ones before critical edits.

  • Evidence-Grounded Semantic Probing: The study operationalizes disclosure with five binary categories: State Correction, Option, Constraint, Consequence, and Proactive Retrieval.These categories capture corrections, feasible choices, rules or eligibility conditions, downstream outcomes, and proactively retrieved environment information.
  • Evidence-Grounded Semantic Probing: Annotators label each agent turn only when an exact response substring directly supports the disclosure category.Internal reasoning traces and tool calls are masked, leaving the system prompt, user utterances, and visible responses for annotation.
  • Counterfactual Information Injection: A failed target trajectory is compared with a matched successful golden trajectory to identify information present in the golden response but absent from the target.The intervention targets the decision point before the first task-critical environment edit.
  • Counterfactual Information Injection: Missing disclosure is constructed from golden evidence spans and minimally rewritten into the target response to preserve natural dialogue.The rewritten response replaces the original target response before rollout with the same agent and user simulator.
  • Counterfactual Information Injection: The intervention evaluates changes in the first environment-edit action and whether the final trajectory matches the golden outcome.The design compares golden-information injection with controls that perturb text using think content or other generic changes.

5. Experiments

The experiments test how information disclosure affects user-service agents through counterfactual interventions and prompt-level guidance. They compare disclosure methods, identify commonly missing information, and evaluate whether disclosure-aware prompting improves task performance.

  • Experiment Setup: FECR measures correction of the first environment-edit action, while RR measures whether the final outcome becomes successful after intervention.FECR conditions on trajectories whose original first edit differs from the golden first edit; RR evaluates final outcome matching.
  • Experiment Setup: The study evaluates causal information injection across two models and two domains using counterfactual interventions on failed enforced-think trajectories.Each retained item pairs a failed enforced-think trajectory with a matched successful golden trajectory.
  • Turn-level Intervention: The intervention comparison tests +Verbosity, +Think Content, and +Golden Info, separating generic response expansion from private-trace and decision-relevant information injection.+Golden Info uses missing information extracted from a matched successful trajectory.
  • Turn-level Intervention: 11.74% micro-averaged FECR for +Golden Info exceeds 6.08% for +Verbosity and 6.50% for +Think Content.The same ordering holds under model-level and domain-level aggregation; final recovery is more downstream and noisier.
  • Information Disclosure Analysis: State Correction and Constraint are the most frequent missing disclosure categories, whereas Option is consistently least frequent.The aggregate pattern aligns with a case study in which an omitted cancellation constraint precedes an invalid user action.
  • Prompt-level Intervention: InfoDis prompting generally improves task performance and mitigates enforced-thinking degradation across models and domains.For enforced-think agents, GPT-4o micro pass^1 rises from 52.32% to 57.58%, while DeepSeek-V3.1 rises from 59.19% to 62.63%.
  • Prompt-level Intervention: The results position information disclosure as both a diagnostic lens and a practical intervention target for user-facing communication before user decisions.The prompt encourages disclosure of relevant states, constraints, options, and consequences.

6. Conclusion

The paper finds that internal reasoning alone does not reliably improve user-service task performance or user-facing communication. It frames timely disclosure of decision-relevant information as both an actionable failure mechanism and a design target.

  • Enforced internal thinking does not reliably improve task performance across models and user-service scenarios.
  • Increasing think effort does not necessarily produce richer user-facing responses.
  • Evidence-grounded probing and counterfactual intervention show that failed trajectories often lack decision-relevant information present in matched successful trajectories.
  • Injecting golden decision-relevant information most clearly improves First Edit Correction Rate, indicating that missing disclosure can causally affect immediate environment-edit behavior.
  • More modest gains in final Recovery Rate show that information disclosure explains an actionable subset of failures rather than all failures.
  • InfoDis-aware prompting mitigates enforced-thinking degradation, supporting information disclosure as a practical design target.

Appendix Contents

The appendices document experimental prompts, performance comparisons, probing and injection configurations, consistency checks, and a counterfactual-intervention case study.

  • Appendix A contains the prompts used in the experiments.
  • Appendix B reports performance comparisons under different think focus settings and user simulators.
  • Appendix C documents configurations for evidence-grounded probing and information injection.
  • Appendix D presents a consistency check of evidence-grounded probing.
  • Appendix E provides a case study of counterfactual intervention.

A. Prompts Used in Experiments

The appendix specifies prompts and annotation procedures for controlling think effort, probing five disclosure categories, and injecting missing information into agent responses.

  • A.1. Think Effort Control: Low Think prompts require concise, intuitive responses without step-by-step reasoning or detailed deliberation.
  • A.1. Think Effort Control: High Think prompts require extensive analytical reasoning, problem decomposition, pitfall evaluation, logic verification, and rigorous deliberation.
  • A.2. Evidence-grounded Information Probing: Evidence-grounded probing evaluates each disclosure category only with respect to the user's upcoming decision and excludes information already provided or inferable from context.
  • A.2. Evidence-grounded Information Probing: The output represents assistant turns, preserves null turns with null annotations, and excludes user turns.
  • A.2. Evidence-grounded Information Probing: The five disclosure categories are state correction, option, constraint, consequence, and proactive retrieval.
  • A.2. Evidence-grounded Information Probing: Annotators must quote an exact span before assigning present or absent labels, without fabricating, paraphrasing, or inferring unstated information.
  • A.3. Information Injection for Counterfactual Intervention: The golden-information injection prompt preserves the original response while naturally integrating only explicitly listed missing information.
  • A.3. Information Injection for Counterfactual Intervention: The think-content injection prompt asks the agent to weave relevant internal reasoning into the user-facing response.

B. Performance Comparison Under Different Think Focus and User Simulator

The appendix tests whether think-tool instability depends on user-simulator capability or think-tool focus. Results remain mixed across configurations, with no generally reliable recovery of the no-think baseline.

  • 56.16% to 51.31% in pass^1 and 43.23% to 39.39% in pass^2: default enforced thinking still reduces GPT-4o micro-average performance with a GPT-5 user simulator.
  • The mixed results suggest that unstable think-tool utility is not merely caused by an insufficiently capable user simulator and mainly arises from the agent side.
  • User-centric and action-centric think focuses help different models differently, so neither focus provides a consistent advantage.
  • With the GPT-5 user simulator, focused variants improve over the default think setting but neither consistently recovers the no-think baseline.
  • The table compares performance across different think-focus instructions and user-simulator configurations.

C. Configurations of Evidence-grounded Probing and Information Injection

Evidence-grounded probing and information injection use GPT-4.1 as an annotator with low temperature and safeguards against annotation-format errors.

  • GPT-4.1 annotates the evidence-grounded information probing and information injection stages.The annotator temperature is set to 0.2.
  • A rule-based format check is applied to prevent annotation format or syntax errors.
  • The annotator can retry at most two times when a format-check exception occurs.

D. Consistency Check of Evidence-Grounded Probing

The study repeats LLM-as-a-judge annotation with and without evidence and finds that evidence grounding substantially improves annotation self-consistency.

  • LLM-as-a-judge annotation is repeated twice for conditions without and with evidence.
  • 0.70∼0.94 to 0.86∼0.94: evidence-grounded criteria improve annotation self-consistency.The comparison is reported for the range of self-consistency results.
  • 0.7079 to 0.8681: Cohen’s 𝜅 on Constraint increases when evidence grounding is added.The passage presents this as an example of improved annotation stability.

E. Case Studies of Counterfactual Intervention

The case studies examine counterfactual interventions that modify failed-trajectory responses with golden information, thinking, or verbosity-related controls. They track where intervention occurs and how missing information is represented in the response.

  • Intervention setup: The counterfactual intervention cases compare +Golden Info, +Think Content, and +Verbosity.
  • Intervention setup: Each intervention links failed_index and golden_index within the same task_id and identifies the intervened turn with turn_id.
  • Intervention setup: The first_edit_turn records when the first environment edit occurs in the failed trajectory.
  • Case study: One enhanced response adds that both a return and an exchange cannot be processed for the same order simultaneously.It asks the user to choose which request to proceed with.
  • Case study: The intervention records missing information as SC and CN, corresponding to the same-order conflict and the one-request-at-a-time constraint.
Loading 2602.07796v2…