Visible Reasoning: User-Facing Decision Transparency for Generative AI Systems

1 Introduction

• Users may be unable to validate generative-AI outputs or determine whether failures reflect missing context, irrelevant evidence, tool errors, or flawed reasoning.For tool-using agents, users also need to know which external sources shaped a recommendation and how much.Source: §1 · p. 2

1.1 Contributions

• The paper contributes Filtered Decision Logs, trace-based provenance reconstruction, deterministic influence ranking, and a multi-domain prototype evaluated with 50 participants.The ranking combines four semantic and structural signals and is also tested with counterfactual tool removal.Source: §1.1 · p. 3

2 Related Work

• The paper focuses on runtime evidence influence in tool-using agent traces, distinct from training-data influence, rationale methods, and provenance tooling.Its target is evidence from the current run, surfaced to end users.Source: §2 · p. 3

2.1 Training-Data Influence and Data Valuation

• Runtime influence scoring identifies which retrieved documents, API responses, or database rows shaped the current answer, unlike training-data attribution methods.The paper presents the approaches as complementary rather than substitutes.Source: §2.1 · p. 3

2.2 Rationales, Interpretability, and Chain-of-Thought

• Chain-of-thought traces may not faithfully represent actual decisions; models can give coherent reasoning for a correct answer reached for the wrong reasons.The paper therefore distinguishes self-narrated reasoning from auditable evidence.Source: §2.2 · p. 4

2.3 Feature Attribution and Explainability

• Unlike feature-attribution methods such as LIME and SHAP, FDLs rank tool-level evidence rather than tokens or individual features.The paper argues that tool-level evidence is more interpretable to nontechnical users.Source: §2.3 · p. 4

2.4 Provenance, Observability, and Evaluation

• The paper complements tracing and evaluation frameworks by converting connected execution traces into influence-ordered evidence for end users.This targets a user-facing transparency artifact rather than developer-facing trace inspection.Source: §2.4 · p. 4

2.5 Why Not Ask the Model?

• Rather than ask the model to report its own evidence, FDLs keep scoring separate from reasoning, allowing users to assess a score independently of the recommendation.The paper frames this separation as evaluation independence.Source: §2.5 · pp. 4–5

Definition

• A Filtered Decision Log accompanies a response and answers what the system concluded, why, and which evidence most shaped that conclusion.It is presented as a transparency artifact, not as raw intermediate reasoning.Source: §3 · p. 5

Components

• FDLs pair an evidence-anchored rationale with readable supporting data points from tool outputs and categorical relevance indicators.The indicators are High at ≥0.70, Medium from 0.40–0.70, and Low below 0.40.Source: §3 · p. 5

Interaction pattern: Accept or Re-Run

• Users can expand signal breakdowns, exclude a potentially misleading source, and rerun the agent to receive a new recommendation and influence scores.The career-advisor panel ranks social-media-profile at 0.74 and job-market-data at 0.37.Source: §3 · p. 5, Figure 1 · p. 6

Score interpretation

• Influence scores measure absolute alignment and proximity, not a source’s causal share of the answer, and they do not sum to 1.0 or 100%.A score of 0.70 denotes high semantic alignment with the answer and reasoning chain, not 70% contribution.Source: §3 · p. 5

4 System Architecture

• The agent and FDL pipelines are independent: the agent is unaware of evaluation, and FDL processing does not influence its reasoning.This separation is the architecture’s basis for evaluation independence.Source: §4 · p. 6

4.1 Agent Pipeline

• The agent calls external tools, reasons over their results across iterations, and produces an answer while its execution is recorded as a hierarchical context tree.The prototype uses a ReAct-pattern tool-augmented LLM agent.Source: §4.1 · p. 6

4.2 FDL Pipeline

• The separate seven-stage FDL pipeline processes the context tree using embeddings and deterministic computation, without generative AI.Its stages extract tool results and ancestors, score and rank evidence, then format the FDL response.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Provenance Reconstruction

• Provenance reconstruction walks the execution tree to find tool-result data nodes and can fall back to the global conversation messages array when expected nodes are missing.The fallback is intended to support different agent trace structures.Source: §4.3 · pp. 7–8

5 Deterministic Influence Scoring

• The four signals use cosine similarity between embedding vectors, making influence scores reproducible for the same inputs.The paper contrasts this with LLM-as-evaluator approaches, which can produce different results when the same evaluation is rerun.Source: §5 · p. 8

5.1 Signal Definitions

• FA measures tool-result overlap with the final answer, AVG averages similarity across reasoning steps, PERSIST measures thresholded breadth, and DEPTH reflects reasoning-ancestor proximity.DEPTH counts only LLM reasoning ancestors; a direct tool result with none receives DEPTH = 1.0.Source: §5.1 · pp. 8–9

5.2 Composite Score

• The composite score weights FA, AVG, PERSIST, and DEPTH at 0.40, 0.30, 0.20, and 0.10 by default, respectively.The weights are configurable and encode a prior that favors direct answer similarity, then reasoning consistency, breadth of reference, and structural proximity.Source: §5.2 · p. 9
• Each evidence item is scored independently: scores neither sum to 1.0 nor represent causal attribution.A score of 0.70 indicates high semantic alignment with the answer, not that the source contributed 70% of it.Source: §5.2 · p. 9

5.3 Adaptive Weight Redistribution

• Evidence without LLM ancestors would otherwise have scores capped near 0.50, so adaptive redistribution reallocates the weights to FA and DEPTH.With default weights, the adapted values are α′ = 0.80 and δ′ = 0.20; the FA-to-DEPTH ratio remains 4:1.Source: §5.3 · pp. 9–10

6 Multi-Domain Demonstration

• The demonstrations used deterministic mock tools returning structured JSON, with each scenario designed around conflicting signals across data sources.The scenarios covered career advising, investment advising, and travel planning.Source: §6 · p. 10

6.1 Scenario 1: Career Advisor (5 tools) — Hero Scenario

• In the career scenario, social-media-profile scored 0.74 (High); excluding it shifted the recommendation from “Policy Analyst” to “Data Scientist.”Project-portfolio rose from 0.66 to the top position, while academic-records rose from 0.60.Source: §6.1 · p. 11, Figure 4 · p. 11

6.2 Scenario 2: Investment Advisor (4 tools)

• For investment advice, company-financials ranked highest at 0.70 (High), followed by economic-indicators (0.68), stock-price-detail (0.67), and financial-news (0.66).The ranking indicated that fundamentals and macro context, rather than news sentiment, shaped the recommendation.Source: §6.2 · p. 11, Figure 5 · p. 12

6.3 Scenario 3: Travel Planner (4 tools)

• The travel-planning scenario used flight, hotel, weather, and local-event tools to frame a tension between budget-conscious and experience-optimized recommendations.The itinerary was for five days in Tokyo in late March.Source: §6.3 · p. 11

6.4 Cross-Domain Observations

• Across all three scenarios, the pipeline ranked the most decision-relevant source highest and surfaced the designed tensions.Removing the top-ranked tool changed answers more than removing the bottom-ranked tool in all three scenarios.Source: §6.4 · p. 12, §6.5 · pp. 12–13, Table 3 · p. 13

7.1 Participants

• The 50 participants comprised 25 working professionals with chatbot or conversational-AI development experience and 25 computer-science or related-field students with AI/ML coursework.The students were graduate or advanced undergraduate students.Source: §7.1 · p. 13

7.2 Procedure

• Participants inspected rankings across all three scenarios, identified the source they thought most influential, could exclude sources and re-run, then gave one overall thumbs-up or thumbs-down rating.The single-item rating combined perceived understandability and trustworthiness and was collected after all scenarios.Source: §7.2 · p. 13

7.3 Results

• 46 of 50 participants (92%) rated the influence-ranked evidence as helpful for understanding and trusting recommendations.All 25 working professionals responded positively, compared with 21 of 25 students; four students cited difficulty interpreting AVG and PERSIST.Source: §7.3 · p. 14, Table 4 · p. 14

7.4 Qualitative Observations

• Participants consistently began with the top-ranked source, and reported that ordering spared them from mentally reconstructing which sources the response used.Participants described that reconstruction as cognitively expensive.Source: §7.4 · p. 14
• Several participants spontaneously excluded the top source to see what changed; in the career scenario, those observing a shift from policy to quantitative careers reported the highest confidence in understanding the system’s reasoning.The exclude-and-re-run interaction did not require instruction.Source: §7.4 · p. 14
• Working professionals who expanded the signal breakdown reported higher confidence in the influence score, particularly when Answer Relevance was high and tool data appeared in the answer.All working professionals gave positive ratings.Source: §7.4 · p. 14

8.1 What We Claim—and What We Do Not

• The authors do not claim optimal scoring: weights are configurable priors, and different domains may benefit from different configurations.Counterfactual validation and formative evaluation are presented as evidence that rankings track decision dependence and that users find the transparency useful.Source: §8.1 · pp. 14–15
• Sorted evidence helps users identify which sources drove a recommendation, while exclusion and re-runs let them compare revised recommendations and make active decisions.The authors also argue that a separate deterministic scoring pipeline can support trust in scores even when users question a recommendation.Source: §8.1 · p. 15

8.2 Evidence-Level Feedback as a Calibration Signal

• Users’ source exclusions and responses to revised recommendations could provide feedback for calibrating ranking weights or learning domain-specific profiles, but no such model was trained.The authors say the interaction produces data needed for this future calibration.Source: §8.2 · p. 15

8.3 Inspiring Further Research

• Future research questions include learning weights, causal attribution, influence across agents and time, and domain-specific weighting.The authors also ask whether expert input could inform domain-specific profiles.Source: §8.3 · p. 15

8.4 Limitations

• Embedding similarity indicates semantic overlap, not causal influence; the evaluation also uses synthetic mock tools, one model, formative feedback, and intuition-chosen default weights.Different models may use tools differently, real tools may affect score stability, and the formative study lacks controlled significance testing.Source: §8.4 · p. 16

• The scenarios use synthetic mock data, so they are not real career, financial, or travel advice; FDLs help inspect evidence, not validate truth.Production evidence exposure must respect access controls and redaction policies, and financial-advice use requires appropriate regulatory compliance and human oversight.Source: §9 · p. 16

• The TypeScript prototype uses a Fastify backend with a Claude 3 Sonnet agent on Amazon Bedrock and three scenario endpoints supporting re-asks.The endpoints support ignoredTools and systemPromptOverride parameters.Source: §10 · p. 16
• The FDL pipeline uses TreeOfFunctions for seven flowchart-based stages, with typed scope carrying data between stages and a POST /api/explain endpoint.The implementation also uses Amazon Titan Embed Text v2 with batch processing and a stage-ID-keyed cache.Source: §10 · p. 16
• The React interface splits chat and influence panels, presenting evidence cards with signal breakdowns and exclude/re-ask controls.The prototype includes more than 75 tests, including property-based tests for scoring invariants, pipeline correctness, and response-format stability.Source: §10 · p. 16

• Future evaluation should compare FDL and standard interfaces, test rankings against expert judgments, vary scoring signals, use live APIs, and assess multiple LLM providers.The proposed user study would measure trust, understanding, and actionability with statistical significance testing.Source: §11 · p. 17

12 Conclusion

• As an external measurement pipeline separate from the agent’s reasoning, FDLs make decision formation visible through deterministic, verifiable transparency.The transparency can be trusted independently of the recommendation itself.Source: §12 · p. 17
• Across career advising, investment analysis, and travel planning, counterfactual tool-removal experiments showed that influence ranking tracks decision dependence, while 92% of 50 participants rated the mechanism useful.The scenarios paired sorted evidence with accept/re-run interaction.Source: §12 · p. 17
• The paper identifies the interaction pattern—not the specific scoring algorithm—as its key contribution: sorted tool influence with the ability to exclude sources and re-run.The authors hope this inspires further research into influence scoring for tool-augmented agents.Source: §12 · p. 17