Source-linked AI summary
ReasoningLens: Hierarchical Visualization and Diagnostic Auditing for Large Reasoning Models
Jun Zhang, Jiasheng Zheng, Boxi Cao, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, Le Sun
TL;DR
Large reasoning models produce lengthy traces in which logical dependencies become difficult to inspect and verify. ReasoningLens addresses this with hierarchical visualization, agentic diagnosis, and systemic profiling, achieving consistent performance across visualization and error-identification tasks while providing auditable diagnostic insights.
Problem
Existing reasoning-trace visualization remains largely heuristic and descriptive, lacking a comprehensive taxonomy-driven framework for structural understanding and diagnostic analysis.
Method
ReasoningLens transforms long chain-of-thought traces into hierarchical reasoning graphs, applies multi-agent diagnosis, and profiles model-level reasoning behavior.
Results
ReasoningLens achieves consistent and reliable performance on hierarchical visualization and error-identification tasks while producing structural transparency and actionable diagnostic insights.
Takeaways & Limitations
ReasoningLens provides a systematic foundation for interpreting, auditing, and profiling long-form reasoning traces at instance and model levels.
Takeaways & Limitations
ReasoningLens primarily targets static chain-of-thought traces, while its deployment remains relatively monolithic and does not yet support dynamic multi-step agentic interactions.
Abstract
from arXiv · showhide
The emergence of Large Reasoning Models has introduced exceptionally long Chain-of-Thought traces, creating a transparency burden where critical logic is often buried under massive procedural text. To address this, we present ReasoningLens, an open-source framework designed for the hierarchical visualization and diagnostic auditing of complex reasoning chains. ReasoningLens addresses information necropsy by: (1) structuring traces into interactive hierarchies that separate high-level strategy from low-level execution; (2) leveraging an agentic auditor for automated error detection and tool-augmented verification; and (3) synthesizing systemic reasoning profiles to reveal model-specific blind spots. By transforming unstructured walls of text into actionable insights, ReasoningLens provides a modular foundation for interpreting, debugging, and optimizing the next generation of reasoning-centric AI.
1 Introduction
ReasoningLens addresses the opacity and verification burden of increasingly long reasoning traces by providing a systematic framework for hierarchical visualization, automated diagnosis, and model profiling. The paper introduces LensBench to evaluate structural visualization and error-annotation capabilities for long-form chain-of-thought reasoning.
- Long reasoning traces bury critical logical dependencies in an unstructured “wall of text,” reducing transparency and increasing burdens for inspection, diagnosis, and safety assurance.
- Prior reasoning-trace visualization methods are largely heuristic and descriptive, lacking a structured taxonomy and comprehensive diagnostic framework for deep-level analysis.
- ReasoningLens provides a principled visualization-to-diagnosis-to-profiling pipeline with multi-layered, interactive reasoning graphs guided by a taxonomy of reasoning behaviors.
- LensBench comprises 130 instances across 5 representative reasoning-failure categories, annotated with exploration-level hierarchy and fine-grained error types.
- Case studies show that ReasoningLens transforms monolithic chain-of-thought traces into an auditable logical framework providing structural transparency and actionable diagnostic insights.
- ReasoningLens is an open-source diagnostic toolkit enabling hierarchical visualization, automated diagnosis, and systematic profiling of large reasoning models.
2 Related Work
Prior work highlights the difficulty of manually inspecting long reasoning traces and develops visualization and structural analyses, but existing methods do not fully capture reasoning structures or localize errors. ReasoningLens addresses these gaps through hierarchical visualization, agentic diagnosis, and systemic profiling.
- Motivation: Long LRM reasoning traces hinder manual inspection and exhibit redundant traces, sensitive-content leakage, and logical inconsistencies.These failure modes motivate research into reasoning transparency, especially error analysis and visualization.
- Existing limitations: Visualization systems reduce cognitive load with graphical CoT interfaces, but rely on superficial text rather than underlying reasoning structures.This limitation constrains their ability to represent the organization of complex reasoning traces.
- Existing limitations: Structural approaches parse traces into step-level taxonomies, yet stop short of actionable error localization.They characterize reasoning behavior without directly identifying where errors occur or how to address them.
- ReasoningLens: ReasoningLens combines hierarchical visualization, agentic diagnosis, and systemic profiling to map reasoning nodes, detect flaws with fixes, and report model behavior patterns.The framework supports interactive CoT-to-node mapping, automated diagnosis of issues such as overthinking, and comprehensive behavioral reports.
3 System Design
ReasoningLens transforms raw chain-of-thought traces into structured, actionable insights through hierarchical visualization, agentic diagnosis, and systemic profiling. Its design supports both precise instance-level inspection and holistic model-level behavioral analysis.
- System Overview: ReasoningLens comprises Hierarchical Visualization, Agentic Diagnosis, and Systemic Profiling to convert raw CoT traces into actionable insights.These components support instance-level inspection and model-level behavioral profiling.
- Hierarchical Visualization: The visualization pipeline classifies reasoning into exploration-level strategies and exploitation-level execution units, then models them as hierarchical graphs.Exploration captures strategic operations such as decomposition, backtracking, and validation, while exploitation captures retrieval, procedural execution, and state assertion.
- Hierarchical Visualization: The framework extracts atomic planning units from unstructured CoT and organizes them into macro-level exploration graphs with local execution subgraphs for precise error attribution.Macro-nodes represent coherent strategic spans, while micro-nodes encode procedural dependencies within selected reasoning paths.
- Agentic Diagnosis: Agentic Diagnosis combines memory compression, external-tool verification, and suggestion modules to localize reasoning errors and support scalable, verifiable detection.The memory module preserves trace-level consistency while localizing local errors; verification resolves internal ambiguity through external tool invocations.
- Agentic Diagnosis: Actionable mitigation strategies are mapped to identified error types through a repository of training-free and post-training interventions.This alignment helps users identify and apply targeted responses to specific reasoning bottlenecks rather than generic advice.
- Systemic Profiling: Systemic Profiling aggregates trajectory sets into model-level behavioral representations using shared structural signals, semantic deduplication, and synthesized reasoning profiles.The profiles capture global invariants such as search topology and node-level error annotations while preserving distinct heuristic features.
4 Experiment
The experiment evaluates REASONINGLENS on LENSBENCH, a benchmark with gold annotations for hierarchical trace structuring and fine-grained reasoning errors in long chain-of-thought traces. Across five backbone evaluators, Agentic Diagnosis improves with model strength, while Hierarchical Visualization remains stable across model scales.
- Benchmark Construction: LENSBENCH provides gold annotations for trace structuring and fine-grained reasoning errors over long CoT traces.The benchmark is manually verified, yielding 130 examples with gold annotations for both evaluation dimensions.
- Benchmark Construction: Controlled, taxonomy-guided error injections create globally coherent failures in otherwise clean reasoning trajectories.GPT-5.4 selects insertion points and compatible failure types, then rewrites targeted spans; human review removes incoherent or ambiguous instances.
- Evaluation Protocol: REASONINGLENS is evaluated with five backbone models spanning multiple families and parameter scales, using NTA and GES for structuring and per-type plus micro-averaged Precision, Recall, and F1 for diagnosis.The evaluators are DeepSeek-V4-Pro, MiniMax-M2.7, Qwen3.5-27B, Gemma-4-26B-A4B, and Qwen3-32B.
- Agentic Diagnosis: Overall diagnostic F1 ranges from 66.3 on Qwen3-32B to 82.3 on DeepSeek-V4-Pro, indicating that Agentic Diagnosis scales with backbone model strength.Safety detection remains consistently high, whereas Knowledge Error and Logical Error reach lows of 39.0 and 34.6 on Qwen3-32B and recover on stronger models.
- Hierarchical Visualization: Hierarchical Visualization maintains average NTA of 75.0 and GES of 69.7 across model scales despite variation in diagnostic F1.Segmenting raw reasoning text into semantically coherent planning units converts long-context understanding into a locally scoped labeling task.
5 Case Analysis
ReasoningLens transforms monolithic chain-of-thought traces into auditable hierarchical structures with high alignment to human annotations. Its Agentic Diagnosis module exposes obscured reasoning vulnerabilities and links them to targeted remediation strategies, supporting interpretability and model alignment.
- Structural Transparency: ReasoningLens decomposes Qwen3-32B reasoning into semantically coherent functional blocks, including Strategy Shift and Verification, with high alignment to human-annotated ground truth.The framework structures reasoning trajectories into a multi-dimensional logical representation for auditing.
- Diagnostic Auditing: Agentic Diagnosis surfaces unsafe manipulation tactics and redundant overthinking that verbosity can obscure, then associates them with targeted remedies such as Early Stopping.The module provides a “How to fix” pathway connecting diagnosed vulnerabilities to remediation strategies.
- Diagnostic Auditing: ReasoningLens links latent reasoning analysis with model alignment, extending interpretability into actionable diagnostic insights.The framework closes the loop between analyzing hidden reasoning behavior and alignment-oriented intervention.
- Structural Transparency: Figure 3 validates structuring accuracy by comparing human-annotated chain-of-thought with ReasoningLens representations, marking errors in red and hierarchy cues in blue.The visualization distinguishes reasoning errors from structural signals used to construct the hierarchy.
6 Conclusion
ReasoningLens reframes Long CoT analysis as active, structured interpretation through hierarchical reasoning graphs and multigranular trajectory views. Its diagnostic and profiling mechanisms localize errors, provide actionable feedback, and uncover model-level reasoning bottlenecks.
- 6 Conclusion: ReasoningLens reframes Long CoT analysis through hierarchical reasoning graphs that provide multigranular views of reasoning trajectories.This shifts analysis from passive observation to active, structured interpretation.
- 6 Conclusion: Its multi-agent diagnostic system integrates memory, verification, and suggestion modules to enable precise error localization and actionable feedback.
- 6 Conclusion: ReasoningLens’s systemic profiling mechanism uncovers model-level reasoning bottlenecks.
Limitations
ReasoningLens currently focuses on static chain-of-thought traces rather than dynamic, multi-step agentic interactions, while future development targets agentic trajectory analysis and modular plugin integration.
- Current scope: ReasoningLens primarily analyzes static chain-of-thought traces, not dynamic, multi-step agentic interactions.The framework is positioned as a pipeline from visualization through diagnosis to profiling for long-form reasoning.
- Future extensions: Future work will capture the Plan-Act-Observe cycle to better model interactive reasoning.This extension is intended to support agentic trajectory analysis.
- Future extensions: The framework is planned for integration into a modular plugin ecosystem supporting lightweight deployment for process-based training supervision.The passage describes this as a subsequent development direction.
A Evaluation Metrics
The evaluation uses Node Type Accuracy to score section-span node matching and Graph Edit Similarity to compare exploration-level reasoning graphs. Both metrics are computed per example and averaged across examples.
- Node Type Accuracy (NTA): Node Type Accuracy matches predicted nodes to gold nodes by section span using greedy highest-IoU matching with an IoU threshold of 0.5.Gold nodes are processed in order, and each is matched only to an unmatched predicted node with IoU(g, p) ≥0.5.
- Node Type Accuracy (NTA): The resulting Node Type Accuracy score is defined as 0 when no nodes are matched and is averaged over all examples.The matched set is denoted M.
- Graph Edit Similarity (GES): Graph Edit Similarity compares gold and predicted exploration-level reasoning graphs while retaining only node and edge functional types.Graph edit distance uses unit insertion/deletion costs and substitution cost 1 for type mismatches.
- Graph Edit Similarity (GES): If graph distance is unavailable within a fixed timeout, it is set to Z, and the resulting per-example similarity is averaged over all examples.Z = |V_G| + |V_ˆG| + |E_G| + |E_ˆG|.
B Human Verification Protocol
LENSBENCH instances underwent manual review of reasoning-error and trace-structure annotations, producing 130 verified examples after exclusions. Acceptance required coherent, unambiguous error labels and valid, functionally typed trace structures.
- Verification Protocol: 130 verified examples remained after manual review of both reasoning-error and trace-structure annotations, with any instance failing a criterion discarded.The review occurred before inclusion in LENSBENCH.
- Reasoning Error Annotation: Reasoning-error spans had to preserve local coherence, notation, and referents without exposing surface cues, while conflicting spans were excluded.The injected error had to remain identifiable from the trace content alone.
- Reasoning Error Annotation: Each error had to satisfy exactly one category among Safety, Logical Error, Knowledge Error, Formal Error, and Overthinking.Ambiguous cases were discarded, and multi-error compositions had to be informative and non-redundant.
- Trace Structuring Annotation: Trace-structure labels required non-overlapping node spans, consistent section assignments, exactly one final-answer node, and a plausible progression.Structural violations led to exclusion.
- Trace Structuring Annotation: Every node required a defensible functional type covering problem decomposition, reasoning step, intermediate answer, check, and final answer.The supplied passage states this taxonomy requirement but ends before the full criterion is shown.
C LensBench Case Study
The LensBench case study presents a complete annotation for ID 30, covering the data schema, normalized section texts, detected or injected errors, and a structured reasoning graph.
- LensBench annotation: The annotation example illustrates how original section texts are paired with injected error labels.This pairing is presented as part of the LensBench data schema.
- LensBench annotation: ID 30’s complete LensBench annotation includes the data schema, original section texts, detected errors, and a structured reasoning graph.The figure specifies ASCII-only normalization for the section texts.
- LensBench annotation: The case annotation connects normalized text, error detection, and structured reasoning representation within one LensBench example.Figure 4 presents these components together for ID 30.