Source-linked AI summary
T2S-Bench & Structure-of-Thought: Benchmarking and Prompting Comprehensive Text-to-Structure Reasoning
Qinsi Wang, Hancheng Ye, Jinhee Kim, Jinghan Ke, Yifei Wang, Martin Kuo, Zishan Shao, Dongting Li, Yueqian Lin, Ting Jiang, Chiyue Wei, Qi Qian, Wei Wen, Helen Li, Yiran Chen
TL;DR
Complex text processing remains difficult because models often lack stable intermediate representations. The paper introduces Structure of Thought and T2S-Bench, finding broad downstream benefits from explicit structuring while identifying substantial remaining capability gaps.
Problem
Models struggle with complex long-context text processing and often lack stable intermediate representations for retrieval and generation.
Method
The paper introduces SoT, which structures text into key nodes and links before answering, and T2S-Bench, a scientific-literature benchmark for text-to-structure capability.
Results
SoT consistently improves performance across eight tasks and three model families, while T2S-Bench evaluation reveals substantial improvement potential.
Takeaways & Limitations
The results support explicit structuring as a fundamental competence for reliable text understanding and show that enhanced structuring transfers to downstream tasks.
Takeaways & Limitations
Stronger text-to-structure capability may enable extraction from sensitive documents, misleading structured reports, and additional computational and environmental costs.
Abstract
from arXiv · showhide
Think about how human handles complex reading tasks: marking key points, inferring their relationships, and structuring information to guide understanding and responses. Likewise, can a large language model benefit from text structure to enhance text-processing performance? To explore it, in this work, we first introduce Structure of Thought (SoT), a prompting technique that explicitly guides models to construct intermediate text structures, consistently boosting performance across eight tasks and three model families. Building upon this insight, we present T2S-Bench, the first benchmark designed to evaluate and improve text-to-structure capabilities of models. T2S-Bench includes 1.8K samples across 6 scientific domains and 32 structural types, rigorously constructed to ensure accuracy, fairness, and quality. Evaluation on 45 mainstream models reveals substantial improvement potential: the average accuracy on the multi-hop reasoning task is only 52.1%, and even the most advanced model achieves 58.1% node accuracy in end-to-end extraction. Furthermore, on Qwen2.5-7B-Instruct, SoT alone yields an average +5.7% improvement across eight diverse text-processing tasks, and fine-tuning on T2S-Bench further increases this gain to +8.6%. These results highlight the value of explicit text structuring and the complementary contributions of SoT and T2S-Bench. Dataset and eval code have been released at https://t2s-bench.github.io/T2S-Bench-Page/.
1. Introduction
The paper addresses weaknesses in long-context text processing by introducing Structure of Thought and T2S-Bench, showing that explicit text structuring improves downstream performance while substantial capability gaps remain.
- Motivation: Current models struggle with complex long-context text processing and lack stable intermediate representations for retrieval and generation.Even state-of-the-art models reach only around 60% on LongBench.
- Structure of Thought: Structure of Thought (SoT) prompts models to organize text into key nodes and links before producing final answers.The strategy is intended to provide text structure as an intermediate representation.
- Structure of Thought: SoT consistently improves performance across eight text-processing tasks and three models.The authors describe the gains as significant and broadly applicable across model families and task types.
- T2S-Bench: T2S-Bench provides 1.8k high-quality samples spanning six scientific domains and 32 structural types for evaluating and improving text structuring.Its components include training, multi-hop reasoning, and end-to-end structuring sets with accuracy and fairness-oriented construction.
- Evaluation: 52.1% average exact match is achieved on T2S-Bench-MR, while Gemini2.5-Pro reaches 58.1% node accuracy on end-to-end structuring.These results indicate substantial remaining room for improvement in text-to-structure capability.
- Evaluation: Up to 8.5% average improvement across eight downstream text-processing tasks follows fine-tuning on T2S-Train-1.2k.The paper reports improved robustness and accuracy in downstream general text workflows.
2. Motivation & Challenges
The paper motivates explicit text structuring as a concrete alternative to unconstrained end-to-end generation, then introduces SoT and a benchmark designed around the challenges of constructing and evaluating text structures.
- Motivation: The paper asks whether explicit text structuring can improve general text-processing ability and by how much.It investigates this question through SoT evaluation and text-to-structure dataset construction.
- Structure of Thought: SoT instructs models to identify key nodes and links, provide their structure, and then answer using both the text and structure.Its expected output separates a JSON-like structure from the final answer.
- Empirical Findings: SoT improves performance by over 5% across eight tasks and by over 10% on 2WikiMultiHopQA and MuSiQue.The reported gains support explicit structuring for general text-processing tasks.
- Empirical Findings: SoT is reported as more consistently effective than Chain-of-Thought for general text processing because it anchors reasoning to an explicit structure.The paper contrasts this with potential noise from CoT in general text tasks.
- Empirical Findings: The benefits of SoT span model families and task types, supporting text structure as a universal intermediate representation.The paper characterizes the benefit as model- and task-agnostic.
- Dataset Challenges: Text-structuring datasets face difficult correctness verification, complex evaluation, and one-to-many mappings between texts and valid structures.These issues make construction expensive and make single-reference scoring difficult.
- T2S-Bench: T2S-Bench addresses these challenges through a construction pipeline and comprehensive evaluation of text-structuring capability.The dataset is presented as a source of practical guidance for future text-to-structure benchmarks.
3. Construction Process of T2S-Bench
T2S-Bench is constructed from academic papers and structural diagrams through automated search, model checks, and expert filtering. The resulting dataset spans diverse scientific domains and structure types while targeting high-quality, validated text–structure pairs.
- Data source: T2S-Bench uses academic papers and their structural diagrams as primary sources for text–structure pairs.The source material is intended to provide structurally accurate diagrams and coherent corresponding text.
- Coverage: The dataset covers 6 scientific disciplines, 17 sub-disciplines, and 32 structural types.These types represent commonly used diagrams within specific sub-disciplines.
- Automated construction: A four-module automated pipeline searches for papers, downloads PDFs, crops figures, and validates candidate paper–structure pairs.The provided passage describes paper search and figure processing; failures trigger a restart from the search stage.
- Automated construction: 1,521 qualified text–structure pairs were obtained after approximately five searches per accepted sample.The construction targeted 50 high-quality samples per structural type.
- Quality control: Human filtering addresses noise in diverse academic diagrams before samples are labeled high-quality.Experts assess structural completeness, noise, node and link counts, and structural singularity.
2. Functional Mapping
Functional Mapping organizes questions around identifying components, roles, and relationships within a text-derived structure.
- Single choice: Single-choice Functional Mapping includes router or selector, aggregator or fusion, buffering or storage, and mediator-versus-direct-cause identification.These categories focus on selecting one structural role or relationship.
- Multiple choice: Multiple-choice Functional Mapping includes controller or tuner identification and parallel division-of-labor mapping.These categories require identifying multiple applicable structural elements.
- Mixed: Mixed Functional Mapping includes measurement or observation node identification.The taxonomy labels this category as mixed single-plus-multiple choice.
3. Boundary Testing
Boundary Testing examines how structural components behave under conditions, including activation, applicability, robustness, redundancy, and bypass scenarios.
- Single choice: Single-choice Boundary Testing covers conditional edge activation, narrow applicability, boundary robustness, and bypass leakage.These templates test structural behavior under specific constraints or edge conditions.
- Multiple choice: Multiple-choice Boundary Testing includes redundant-path checks and invariance checks.These questions assess whether multiple paths or changes affect the structure as expected.
- Mixed: The Boundary Testing taxonomy contains no mixed single-plus-multiple-choice category.The taxonomy explicitly marks the mixed category as unavailable.
4. Counterfactual Reasoning
T2S-Bench combines multi-hop reasoning and end-to-end structuring evaluations with quality-controlled construction procedures. Its evaluation separates node and link extraction to support standardized comparisons across models.
- Counterfactual Reasoning: Counterfactual Reasoning includes edge removal, module replacement, feedback disabling, upstream-source changes, shortcut edges, and multi-point interventions.The taxonomy also includes condition flipping and distinguishes single, multiple, and mixed formats.
- Multi-hop reasoning: Multi-hop questions are designed to require text structuring, depend on reference diagrams, and remain answerable from text alone.Questions use multiple-choice formats grounded in paper reference diagrams.
- Multi-hop reasoning: Four question categories each receive eight templates, making structure-aware interpretation necessary for solving the questions.The categories are Fault Localization, Functional Mapping, Boundary Testing, and Counterfactual Reasoning.
- Quality control: Generated questions undergo template compliance, diagram-based correctness checks, and text-dependency verification before repeated generation or inclusion.GPT-o3 generates questions, while GPT-5.2 and Gemini-2.5-Pro perform verification.
- Quality control: Approximately 1.7k high-quality text-structure-question samples remained after review by 15 PhD-level experts.Reviewers assessed correctness, difficulty, coherence, and format.
- Multi-hop reasoning: The multi-hop dataset uses a stratified 7:3 domain split, yielding T2S-Train-1.2k and a 500-sample test set evaluated with EM and F1.Models receive only texts and questions during evaluation.
- End-to-end structuring: End-to-end construction retains key nodes and links, controls graph complexity, and partially constrains generation to improve comparability.The E2E set contains 87 finalized text-key-structure pairs after model and human checks.
- End-to-end structuring: E2E evaluation scores links with F1 and nodes with semantic similarity, separately isolating the two extraction abilities.Models predict links when nodes are provided and predict nodes when links are provided.
4. Evaluation
Evaluation shows substantial variation across models and tasks, with multi-hop reasoning and end-to-end node extraction remaining difficult. Structuring skills also correlate with and improve downstream long-context performance, while domain and graph complexity affect robustness.
- Overall Results: 81.40% EM and 91.56% F1 make Gemini-2.5-Pro the strongest overall model, ahead of Claude-sonnet and GPT-5.2.
- Task Breakdown: Counterfactual reasoning and boundary testing exceed 80% EM for top systems, while fault localization trails by 15%–20%.Functional mapping is intermediate, but weaker models answer fewer than 10% correctly.
- Structure Extraction: Node similarity rarely exceeds 60% in end-to-end extraction, and node identification is harder than linking once nodes are found.The node score constrains the potential link score.
- Domain Performance: Proprietary models show more balanced domain performance, whereas open-source models fluctuate more across scientific domains.Kimi-K2 excels in environmental science but underperforms in physics, while MiniMax-Text-01 struggles across domains.
- Downstream Transfer: Fine-tuning on T2S-Train-1.2k improves both in-domain and out-of-domain results, including Qwen2.5-7B EM gains from 60.0% to 68.2% on HotpotQA.SoT outperforms CoT, and learned structuring skills generalise to long-context tasks.
- Correlation Analysis: T2S-Bench-MR performance has a positive correlation with LongBench scores across models.Gemini-2.5-Pro, Claude sonnet, and DeepSeek-R1 occupy the upper-right region, while weaker models cluster lower-left.
- Graph Complexity: As reference graphs grow from 1–5 to 14–20 nodes, Link F1 declines sharply, with smaller models approaching zero beyond 14 nodes.DeepSeek-R1 and Qwen3-235B remain robust up to 10–14 nodes before degrading.
5. Conclusion
The paper introduces T2S-Bench as a comprehensive benchmark for text-to-structure capabilities and evaluates it through multi-hop reasoning and end-to-end extraction. Results show significant improvement potential, while fine-tuning transfers enhanced structuring skills to downstream tasks.
- 5. Conclusion: T2S-Bench evaluates text-to-structure capabilities across six scientific domains and 32 structural types using multi-hop and end-to-end evaluations.
- 5. Conclusion: Benchmarking 45 mainstream models reveals significant improvement potential, particularly for node extraction.
- 5. Conclusion: Fine-tuning experiments show that enhanced structuring skills transfer to downstream tasks.
Impact Statement
The work positions structured text processing as relevant to document-centric workflows and auditability, while acknowledging dual-use risks and recommending safeguards. Its broader context includes Find–Fuse–Form tasks and prior structure-based methods with limited coverage.
- Potential positive impacts: Intermediate structures may support literature review, evidence-grounded question answering, and structured report generation.
- Potential negative impacts and dual use: Stronger text-to-structure capability may enable sensitive-document extraction, surveillance, profiling, targeted manipulation, and misleading structured reports.
- Mitigations and responsible use: Responsible deployment should use privacy safeguards for private corpora and human verification in high-stakes settings.
- Text-processing benchmarks: Text-processing workflows can be organized as Find, Fuse, and Form: locating information, integrating viewpoints or facts, and generating structured outputs.
- Information structuring: Prior structure-based methods showed task-specific benefits, but inconsistent definitions and protocols left a gap for broader evaluation.
B.1. Fault Localization Task Examples
The fault-localization examples test whether models can trace failures through directed structures, identify bottlenecks, and distinguish affected branches from independent or compensating paths.
- Task framing: Fault localization identifies minimal text spans or upstream components responsible for errors, inconsistencies, or wrong outputs.The task emphasizes fine-grained evidence attribution and distinguishing true evidence from distractors.
- Propagation reasoning: Downstream propagation ordering requires counting directed hops to identify the node affected at a specified position after an upstream change.Removing Non-adherence makes prolonged hyperglycemia the initial effect and beta islet failure the second downstream element.
- Path interruption: Minimal cut set identification selects the smallest node sets that block every directed path from a source to a target.The examples contrast sets that cover independent branches with alternatives that leave shortcut paths intact.
- Bottleneck reasoning: Mandatory bottleneck identification finds a component shared by every path between request sources and a target.The master-region leader cache is required for both local and remote writes before database commitment.
- Compensation and independence: The examples also test partial failure with compensation, where one disrupted branch can coexist with an unaffected parallel path.A failure on the DSO-to-EVSE-owner control path does not affect the independent compensation-payment path.
- Additional fault patterns: Other examples require identifying shared upstream causes, feedback dependencies, or structural consequences when a component fails.The cited cases include coverage feedback, shared climate drivers, receptor-mediated inhibition, and parallel cardiovascular regulation.
C. Dataset Curation
The curation pipeline sources scientifically grounded diagrams, normalizes and validates their metadata, extracts node-link graphs, and generates topology-based questions with layered quality checks.
- Curation pipeline: The dataset builder uses four sequential stages: Candidate Search, Schema Normalization, Caption Consistency, and Graph Extraction.These stages identify papers and figures, enforce JSON schemas, verify figure metadata, and parse diagrams into node-link graphs.
- Source and figure selection: Candidate papers must match the requested topology, use an exact verified figure identifier, and provide a connected graph with readable labeled nodes and explicit links.Plots, tables, photos, disconnected diagrams, and semantically mismatched figures are rejected.
- Validation: Caption consistency checking compares a web snippet with PDF caption text and returns a mismatch only when the references are clearly unrelated.The checker is instructed not to penalize incompleteness and to accept uncertain matches.
- Graph extraction: Feasible diagrams are converted into graphs with uniquely identified labeled nodes and directed source-target links, while decorative noise and unsuitable figures are excluded.Bidirectional relations are emitted as two directed links.
- Question generation and QA: Question generation instantiates one reasoning template per sample and uses diagram consistency and text-only solvability checks for quality control.Questions are multiple-choice, topology-grounded, non-trivial, and require at least two reasoning steps.
- Question type templates: The benchmark includes specialized templates such as bottleneck identification, feedback-loop failure amplification, mediator selection, and threshold or saturation points.Each template defines a target structural relation and topology-based distractors.
D. Model Evaluation Details
Evaluation uses separate multiple-choice and structure-extraction tracks with strict schemas, deterministic decoding, and consistent scoring across API and open-source models.
- Evaluation tracks: The benchmark evaluates multiple-choice question answering and structure extraction as separate tracks.Structure extraction uses node labeling followed by link extraction.
- Question answering: Multiple-choice prompts concatenate the extracted text paragraph, answer options, and a strict final-answer format.The evaluator expects an answer string using fixed single-select or multi-select syntax.
- Structure extraction: Structure extraction prompts separately request JSON node labels and JSON links based on the text and graph inputs.The node-labeling and link-extraction prompts use distinct output markers and schemas.
- Decoding: All evaluation paths use temperature-zero decoding and task-specific generation limits to improve determinism.Longer limits are used for structure JSON and shorter limits for multiple-choice answers.
- Execution infrastructure: API evaluation supports compatible gateways, retries, exponential backoff, streaming, configurable timeouts, and provider-specific fallback handling.Open-source endpoint evaluation also supports resumable per-model, per-sample caches.
- Cross-track consistency: API and open-source tracks retain identical inputs, formatting, answer schemas, and EM/F1 computation so score differences primarily reflect model capability.This consistency is intended to reduce evaluator artifacts.
E.2. Full sample examples
The full sample format combines extracted paper text, a ground-truth topology, and a multi-hop question, while the benchmark spans diverse scientific domains and model families.
- Sample format: Each dataset instance contains an extracted text paragraph, a ground-truth node-link graph, and a multi-hop reasoning question with analysis and distractors.The sample format exposes the structural evidence and reasoning plan used for evaluation.
- Domain and topology coverage: The benchmark visualizes representative topologies from Computer Science, Economics, Environmental Sciences, Life Sciences, Physical Sciences, and Social Sciences.Examples include sequential flows, feedback loops, and parallel hierarchies.
- Computer Science example: The MemGPT example asks for the mediator between Archival Storage and Working Context.The Function Executor forms the two-hop path Archival Storage → Function Executor → Working Context, with no direct edge between the endpoints.
- Dataset scale and metrics: T2S-Bench uses a training set of 1.2k instruction-answer pairs, a 500-question multi-hop test set, and an 88-sample end-to-end extraction test set.Metrics include EM and F1 for multiple-choice tasks and NodeF1/LinkF1 for end-to-end extraction.
- Model coverage: The evaluation covers more than forty proprietary and open-source models from multiple families, sizes, and instruction variants.The listed families include Gemini, GPT, Claude, DeepSeek, Qwen, GLM, MiniMax, Mistral, and LLaMA.
- Training setup: Qwen2.5-7B-Instruct and LLaMA3.1-8B-Instruct were trained with GRPO for approximately 200 steps on a single node with eight A100 GPUs.The training setup used batch size 32 and the veRL library.
F.3. Observation and Insight
The observations link structural extraction and reasoning performance while revealing persistent bottlenecks, scaling limits, uneven reasoning skills, and specialization–generality trade-offs.
- Models with stronger node and link extraction also achieve higher overall question-answering accuracy.Gemini-2.5-Pro, Claude-sonnet, and GPT-5.2 combine top reasoning accuracy with NodeF1 above 50 and LinkF1 above 77.
- Open-source models are narrowing the performance gap through instruction tuning and publicly available training resources.DeepSeek-reasoner (R1) and Qwen3-32B exceed 69 EM and 83 F1 in reported categories, while some mid-sized models surpass 65 EM.
- Node extraction remains the main structural bottleneck: strong models approach 90 F1 on QA while NodeF1 stays in the mid-50s or below 40 for many open-source systems.LinkF1 often exceeds 80, but weaker node identification limits end-to-end structure utility.
- Larger parameter counts do not consistently produce better performance, as shown by substantial within-family reversals.The LLaMA 70B instruct model scores 74.90 F1 versus 60.85 for the 405B variant, while Qwen3-Next comparisons also favor a smaller model on several categories.
- Reasoning abilities are uneven across categories, and strong specialization does not guarantee high overall performance.Kimi-K2 and Mistral-Small-24B perform strongly on counterfactual reasoning but poorly on fault localization, while GLM-4.5 shows the reverse pattern; GLM-4.5 also reaches 68.12 EM on functional mapping but only 37.00 overall EM.