PaperlayerView exact revision on arXiv

Source-linked AI summary

Graphologue: Exploring Large Language Model Responses with Interactive Diagrams

Peiling Jiang, Jude Rayan, Steven P. Dow, Haijun Xia

arXiv:2305.11473v2cs.HCcs.AIcs.CL

TL;DR

Complex LLM tasks are constrained by verbose text responses and linear conversations that make information difficult to comprehend, organize, and explore. Graphologue converts LLM responses into interactive, real-time node-link diagrams with context-aware graphical interaction. In evaluation, it supported comprehension and control, while annotation and relationship extraction reached high F-scores with one correction round.

  • Problem

    Text-based, linear LLM interfaces can produce verbose, difficult-to-track responses and limit flexible interaction during complex information tasks.

  • Method

    Graphologue uses prompting strategies and interactive diagrams to annotate entities and relationships in LLM responses, support real-time node-link construction, and translate graphical manipulation into context-aware prompts.

  • Results

    97.24% entity-annotation F-score and 92.39% relationship-annotation F-score were achieved with one correction round, while users quickly grasped concepts and connections with control over diagram complexity.

  • Takeaways & Limitations

    Graphologue provides flexible graphical dialogue and combines diagrams, raw text, and outlines to support information exploration, organization, and comprehension.

  • Takeaways & Limitations

    Participants mainly chose topics they were familiar or very familiar with, leaving support for users with varying topic familiarity for future study.

Abstract

from arXiv · show

Large language models (LLMs) have recently soared in popularity due to their ease of access and the unprecedented ability to synthesize text responses to diverse user questions. However, LLMs like ChatGPT present significant limitations in supporting complex information tasks due to the insufficient affordances of the text-based medium and linear conversational structure. Through a formative study with ten participants, we found that LLM interfaces often present long-winded responses, making it difficult for people to quickly comprehend and interact flexibly with various pieces of information, particularly during more complex tasks. We present Graphologue, an interactive system that converts text-based responses from LLMs into graphical diagrams to facilitate information-seeking and question-answering tasks. Graphologue employs novel prompting strategies and interface designs to extract entities and relationships from LLM responses and constructs node-link diagrams in real-time. Further, users can interact with the diagrams to flexibly adjust the graphical presentation and to submit context-specific prompts to obtain more information. Utilizing diagrams, Graphologue enables graphical, non-linear dialogues between humans and LLMs, facilitating information exploration, organization, and comprehension.

1 INTRODUCTION

Text-based LLM conversations can overwhelm users during complex information tasks because responses are verbose, structurally limited, and difficult to explore flexibly. Graphologue addresses these constraints by turning LLM responses into interactive node-link diagrams and evaluating their benefits.

  • Text-based conversational interfaces struggle with complex information activities because text poorly communicates intricate relationships, while linear exchanges hinder non-linear exploration.These constraints can produce verbose exchanges in which users lose track of information.
  • Graphical representations can reduce comprehension effort by organizing concepts non-linearly and support information manipulation through interaction.The project uses these properties to support exploratory information-seeking, concept explanation, and question-answering with LLMs.
  • A formative study with ten participants found that LLM responses were verbose and time-consuming to comprehend, while linear conversations required cumbersome copy-and-paste and scrolling.The study examined how participants used ChatGPT to explore and learn about a domain.
  • Graphologue converts LLM text into graphical diagrams in real time using prompting strategies that annotate entities and relationships for node-link construction.The system also supports user control over diagram complexity.
  • Users can manipulate diagrams to obtain context-specific information, creating a graphical dialogue with LLMs while adjusting, collapsing, and combining diagram content.These interactions support flexible exploration of concepts and relationships.
  • An evaluation with seven experienced LLM users found that Graphologue helped participants grasp key concepts and connections while controlling diagram complexity.Synchronized raw text and outline views let participants use multiple representations at different levels and scales.

2 RELATED WORK

Prior work established natural-language interaction and graphical representations as complementary ways to communicate and organize information, but Graphologue targets dynamic LLM-generated text. It combines real-time diagram construction with flexible, synchronized interaction across representations.

  • Natural Language User Interfaces and LLMs: LLMs broaden natural-language interaction across domains but introduce usability challenges, including arduous prompt engineering for aligned results.This motivates interface support beyond purely textual prompting.
  • Natural Language User Interfaces and LLMs: Natural-language interfaces let users express actions and goals directly, but often require translation into rigid commands or queries.Later systems inferred intents from natural expressions to make interaction less rigid.
  • Natural Language User Interfaces and LLMs: Graphologue uses GPT-4 to obtain information and annotate its own responses, extending graphical communication between humans and machines in the spirit of Sketchpad.The system combines language generation with graphical representation.
  • Generating Graphical Representations from Text: Graphical representations support comprehension, memory, inference, and sensemaking by exploiting human visual information processing and enabling modification of representations.These benefits motivate graphical support for LLM responses.
  • Generating Graphical Representations from Text: Earlier systems generated visualizations, videos, or node-link diagrams from static text sources such as documents, transcripts, and social media data.Examples include UML diagrams from specifications and concept maps from multiple YouTube videos.
  • Visualization and Multilevel Abstraction: Unlike prior static-text approaches, Graphologue constructs and supports interaction with node-link diagrams from dynamic LLM output in real time.Its design also draws on focus-plus-context, semantic zooming, and progressive disclosure principles.

3 FORMATIVE STUDY

The formative study examined how ten participants used ChatGPT for exploratory learning and identified problems with response verbosity, interaction flexibility, and organization. These findings motivated the need for a more structured and interactive representation of LLM information.

  • Formative Study: The formative study investigated participants’ experiences and challenges when using conversational interfaces to interact with LLMs.Its findings informed Graphologue’s design choices.
  • Participants and Procedure: Ten participants included first-time, casual, and experienced ChatGPT users, providing varied levels of familiarity with the system.Sessions lasted one hour and were conducted over Zoom.
  • Participants and Procedure: Participants selected one of four topics and spent 30 minutes using ChatGPT to explore related concepts and questions.The tasks were designed to support both broad and deep topic exploration.
  • Findings and Discussion: Participants averaged nine conversational exchanges while exploring the assigned concepts and questions.The interviews were analyzed for recurring challenges.
  • C1. Response Content is Verbose and Lacks Structure: Participants found ChatGPT responses verbose, overwhelming, and lacking visual hierarchy, making large amounts of information difficult to navigate and comprehend.They suggested different formats, collapse controls, and shortening repetitive content.
  • C2. Lack of Flexible Interaction with the Response Text: All participants extracted response passages to query them further and wanted direct interaction with the response instead of sequential conversational follow-ups.They wished to highlight a passage and ask for an explanation directly.
  • C3. Lack of Organization Across Multiple Responses: Participants struggled to remember and retrieve information across multiple exchanges because redundant answers formed a single, poorly organized stream.They proposed bookmarking and overview mechanisms to support return and navigation.
  • Summary: Overall, the study identified problems in the quantity, organization, presentation, and interaction of ChatGPT responses, motivating better support for intuitive understanding and flexible exploration.The summary connects these challenges to the need for improved information representation.

4 DESIGN GOALS AND RATIONALE

Graphologue’s design goals make diagrams the entry point for LLM information, keep their complexity adjustable, support diagram-based exploration, and synchronize diagrams with text. Together, these goals address comprehension, control, and context during interaction.

  • D1. Diagram as Entry Point: Diagrams should be generated concurrently with LLM text and serve as the entry point to reduce comprehension effort.Presenting complete text before subsequently displaying diagrams was found to increase cognitive effort.
  • D2. Flexible Control of Diagram Complexity: Users should control both how much information appears in a diagram and how that information is revealed.This goal addresses complexity arising from source-text content and diagram presentation.
  • D3. Diagram-Based Exploration: Users should explore LLM information through diagram nodes and links, including requesting explanations or examples and collapsing or trimming content.Diagram interactions are intended to support typical information tasks directly.
  • D4. Synchronized Interaction Between Diagrams and Text: Diagrams and text should remain synchronized because diagrams provide overview while the original text supplies detailed context and supports entity-correction needs.The design combines multiple representations rather than replacing the textual response.

5 ENVISIONED SCENARIO

Graphologue supports exploratory learning by constructing diagrams alongside streaming LLM responses, allowing users to inspect concepts incrementally and then integrate them into a holistic view.

  • 5 ENVISIONED SCENARIO: The scenario frames Graphologue as a quicker way to understand an unfamiliar topic than relying on a long text response alone.Margaret is tired from writing and seeks an easier way to learn about earthquakes.
  • 5 ENVISIONED SCENARIO: Graphologue constructs a node-link diagram piece by piece as the LLM response streams, making key concepts and connections available during generation.In the scenario, Margaret identifies relationships among tectonic plates, fault lines, and seismic waves while the response is still arriving.
  • 5 ENVISIONED SCENARIO: Synchronized highlighting lets users verify diagram relationships against the corresponding details in the original response.Pointing to “seismic waves” highlights its associated text for direct reference.
  • 5 ENVISIONED SCENARIO: Separate paragraph-level diagrams let users learn one aspect at a time before merging them to see how the concepts relate as a whole.The merged view combines the smaller diagrams into one complete picture.

6 PROMPTING FOR DIAGRAM GENERATION

Graphologue uses prompt-mediated annotations and response structuring to build manageable diagrams in real time, while additional prompting and controls address annotation errors and information overload.

  • 6 PROMPTING FOR DIAGRAM GENERATION: Inline entity and relationship annotations allow diagrams to be constructed concurrently with streaming GPT-4 responses rather than after a separate processing round.The system removes annotations for presentation, extracts entities and relationships, and renders corresponding diagrams as tokens arrive.
  • 6.1 Diagram Construction: Prompt components structure the initial response into short thematic paragraphs, each paired with a smaller diagram to reduce overload and support navigation.Each paragraph focuses on one theme, aspect, or topic.
  • 6.1.2 Annotating Entities: GPT-4 annotates entities and assigns shared identifiers for co-references, enabling repeated mentions of one concept to map consistently to diagram nodes.The system uses identifiers such as N1 to associate entities through relationships.
  • 6.1.2 Annotating Entities: GPT-4 annotates relationships as diagram links, including multi-entity connections that can be organized together during rendering.Relationship annotations may represent several connections between entity pairs.
  • 6.1.3 Annotating Relationships and Saliency: Saliency filtering renders only high-saliency relationships by default, while users can reveal all relationships to control diagram clutter.This balances a less cluttered default view with access to lower-saliency connections.
  • 6.1.3 Annotating Relationships and Saliency: Prospective relationship annotations can create placeholder nodes before the corresponding entities finish streaming, making diagram construction more responsive.The placeholder later transforms into the actual node when its entity arrives.
  • 6.2 Error Prevention and Correction: GPT-4 annotation is imperfect, with errors that can produce redundant, unparsable, or incorrect diagram content.Reported errors include assistant-style text, inconsistent annotation formats, spurious entities, and recurring entity-relationship problems.
  • 6.2.2 Self-Correction with Additional Rounds of Processing: Self-correction prompts identify problematic sentences and request re-annotation or rewriting, with corrections performed in parallel so diagram interaction remains available.Updated annotations adjust and animate the diagram after correction.

7 GRAPHOLOGUE INTERFACE

Graphologue makes diagrams the primary interface for exploring LLM responses while synchronizing them with text-based views and providing controls for filtering, expanding, merging, and correcting information.

  • 7 GRAPHOLOGUE INTERFACE: Graphologue parses streamed annotations and immediately adds entities and relationships to keep diagrams responsive and up to date.Placeholder nodes can represent entities whose text has not yet arrived.
  • 7.2 Bidirectional Synchronization: Bidirectional synchronization connects diagram nodes and relationships with their text tokens, enabling cross-reference and verification between representations.Hovering over diagram nodes highlights co-references and corresponding text, while hovering over text highlights diagram elements.
  • 7.3 Interaction with Diagram Nodes: Collapsing nodes and greying corresponding text tokens reduce the saliency of less pertinent information during exploration.This helps users focus on active nodes and relevant response content.
  • 7.2 Bidirectional Synchronization: Outline, summary, and original-text views provide alternative levels and forms of detail alongside the diagrams.The interface supports outline and summary presentations in addition to the raw response.
  • 7.3 Interaction with Diagram Nodes: Node-oriented exploration lets users request explanations or examples for unfamiliar concepts and integrate the generated material into the existing diagram.New entities and relationships are co-referenced or cumulatively added as nodes and links.
  • 7.4 Diagram-Level Information Managing: Users can collapse or hide leaf nodes to remove irrelevant information and reduce diagram complexity.These controls support concentrating on selected areas of interest.
  • 7.3.3 User-Initiated Correction Through Trimming and Merging: Graphologue supports manual correction by trimming incorrectly extracted nodes and merging nodes that refer to the same concept.These operations also update associated relationships and annotations.
  • 7.4 Diagram-Level Information Managing: Separate diagrams can be merged into a holistic view, with text-block highlighting and selective hiding preserving links between source passages and combined nodes.Users can merge any selected subset of paragraph diagrams.

8 TECHNICAL EVALUATION

Graphologue’s technical evaluation assessed GPT-4 prompts for inline entity and relationship annotation, including one round of self-correction. The prompts achieved high annotation F-scores, with correction improving both entity and relationship performance.

  • Evaluation Setup: Semantic correctness was accepted across different annotation granularities, while syntactic and semantic errors were classified during evaluation.Examples included reversed relationships and alternative segmentations of the same phrase.
  • Initial Annotation Performance: Before correction, entity annotation achieved an F-score of 95.82%, while relationship annotation achieved an F-score of 84.77%.The largest error groups were missing entity phrases, incomplete entities, missing relationships, and dead-end relationships.
  • Initial Annotation Performance: Graphologue detected 133 orphan nodes and 37 dead-end relationships in the annotated responses, enabling GPT-4 to perform a correction round.The system detected these structural errors before requesting corrections.
  • Correction Results: After one correction round, entity annotation reached an F-score of 97.24% and relationship annotation reached 92.39%.Entity precision and recall rose to 97.11% and 97.37%, while relationship precision and recall rose to 94.10% and 90.75%.

9 USER EVALUATION

The user evaluation examined Graphologue’s potential for supporting exploratory information-seeking through interactive diagrams. Participants reported benefits for understanding relationships, controlling information complexity, obtaining context-specific information, and combining representations, while also identifying space, mental-model, animation, latency, and annotation limitations.

  • Participants used Graphologue to gather information for a hypothetical lecture covering related concepts and questions.
  • 9.4.1 Graphologue Facilitates Information Comprehension: Participants reported that node-link diagrams clarified relationships, provided an overall view, and made information faster to retrieve than wordy text responses.
  • 9.4.1 Graphologue Facilitates Information Comprehension: Bidirectional highlighting between diagrams and paragraphs helped participants attend to relationships and locate explanatory terms and concepts.
  • 9.4.2 Graphologue Enables Flexible Information Exploration: Participants appreciated concise presentations and controls over detail, including paragraph-level diagrams and flexible split or merged views.
  • 9.4.3 Graphologue Reduces Prompting Effort and Facilitates Exploration: Graphologue reduced prompting effort by letting users control complexity and request examples or explanations through diagram interactions.Selecting a node and choosing “Explain” extended the paragraph and added a diagram branch for context-specific clarification.
  • 9.4.4 Graphologue Combines the Strengths of Multiple Representations: Participants combined diagrams, outlines, and original text to obtain complementary relational, hierarchical, and detailed views of information.The outline organized key ideas, diagrams showed interconnections, and text supplied details missing from diagrams.
  • 9.4.4 Graphologue Combines the Strengths of Multiple Representations: Separating responses into smaller diagrams reduced complexity but increased scrolling, exposing a trade-off between diagram complexity and spatial efficiency.The authors propose allowing users to merge any selected diagrams in future iterations.
  • 9.4.5 Limitations: Some participants struggled when diagrams conflicted with mental models or contained overwhelming detail, while animations, generation latency, and annotation errors also limited the experience.Providing original text alongside diagrams helped mitigate mental-model mismatches and misleading annotations.

10 DISCUSSION

The discussion frames Graphologue as a graphical interaction approach for exploring interconnected information from LLM responses. It highlights direct manipulation, possible applications and verification uses, while recognizing annotation errors and the need for task-appropriate representations.

  • Graphologue’s inline annotations and interface designs offer a direct representation of concepts and relationships in LLM-generated information.
  • Annotation errors, including links to nonexistent entities, can produce misleading diagrams that require cross-reference with the original text.
  • Improved prompts, domain knowledge, and comparisons with traditional NER and SRL methods are identified as ways to improve annotation performance.
  • Direct manipulation lets users request explanations and examples without manually composing textual prompts, with future work adding more graphical and customized interactions.
  • Node-link diagrams are especially suited to exploratory information seeking involving diverse concepts and their interconnections, with education and professional training identified as applications.
  • Breaking LLM responses into individually validated node pairs and relationships could support verification against external knowledge bases.
  • Node-link diagrams may not suit every information task; tables, storyboards, animations, and flowcharts can better represent other information structures.

11 CONCLUSION

The conclusion positions Graphologue as an alternative to linear, text-based LLM interfaces that uses dynamic interactive diagrams to address challenges in usability, readability, and interactivity.

  • Graphologue addresses three challenges of textual LLM responses—limited usability, readability, and interactivity—through dynamic, interactive diagrams.

A PROMPTS

Graphologue uses GPT-4 prompts to produce annotated responses whose entities and relationships support real-time diagrams, summaries, outlines, and context-specific expansions. Self-correction prompts repair annotation errors while preserving meaning and identifier consistency.

  • A PROMPTS: Graphologue’s prototype uses predefined System, User, and Assistant roles when querying the OpenAI GPT-4 API.
  • A.1 Initial Query: Initial prompts request concise, structured paragraphs that annotate entities and relationships inline for concept-map construction.
  • A.1 Initial Query: Entity annotations identify noun phrases, while relationship annotations connect existing entities and mark relationship saliency as high or low.
  • A.1 Initial Query: Example prompts demonstrate consistent identifiers and relationship annotations across explanations of artificial intelligence, human-computer interaction, and bird flight.
  • A.2 Self-Correction: Self-correction prompts detect orphan entities and relationships pointing to nonexistent entity identifiers, then request corrected annotations without changing sentence meaning.
  • A.2 Self-Correction: Correction instructions assign new identifiers to newly added entities, reuse identifiers for previously mentioned entities, and return only the re-annotated sentence.
  • A.3 Summary: Summary prompts generate one short annotated sentence that preserves original entity identifiers and includes only high-saliency relationships.
  • A.4 Outline / A.5 Node Explanation: Additional prompts structure source text into markdown presentation slides and request concise, relationship-annotated explanations for selected entities.

B TECHNICAL EVALUATION EXPLANATION AND EXAMPLES

The technical evaluation defines annotation error categories by the affected entity or relationship structure. These categories distinguish malformed entities, missing or invalid links, and direction or attribution errors.

  • Table 2 presents explanations and examples for the annotation error types.
  • Annotation errors include incomplete entities, orphan nodes, incorrect entities, and inconsistent co-reference labels.Orphan nodes are entities without relationships, while incorrect co-reference labeling assigns different identifiers to co-referenced entities.
  • Relationship errors include missing relationships, incomplete relationship phrases, dead-end links, reversed directions, and misattributed entity pairs.
  • Dead-end relationships are links to nonexistent nodes and are measured as a percentage of total extracted relationships.
Loading 2305.11473v2…