PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System

Bowen Yu, Claudio T. Silva

arXiv:1908.00681v2cs.HCcs.LG

TL;DR

Dataflow visualization systems offer flexible exploration but can be difficult to learn and operate. FlowSense addresses this gap with a context-aware semantic parser that maps natural-language queries to VisFlow functions using special utterance tagging and placeholders. The case study and user study demonstrate its effectiveness while identifying limitations in query correction, concept equivalence, and composite-query support.

  • Problem

    Dataflow visualization systems provide analytical flexibility, but their diagram complexity creates learning overhead, while many visualization natural-language interfaces do not support relationships among multiple views.

  • Method

    FlowSense uses semantic parsing, special utterance tagging, and special utterance placeholders to map natural-language input to VisFlow functions independently of datasets and diagrams while retaining dataflow context.

  • Results

    FlowSense supports natural-language queries for the majority of VisFlow dataflow diagram editing operations and was evaluated through a domain-expert case study and formal user study.

  • Takeaways & Limitations

    FlowSense combines natural-language usability with dataflow flexibility to simplify diagram construction and support interactive, iterative visual data exploration.

  • Takeaways & Limitations

    Users found it unclear how to revise rejected queries, and the system struggles with concept equivalence and concise composite queries.

Abstract

from arXiv · show

Dataflow visualization systems enable flexible visual data exploration by allowing the user to construct a dataflow diagram that composes query and visualization modules to specify system functionality. However learning dataflow diagram usage presents overhead that often discourages the user. In this work we design FlowSense, a natural language interface for dataflow visualization systems that utilizes state-of-the-art natural language processing techniques to assist dataflow diagram construction. FlowSense employs a semantic parser with special utterance tagging and special utterance placeholders to generalize to different datasets and dataflow diagrams. It explicitly presents recognized dataset and diagram special utterances to the user for dataflow context awareness. With FlowSense the user can expand and adjust dataflow diagrams more conveniently via plain English. We apply FlowSense to the VisFlow subset-flow visualization system to enhance its usability. We evaluate FlowSense by one case study with domain experts on a real-world data analysis problem and a formal user study.

1 INTRODUCTION

FlowSense addresses the learning overhead of dataflow visualization systems by combining natural-language usability with dataflow flexibility. Its context-aware semantic parsing supports multi-view diagram construction and is evaluated through a domain-expert case study and formal user study.

  • Motivation: Dataflow visualization systems support flexible, multi-view analysis but impose greater learning overhead because users must manage complex dataflow diagrams.Natural-language interfaces can reduce required prior knowledge, but many existing systems focus on single visualization answers rather than relationships among multiple views.
  • Approach: FlowSense combines natural-language interaction with dataflow flexibility to simplify construction and adjustment of multi-view visualization diagrams.It is implemented on VisFlow, whose visualizations support interactivity, brushing, and linking.
  • Approach: FlowSense uses semantic parsing with special utterance tags and placeholders to remain independent of datasets, diagram elements, and analytical tasks while retaining dataflow context.Recognized utterances are shown interactively, helping users understand parser state, identify errors, and resolve ambiguity.
  • Evaluation: FlowSense supports natural-language queries for the majority of VisFlow dataflow editing operations and was assessed through a traffic-speed case study with domain experts and a formal user study.The evaluation measured task completion time, collected user feedback, and analyzed natural-language query logs.

2 RELATED WORK

The related work positions FlowSense at the intersection of natural-language visualization interfaces and flexible dataflow systems. Unlike interfaces centered on answering individual questions, FlowSense targets dataflow-context interaction and diagram construction.

  • Dataflow visualization systems: Dataflow visualization systems let users compose modules in diagrams to configure functionality and support flexible visualization pipelines.Prior systems have used dataflow to render scientific data and manage visualization pipelines.
  • Natural-language visualization interfaces: Visualization-oriented natural-language interfaces commonly answer queries with visual representations, conversational follow-ups, ambiguity handling, or recommendations for relevant views.Examples include DataTone, Eviza, Evizeon, Iris, Orko, Analyza, IBM Watson Analytics, and Microsoft Power BI.
  • FlowSense positioning: FlowSense differs by addressing dataflow context and assisting diagram construction rather than directly answering free-form analytical questions or selecting a best visualization.Its output is a complete, interactive, iterative exploration process that preserves workflow provenance and user preferences.
  • FlowSense positioning: FlowSense maps natural-language input to VisFlow functions through a predefined grammar and semantic parser.The parser recursively expands grammar variables to match queries and interpret the rules applied during derivation.

3 SEMANTIC PARSER

FlowSense builds a context-aware semantic parser for VisFlow by combining a general grammar with runtime-resolved dataflow entities. It supports common diagram-construction patterns while exposing recognized utterances and resolving ambiguity during parsing.

  • Dataflow context: The system identifies columns, node labels, node types, and dataset names as special utterances and presents them through consistently colored interface tags.In the Auto MPG example, “mpg,” “horsepower,” and “origin” are columns, “MyChart” is a node label, and “parallel coordinates” is a node type.
  • Grammar and parsing: FlowSense derives VisFlow functions from natural-language queries using a context-free grammar and semantic parsing.The parser uses grammar variables for visualization requests, verbs, columns, and visualization types, with recursive derivation implemented through SEMPRE.
  • Dataflow context: FlowSense uses special utterance placeholders that are replaced at runtime by dataset and diagram elements, keeping grammar rules independent of datasets, diagrams, and tasks.For example, a generic column placeholder is mapped to a recognized dataset column such as “mpg” after parsing.
  • Dataflow context: Special utterance tagging reduces grammar size and gives users real-time feedback about available operations and query interpretation.The design lets VisFlow functions operate on context-specific diagram elements without writing rules for each dataset or diagram.
  • Derivation ambiguity: FlowSense resolves some syntactic ambiguity with supervised derivation scoring rather than adding many restrictive grammar rules.Fewer than twenty training examples guided preferred derivations for a rule set of around 500 rules, while the authors note this does not solve natural-language ambiguity generally.
  • Query pattern: The parser is designed around five query components: function type, options, source nodes, target nodes, and port specification.Its variable hierarchy accepts different component orders and supports equivalent phrasings for the same operation.

4 QUERY EXECUTION

FlowSense executes parsed queries by recognizing utterances, classifying query intent, filling missing components, and selecting an editing focus in the current diagram. It also applies defaults to resolve execution choices, including ambiguous filtering operations.

  • Recognition and parsing: FlowSense recognizes special utterances and part-of-speech information while the user types, then uses these signals to support query execution.Approximate matching improves typo tolerance, while POS tagging supports generalized grammar rules for interchangeable prepositions.
  • Recognition and parsing: Keyword classification maps semantically related words such as “show,” “visualize,” and “draw” to appropriate VisFlow functions.Words such as “find” may indicate a filtering requirement, helping the system infer the intended operation.
  • Query completion: After identifying query components, FlowSense fills missing information with defaults or the current diagram editing focus.A simple scatterplot request can receive two default numerical columns, and other missing specifications can be completed from interaction context.
  • Editing focus: When a query omits source or target nodes, FlowSense selects an editing focus using node activeness and proximity to the mouse cursor.Activeness decays over time after clicks; the highest-scoring node is selected, with decreasing scores used to order multiple source nodes.
  • Editing focus: Node-type references can identify diagram nodes, with focus scores breaking ties when multiple nodes share the requested type.For example, a query can refer to an existing scatterplot by node type.
  • Execution ambiguity: For filtering queries with multiple valid executions, FlowSense defaults to filtering a visualization’s input and allows refined wording to specify another behavior.Adding a phrase such as “from the plot” explicitly indicates that filtering should apply to the existing visualization’s output.

4.4 Diagram Update

FlowSense executes accepted queries by creating, updating, or operating on VisFlow nodes, then locally adjusts the diagram layout. It distinguishes grammar, dataflow-context, and expectation errors, providing correction messages for the first two while leaving later adjustments to the user.

  • Diagram Update: Accepted queries invoke VisFlow functions that can create nodes, update existing nodes, or operate on multiple nodes simultaneously.For example, visualization creates one plot, highlighting creates three nodes, and rendering-color changes can update nodes without creating new ones.
  • Diagram Update: FlowSense applies a modified force-directed layout after diagram updates to reduce clutter and node overlap around the editing focus.The layout accounts for rectangular node sizes, giving larger nodes stronger repulsive force; user-adjusted positions are remembered.
  • Error Recovery: FlowSense identifies grammar rejection, invalid dataflow references, and fully executed queries that fail to match user expectations as distinct error types.Examples include unsupported requests, references to nonexistent visualization nodes, and ambiguous or unintended operations.
  • Diagram Update: The case-study visualization combines a speed-distribution histogram and a time-series line chart, both color-encoded by road speed limit.The smaller pre-step-3 histogram shows the distribution before color encoding was added.
  • Error Recovery: For grammar and dataflow-context errors, the system displays a message and requests correction; expectation mismatches require user adjustment of the diagram.Users can undo FlowSense actions or make partial adjustments in the underlying VisFlow system when the result is not exactly desired.

5 EVALUATION

FlowSense was evaluated through a domain-expert case study and a formal user study, showing support for practical dataflow analysis and generally successful task completion. Feedback indicates that FlowSense simplifies diagram construction and speeds exploration, while rejected-query correction and natural-language variation remain limitations.

  • 5.1 Speed Reduction Study: The case study used NYC taxi speed data to examine speed-limit reductions through progressively constructed FlowSense visualizations.Analysts created histograms, color encodings, time-series charts, maps, filters, and linked views; they observed speed reductions across speed-limit groups beginning around mid-2013.
  • 5.1 Speed Reduction Study: Domain experts found FlowSense helpful for completing a comprehensive analysis and learning how to build VisFlow diagrams.The case study specifically highlighted FlowSense’s support for a practical analytical task and its role in facilitating DFVS learning.
  • 5.2 Experiment Overview: The formal study recruited 17 computer-science participants with varied backgrounds and asked them to complete tutorials and analytical tasks using FlowSense and VisFlow.Participants included visualization designers, data scientists, and software engineers; the task phase used the SDE Test dataset and logged diagram edits, answers, completion times, feedback, and query logs.
  • 5.2.3 Task Completion Quality: Most participants answered the user-study tasks correctly, and tutorial and task completion times were mostly as expected.Task time increased when tasks involved heterogeneous tables and interactive filtering; four interruption-related outliers were excluded from the displayed completion-time plot.
  • 5.2.4 Quantitative Feedback: Participants reported that FlowSense simplified diagram construction and accelerated data exploration compared with standalone VisFlow.Twelve participants agreed that FlowSense simplified construction, while ten agreed that it sped up exploration.
  • 5.2.5 Query Log Analysis: The evaluation identified limitations in rejected-query correction, logical inference, concept equivalence, and concise composite-query parsing.Suggested corrections were left for future work, while users could often circumvent parsing limitations by rephrasing queries or splitting composite queries into smaller steps.

6 DISCUSSION AND LIMITATIONS

FlowSense’s grammar and placeholder design supports dataflow-context-aware interaction, but its functionality remains coupled to grammar coverage and underlying VisFlow operations. Users can often recover from rejected or unsupported queries, while analytical question answering remains outside the current scope.

  • Grammar maintenance: The grammar currently contains about 200 variables and around 500 rules, with ongoing refinement to expand functionality.
  • Generalizability: FlowSense uses special utterance placeholders to represent data- and diagram-independent elements, making the approach potentially applicable to other modular dataflow systems.The grammar remains coupled to VisFlow functionality, but the placeholder approach can be extended to systems such as KNIME.
  • Limitations: Grammar-based parsing can reject acceptable or out-of-scope queries, although users were willing and able to refine rejected queries through a small number of attempts.
  • User behavior and efficiency: FlowSense enables composite and batch operations that would otherwise require sequential mouse and keyboard interactions, including complicated sub-diagram creation such as highlighting.
  • Design rationale: The authors prefer semantic parsing because visualization-oriented NLI and DFVS lack suitable training data for deep-learning methods, though future user queries could provide such a dataset.
  • Scope boundary: FlowSense currently supports dataflow diagram editing rather than analytical questions, so answering questions such as whether vehicle speed decreases over years requires further research.

7 CONCLUSIONS

FlowSense applies semantic parsing to map natural-language input to VisFlow functions while using special utterances to remain dataset- and diagram-independent yet dataflow-context aware. Its feedback and completion features support understanding of parsing state, and the case study and user study demonstrate effectiveness while identifying future improvements.

  • System: FlowSense is a natural-language interface for visual data exploration within a dataflow visualization system, implemented for VisFlow.
  • Approach: Semantic parsing maps natural-language input to VisFlow functions, while special utterances and placeholders preserve dataflow context across datasets and diagrams.
  • Interaction: Real-time tagged-utterance feedback, query completion, and token completion help users understand the underlying parsing state.
  • Evaluation: The case study and user study demonstrate the proposed interface’s effectiveness and identify directions for future improvement.

A FlowSense Grammar Design

The FlowSense implementation is distributed as an open-source repository containing its grammar, backend, and tests. Its grammar is organized in SEMPRE format, with main.grammar as the entry point.

  • Repository: The open-source repository includes FlowSense’s grammar rules, TypeScript and Python backend API, integration tests, and setup documentation.
  • Grammar organization: The grammar rules are stored in *.grammar files, and main.grammar serves as the entry point.
  • Grammar format: FlowSense grammar rules use the SEMPRE grammar format, including SEMPRE formulas.

B Characteristics of the Diagram Editing Focus Heuristics

FlowSense selects diagram elements for natural-language operations using a focus score based on recent mouse interaction and spatial proximity. Background clicks shift selection toward nearby nodes, which can occasionally produce unintended targets.

  • Focus score: The focus score tracks the most recently interacted diagram element using activeness from mouse clicks and a distance-to-mouse bonus.
  • Node clicks: Clicking a node gives it an activeness score of one, generally making it the highest-scoring query target despite nearby nodes’ proximity bonuses.
  • Background clicks: After a background click, nearby nodes can dominate the focus score, so accidental canvas interactions may select a node the user did not intend to target.

C Additional User Study Remarks

The study reports scoring and timing details that qualify interpretation of the user-study results. One timing summary excludes four interruption-related outliers.

  • Negative penalties made skipping potentially worthwhile in the SDE test, but skipping required an explicit button click.An unanswered result was recorded when participants took no action before the question time limit expired.
  • Four interruption-related task-completion-time outliers were omitted from Fig. 6(b).The excluded measurements were 2550 minutes for Task1 and 109, 119, and 212 minutes for Task3.
  • VisFlow users generally understood the dataflow visualization system and used it effectively for visual data exploration.The survey feedback used colored score-distribution bars, with darker green representing higher scores.

D VisFlow Survey Results

The VisFlow survey assessed whether participants understood the subset-flow model and could use its features for visual data exploration. Most participants gave favorable effectiveness ratings.

  • 17 computer-science participants completed the VisFlow study, including 12 with a data visualization background.The group comprised 11 male and 6 female participants aged 20–30; the study also included visualization designers, data scientists, and software engineers.
  • Participants assessed VisFlow effectiveness using a 1–5 Likert scale, where 5 meant strongly agree and 1 meant strongly disagree.
  • Most participants understood VisFlow’s subset-flow model and agreed that it effectively supported visual data exploration.The majority also agreed, with scores of at least 4, that they could successfully use VisFlow features for exploration.

E Query Analysis – Failure Category Description

The query analysis classifies failures by whether functionality was unimplemented, invalid, rephrased, unsupported, mistagged, composite, or buggy. These categories distinguish grammar and implementation issues from limitations of the underlying visualization framework.

  • Not Implemented: Not Implemented failures occur when grammar support or query-execution handlers for a function have not yet been added.Changing a node’s x column is an example, and the category can be addressed by extending the grammar and web-client rules.
  • Invalid/Mistyped: Invalid/Mistyped failures involve malformed sentences or mistyped words that do not identify the intended data entity or dataflow element.
  • Rephrased: Rephrased failures arise when users employ grammar structures or dataset terminology that FlowSense does not expect.For example, “degree” is not recognized as equivalent to the HighestLevelOfEducation column without additional equivalence knowledge.
  • Not Supported: Not Supported failures exceed VisFlow’s underlying dataflow capabilities rather than reflecting an NLI grammar problem.Answering an analytical question such as how many questions were skipped would require constructing potentially complex dataflow sub-diagrams.
  • Tagging Error: Tagging Errors occur when special utterances are incorrectly tagged or left untagged, creating parsing ambiguity.Users may manually override a dataset-name tag when a term such as “iris” also describes a data entity.
  • Composite: Composite failures occur when one query attempts more VisFlow functions than the grammar or web-client handler supports.Highlighting bachelors and masters with different colors requires multiple filters, visual edits, and a set operation.
  • Bug: Bug failures occur when supported queries produce incorrect parsing or execution because of an implementation defect.
Loading 1908.00681v2…