PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow

Wenqi Zhang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Weiming Lu, Yueting Zhuang

arXiv:2306.07209v8cs.CLcs.AIcs.CE

TL;DR

LLMs face context, numerical-processing, and output-format limits when used to analyze massive data, while data work is often tedious and requires specialized expertise. Data-Copilot combines code-based analysis with advance exploration and reusable interfaces for real-time requests. Its evaluations report higher success rates and lower token consumption than other strategies, and its interface workflows are presented as easier for people to inspect and interact with than lengthy code.

  • Problem

    Directly using LLMs to process massive data is constrained by context limits, weak numerical and table-manipulation abilities, and limited output formats, while data tasks often require tedious work and specialized expertise.

    Source: §1 · p. 1

  • Method

    Data-Copilot uses code to query, process, and visualize data, and explores data in advance to design, test, and optimize reusable interfaces for real-time workflows.

    Source: §1 · p. 2, §2 · p. 3, §2.1 · p. 5

  • Results

    Evaluations report higher success rates and lower token consumption than other strategies for Data-Copilot on Chinese financial data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present interface workflows as more efficient and easier for human inspection and interaction than generating lengthy code.

    Source: §1 · p. 3

    The described release focuses on Chinese financial data, while effectiveness in other domains is presented as potential.

    Source: §1 · p. 3, Limitations · p. 9

Abstract

from arXiv · show

Industries such as finance, meteorology, and energy generate vast amounts of data daily. Efficiently managing, processing, and displaying this data requires specialized expertise and is often tedious and repetitive. Leveraging large language models (LLMs) to develop an automated workflow presents a highly promising solution. However, LLMs are not adept at handling complex numerical computations and table manipulations and are also constrained by a limited context budget. Based on this, we propose Data-Copilot, a data analysis agent that autonomously performs querying, processing, and visualization of massive data tailored to diverse human requests. The advancements are twofold: First, it is a code-centric agent that receives human requests and generates code as an intermediary to handle massive data, which is quite flexible for large-scale data processing tasks. Second, Data-Copilot involves a data exploration phase in advance, which explores how to design more universal and error-free interfaces for real-time response. Specifically, it actively explores data sources, discovers numerous common requests, and abstracts them into many universal interfaces for daily invocation. When deployed in real-time requests, Data-Copilot only needs to invoke these pre-designed interfaces, transforming raw data into visualized outputs (e.g., charts, tables) that best match the user's intent. Compared to generating code from scratch, invoking these pre-designed and compiler-validated interfaces can significantly reduce errors during real-time requests. Additionally, interface workflows are more efficient and offer greater interpretability than code. We open-sourced Data-Copilot with massive Chinese financial data, such as stocks, funds, and news, demonstrating promising application prospects.

1 Introduction

Data-Copilot uses code and pre-designed interfaces to automate data analysis and visualization, addressing limits of direct LLM processing and the errors and costs of generating lengthy code. Its evaluations on Chinese financial data report higher success rates and lower token consumption than other strategies.

1 Introduction

  • Direct LLM processing is constrained by context limits, weak numerical and table-manipulation abilities, and text-only output, while code generation can be error-prone and costly in time and tokens.Data-related tasks also often require specialized knowledge and tedious manual work.Source: §1 · pp. 1–2
  • Data-Copilot generates code to query, process, and visualize massive data, while exploring data in advance to design reusable interfaces for real-time requests.For requests the interfaces do not cover, it can combine them with raw code.Source: §1 · pp. 2–3
  • Pre-designed interfaces are tested before deployment, reducing real-time errors while improving inference efficiency and making workflows easier to inspect and interact with than lengthy code.The paper also describes flexible interface workflows for different requests.Source: §1 · p. 3
  • Evaluations on Chinese financial data report higher success rates and lower token consumption than other strategies.The system is open-sourced for analysis of stocks, funds, and live news.Source: §1 · p. 3

2 Data-Copilot

Data-Copilot prepares reusable interfaces through data exploration, then deploys them in workflows for real-time requests. It can also retrieve relevant interfaces, combine them with generated code for uncovered requests, and update its library as new requests accumulate.

2 Data-Copilot

  • The code-centric agent explores data schemas, designs interfaces, and tests and optimizes them in advance for later human-instructed data analysis and visualization.At deployment, it can invoke pre-designed interfaces or generate raw code for a request.Source: §2 · p. 3

2.1 Data Exploration

  • Seed requests and data parsing files guide request synthesis; backward verification filters requests that rely on data that does not exist.Interfaces are code modules with names, parameters, descriptions, and implementations, designed from synthesized requests while prioritizing existing modules.Source: §2.1 · p. 5
  • New interfaces are tested on similar requests and revised using compiler feedback; similar interfaces can be merged, then checked against the original interfaces’ outputs.Design, testing, and optimization repeat as the library expands across task and operation types.Source: §2.1 · pp. 5–6

2.2 Workflow Deployment

  • At deployment, hierarchical retrieval selects interfaces by task and operation, and workflows can use ordered, parallel, or loop structures.If existing interfaces cannot resolve a request, Data-Copilot can generate raw code or combine code with interfaces, record uncovered requests, and later update the library.Source: §2.2 · p. 6

3 Dataset Synthesis

Data-Copilot builds a request and interface library for Chinese financial data, then uses those interfaces in workflows that can combine existing modules with raw code. The dataset and qualitative examples illustrate the library’s construction, scope, and outputs.

3.1 Environment and Data Sources

  • Data-Copilot was developed on Chinese financial data, including more than 20 years of time-series data for over 4,000 stocks, funds, and futures.The data also encompasses economic data, real-time news, and company financial data.Source: §3.1 · p. 6

3.2 The Creation of Dataset

  • Ten economics students proposed 500 seed requests, which were filtered to 173 high-quality requests for data exploration.The requests covered stock and fund tasks at different complexity levels.Source: §3.2 · p. 6
  • The synthesized requests closely align with human-proposed requests in task-type and complexity distributions.The paper reports these distributions in Table 1.Source: Table 1 · p. 7, §3.2 · p. 7
  • 547 test requests were sampled from 3,547 retained synthesized requests, while the remaining 3,000 supported interface design.Exploration generated 6,480 requests before highly similar requests were filtered out.Source: §3.2 · p. 7

3.3 Qualitative Evaluation

  • Analyzing Chinese financial data and 3,000 self-exploration requests produced 73 interfaces across five functionalities.The functionalities are data acquisition, index calculation, table manipulation, visualization, and general processing.Source: §3.3 · p. 7
  • For unsupported requests, Data-Copilot can combine existing interfaces with raw code, such as retrieving data through an interface and generating radar-chart code.This mixed workflow is presented as a way to accommodate evolving demands.Source: §3.3 · p. 7
  • Step-by-step interface calls make complex workflows easier to inspect than code, alongside visual outputs and workflow summaries.The paper illustrates these outputs in Figures 2, F2, and F4–F7.Source: §3.3 · p. 7

4 Quantitative Evaluation

Across the reported comparisons, Data-Copilot improves accuracy and token efficiency over agent baselines, with particularly larger gains on complex requests. The evaluation also examines deployment settings, repeated-generation savings, and the roles of interface design and optimization.

4.1 Experiments Settings

  • Data-Copilot was compared with Direct-Code, ReAct, Reflexion, and multi-agent methods on 547 test cases using accuracy and token consumption.Accuracy combines data-table and image evaluation; GPT-4o evaluates the outputs.Source: §4.1 · p. 7

4.2 Comparison Results

  • Compared with Direct-Code, smaller deployment models perform at GPT-4 level, and GPT-4 gains a further 4%.The authors connect reduced errors to interface optimization and validation during design.Source: §4.2 · p. 8, Table 2 · p. 8
  • Against Reflexion, Data-Copilot improves accuracy by 11.2% and reduces token consumption by 75%.The comparison reports accuracy and efficiency for GPT-3.5-turbo.Source: §4.2 · p. 8, Table 3 · p. 8

Data-Copilot Reduces Repetitive Generation

4.3 Why Data-Copilot Brings Improvements?

  • For multiple-entity requests, accuracy improvements reach 5.3%, rising to 19% for multiple entities with complex relations; single-entity improvement is 4%.The paper attributes the stronger complex-scenario performance to versatile interfaces reducing response complexity.Source: §4.3 · p. 8, Table 4 · p. 8
  • Using GPT-3.5-turbo for interface design and GPT-4-turbo for deployment scored 49.6, below Direct-Code with GPT-4-turbo at 70.2.The authors report that GPT-3.5-turbo-designed interfaces tend to fail and generalize poorly, and use GPT-4-turbo for interface design and optimization instead.Source: §4.3 · pp. 8–9, Table 5 · p. 8

5 Conclusion

Data-Copilot is a code-centric agent for large-scale data analysis that uses exploration to build interfaces for reliable real-time requests. The reported experiments show higher success rates with lower token consumption.

5 Conclusion

  • Data-Copilot is a code-centric agent that generates code for large-scale data processing and creates interface modules through data exploration.The conclusion describes these modules as universal interfaces invoked to solve varied requests.Source: §5 · p. 9
  • The paper reports that Data-Copilot improves real-time request success and achieves higher success rates with lower token consumption.The conclusion characterizes the interfaces as supporting reliable problem-solving across data types.Source: §5 · p. 9

Limitations

The limitations concern offline interface design, occasional instability in workflow deployment, and the financial-domain scope of the current development. The workflow deployment also parses requests and returns results in multiple forms.

Limitations

  • Interface design is currently offline, and the authors identify simultaneous online design and deployment as important for broadening application scenarios.The authors describe online design as a future direction.Source: Limitations · p. 9
  • LLM instruction failures or incorrect answers can cause anomalies in interface workflows, making deployment occasionally unstable.The authors identify reducing uncertainty during interface dispatch as future work.Source: Limitations · p. 9
  • The current development focuses on data tasks using China Financial Data, while effectiveness in other domains is presented as potential.The chosen data includes stocks, futures, finance, macroeconomics, and financial news, with diverse schemas and requests.Source: Limitations · p. 9

A Preprocessing of Workflow Deployment

  • Intent analysis parses request time, location, data object, and output format, using local time and network IP information alongside the original request.The example converts a request about local GDP and CPI trends into a chart request for China’s quarterly data from May 2019 to May 2024.Source: A Preprocessing of Workflow Deployment · p. 14
  • Workflow deployment can return graphics, tables, and descriptive text, alongside a summary of the workflow.The GDP forecasting example uses data retrieval, prediction, and visualization interfaces in sequence.Source: A Preprocessing of Workflow Deployment · p. 14

B.1 Baselines and Experiments Details

The appendix describes the comparison baselines, defines three test-request complexity levels, and reports a human evaluation of synthesized requests against human-proposed requests.

B.1 Baselines and Experiments Details

  • The experiments compare Data-Copilot with direct code generation, ReAct, Reflexion, and three-agent collaboration, using the same LLM for baseline interface invocation.Reflexion is limited to two iterations; the collaboration baseline uses two coders and one manager.Source: §B.1 · p. 14

B.2 The Definitions of Three Complexity Levels for Testset

  • Test requests are grouped as single-entity, multiple-entity, or multiple-entity requests with complex relations such as loops or nesting.The groups distinguish step-by-step requests, simultaneous processing across entities, and requests with more intricate logic.Source: §B.2 · p. 15

B.3 The Quality of Self-exploration Requests.

  • Four graduate students evaluated 546 synthesized and 173 human-proposed requests on difficulty, rationality, ambiguity, and answer accuracy.Synthesized requests had comparable quality, with slightly higher difficulty and ambiguity and similar rationality; test-set answer-table labels received a 4.8 score.Source: §B.3 · p. 15

B.4 The detail of GPT-4o Evaluation

The appendix describes data-table and image evaluation procedures and compares GPT-4o image scores with human ratings on a sample of 50 examples.

B.4 The detail of GPT-4o Evaluation

  • GPT-4o compares each predicted data table with its human-annotated counterpart and marks it false if it identifies any inconsistency.The example is judged false because the predicted table omits the 2020 value.Source: §B.4 · p. 15
  • Image evaluation uses a checklist of five categories and ten sub-dimensions, with scores above 60 considered meeting expectations.The categories include numerical points, lines, axes, layout aesthetics, and chart design.Source: §B.4 · p. 15
  • On 50 sampled images, GPT-4o and human evaluations had average scores of 67.2 and 65.8, respectively, with a score-sequence correlation of 0.894.The appendix reports the average scores and correlation as close and strong, respectively.Source: §B.4 · p. 15

B.5 Human Evaluation on Test Benchmark

The appendix outlines human-evaluation criteria and identifies tables and a figure describing test-set composition and evaluation comparisons.

B.5 Human Evaluation on Test Benchmark

  • Four graduate students evaluated the benchmark on task difficulty, request rationality, expression ambiguity, and answer accuracy.The criteria assess task demands, whether requests fit human habits, clarity of phrasing, and whether answers meet expectations.Source: §B.5 · p. 15
  • Table B1 reports test-set statistics across four task types and three complexity levels.Source: Table B1 · p. 16
  • Table B2 compares human-proposed and synthesized requests across four evaluation dimensions.Source: Table B2 · p. 16
  • Table B3 is titled as a comparison of human and GPT-4o evaluations.Source: Table B3 · p. 16
  • Figure B1 counts test-set keywords by request type: Stock, Corp., Fund, and Others.Source: Figure B1 · p. 16

B.6 Ablation Study

The section reports performance drops when exploration, interface optimization, or hybrid generation is removed, and compares performance across programming languages. It also describes related work, visualization cases, and supporting prompts and workflows.

B.6 Ablation Study

  • 34.3 performance drop follows removal of self-exploration, 28-point drop follows removal of interface optimization, and 6.4-point drop follows removal of hybrid generation.The authors attribute the first drop to too few seed requests, the second to similar interfaces, and the third to uncovered requests.Source: §B.6 · p. 16

B.7 Expanding to Other Programming Languages

  • Accuracy was 70.2 for Python, 54.2 for C++, and 36.5 for Matlab; the authors ultimately selected Python.They report Matlab formatting errors or dimensional discrepancies that made code non-executable, and speculate that limited Matlab code in pretraining may contribute.Source: §B.7 · p. 16

C Related Works

  • Most surveyed LLM-agent applications focus on daily tools or code generation rather than the specificity of data-related tasks (Xie et al., 2023).The related-work discussion also covers LLM applications to data science, including spreadsheet manipulation, structured data, exploration, and Text2SQL.Source: C Related Works · p. 17

D Visualization

  • Visualization cases use loop and parallel workflows for complex requests, and the system handles varied sources through self-designed interfaces with minimal human intervention.The examples include parallel graph-and-table output and financial data from stocks, funds, news, and other sources.Source: D Visualization · p. 17

E Detailed Prompts

  • Prompts cover self-exploration, interface design and optimization, intent analysis, task selection, workflow planning, and four baseline strategies.The baseline prompts are for Direct-Code, ReAct, Reflection, and Multi-Agent Collaboration.Source: E Detailed Prompts · p. 18

E.2 Example for Data Exploration

  • The exploration prompt expands seed requests into diverse common needs using dataset descriptions, access methods, output schemas, and sample rows.Its example includes GDP and stock data.Source: §E.2 · p. 18

F Case Study

  • The paper points to detailed data-exploration and real-time workflow examples in Appendices F.1 and F.2.These examples are presented as the case study for Data-Copilot.Source: F Case Study · p. 18

Intent Analysis Phase

  • The prompt asks the system to parse time, place, object, and output format, then rewrite the request based on today’s date into a semantically consistent instruction with a specific time and specific indicators.Source: Intent Analysis Phase · p. 21

F.1 Cases for Data Exploration ----

The case studies show how Data-Copilot designs and combines interfaces during exploration, then invokes workflows for real-time requests across financial data and forecasting tasks. Examples include stock, fund, and GDP requests, with outputs such as charts and tables.

F.1 Cases for Data Exploration ----

  • A bank-price plotting workflow chains data retrieval, preparation for graphical representation, and trend-chart plotting.The example proposes separate interfaces for each step, then uses them in that order.Source: §F.1 · p. 24
  • The optimization example merges getBankData() and getStockData() into get-financial-data(), while retaining reusable preparation and plotting interfaces.The merged function is intended to fetch bank and stock financial data.Source: §F.1 · p. 24

F.2 Cases for Real-time Request

  • A rewritten Guizhou Maotai request becomes stock-data and visualization tasks, with daily prices retrieved, closing prices selected, and a line chart plotted.The request specifies the period January 23, 2018 to March 13, 2019.Source: §F.2 · p. 25
  • A GDP-growth forecast request retrieves historical year-over-year data, predicts the next four quarters, and prints and saves a table.The example uses China GDP data through June 7, 2023.Source: §F.2 · p. 26
  • Examples show loop workflows for repeated stock queries and parallel workflows for comparisons, with outputs including graphs, tables, and multiple indicators on one canvas.The paper also describes handling stocks, funds, news, and financial data through interfaces despite differences in format and access method.Source: Figure F2 · p. 26, Figure F4 · p. 28, Figure F5 · p. 29
  • For a fund-manager request, the workflow retrieves managed funds, calculates each fund’s return over the specified period, and plots the cross-sectional results.The example uses Zhou Haidong’s funds from January 1 to June 6, 2023, and plots a histogram.Source: §F.2 · p. 30
  • A Bank of Chengdu request combines candlestick and KDJ data for the past year and plots both indicators.The workflow retrieves price and technical-indicator series, calculates the indicators, and merges them before plotting.Source: §F.2 · p. 31

B. Text Output Panel

  • The interface figure identifies the user-input panel separately from the system’s returned results.The input panel is marked A; the results are marked B and C.Source: Figure F9 · p. 33

Related papers

Loading 2306.07209v8…