Source-linked AI summary
PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns
Syrym Abdikhan, Mazhar Hameed
TL;DR
Real-world Pandas workflows and their evolution within Jupyter notebooks have remained insufficiently characterized. PandasCorpus addresses this gap by systematically collecting and analyzing 139k notebooks, finding improving syntactic validity, recurring core operations, and compositional workflow regularities.
Problem
Systematic evidence about how Pandas workflows are constructed, which operations dominate, and how they evolve in real-world notebooks remains limited.
Method
PandasCorpus collects and organizes Pandas-containing Jupyter notebooks from GitHub between 2015 and 2025, then analyzes structural, temporal, executability, and operation-composition patterns.
Results
Notebook syntax-error rates fell from 74.6% in 2015 to 10.7% in 2024, while workflows consistently centered on a compact set of recurring operations and compositions.
Takeaways & Limitations
The corpus provides a structured basis for studying workflow sequences and recurring patterns in code retrieval, recommendation, prediction, analysis, and optimization.
Takeaways & Limitations
The dataset does not support attributing observed changes in notebook error rates to a single cause, so causal interpretation requires further investigation.
Abstract
from arXiv · showhide
Pandas has emerged as the de facto library for data processing and machine learning, widely used for tasks, such as data loading, transformation, and analysis. Despite its ubiquity, there has been limited systematic investigation into how Pandas is used in real-world projects and how typical workflows are composed in practice. To address this gap, we introduce PandasCorpus, a dataset curated from GitHub repositories that captures real-world Pandas workflows at scale. In this work, a workflow refers to Pandas-based code contained in Jupyter notebooks, a prevalent medium for writing, executing, and sharing data analysis code. The dataset comprises 139k notebooks from approximately 100k repositories and captures more than 4M Pandas API calls spanning 136 distinct operations. Beyond dataset construction, we characterize workflows using structural and Pandas-specific features and analyze notebook evolution between 2015 and 2025. Our study examines code executability, notebook size, and recurring sequences of Pandas operations, providing empirical insights into how Pandas is used in practice. The resulting corpus offers a reusable resource for studying data analysis workflows, Pandas usage patterns, and library-aware code composition. Both the dataset and the extraction pipeline are publicly available via GitHub and Zenodo.
1 PANDAS AS THE DE-FACTO LIBRARY
Pandas is the de-facto standard for data-driven workflows, yet how Pandas-based workflows are constructed and evolve in Jupyter notebooks remains poorly understood. PandasCorpus addresses this gap with a large, structured collection of real-world notebooks and a reproducible analysis pipeline.
- Motivation: Pandas has become the de-facto standard for data-driven workflows, supporting data loading, transformation, and analysis.Its flexibility places responsibility on users to compose operations effectively and efficiently.
- Motivation: Jupyter notebooks are the dominant medium for experimentation, teaching, and practical data science work.They are lightweight, shareable, and support a wide ecosystem of libraries across academia and industry.
- Research gap: Empirical understanding of how Pandas-based workflows are constructed and evolve within notebooks remains limited.Existing studies have largely examined general notebook properties, machine learning practices, reproducibility, coding habits, or other libraries such as NumPy.
- Research gap: The lack of a dedicated resource prevents systematic Pandas-specific analysis at scale, including which operations dominate real workloads and how workflows evolve over time.GitHub hosts many Jupyter notebooks, but they have not been systematically analyzed for these questions.
- Contribution: 139k Jupyter notebooks from approximately 100k distinct repositories form PandasCorpus, covering notebooks created between 2015 and 2025.The corpus is collected from GitHub, organized into structured corpora, and stored in Parquet format for efficient access.
- Contribution: PandasCorpus provides a reproducible collection and analysis pipeline for characterizing notebook structure, Pandas-specific workflows, temporal evolution, and recurring operation patterns.The pipeline collects, filters, and organizes Jupyter notebooks containing Pandas-based workflows for structured notebook-level analysis.
2 PandasCorpus
PandasCorpus is constructed from public GitHub repositories by collecting and filtering Jupyter notebooks containing Pandas-based workflows. It characterizes these workflows using AST-based extraction and complementary notebook-level and Pandas-specific features.
- Data collection: 136 distinct Pandas API operations guided GitHub queries for Jupyter notebooks containing Pandas-based workflows.Collection targeted DataFrame-centric methods and functions as search targets.
- Collection pipeline: 198k candidate notebooks from 137k repositories were reduced through deduplication, retrieval, and relevance filtering to 139k notebooks.The pipeline first discovers notebook URLs, then downloads candidates and applies relevance checks.
- Code parsing: Cleaned notebook code is parsed into an Abstract Syntax Tree to identify function calls and syntactic constructs precisely.Sanitization removes IPython magic and shell commands and resolves minor syntax inconsistencies before parsing.
- Workflow characterization: PandasCorpus organizes workflow characteristics into general notebook-level properties and Pandas-specific API usage and composition patterns.The Pandas-specific group includes usage frequency, chained operations, chained indexing, visualization activity, and magic or shell-command use.
3 ANALYSIS AND FINDINGS
Analysis of PandasCorpus shows improving notebook syntactic validity, highly concentrated Pandas API usage, and recurring multi-stage workflow patterns. These workflows commonly progress through integration, exploration and validation, cleaning and transformation, and aggregation.
- Notebook executability: 74.6% of notebooks contained syntax errors in 2015, declining to 10.7% in 2024 before rising slightly to 11.6% in 2025.The dataset also grew substantially, especially after 2022; the decline’s causes cannot be attributed to a single factor.
- Notebook executability: 42.07% of parsing errors involved unmatched or unclosed brackets, followed by unterminated strings at 21.05%, generic syntax errors at 16.51%, and indentation errors at 10.4%.These categories reflect statically detected syntactic inconsistencies rather than runtime or semantic errors.
- Pandas API usage: The top 15 Pandas operations account for more than half of all recorded Pandas calls, while specialized operations contribute negligibly.Frequently used operations include head, mean, sum, groupby, and drop; their usage grows with notebook volume.
- Workflow composition: Recurring sequences were filtered to length-two minimum, occurrence in at least 1% of notebooks (≈1,400 notebooks), and exclusion of trivial repetitions.The resulting patterns were grouped into integration and construction, exploration and validation, cleaning and transformation, and aggregation and statistics.
- Workflow composition: Integration commonly combines sources through read_csv → concat or read_csv → merge, while exploration follows loading with head → describe or head → info and quality checks.Validation sequences include head → isnull → sum and sum → duplicated → sum, indicating checks for missing values and duplicate records.
- Workflow composition: Cleaning typically follows validation through isnull → sum → dropna or duplicated → sum → drop_duplicates, while aggregation centers on groupby → agg → reset_index (4.86%).Additional patterns show staged imputation, time-series preprocessing, iterative regrouping, and multi-stage aggregation, supporting a recurring layered progression.
4 IMPLICATIONS AND CONCLUSION
PandasCorpus is a large-scale resource of real-world Pandas workflows from GitHub, supporting empirical analysis of Pandas usage and research on structured data-manipulation workflows. By preserving operation sequences and contextual dependencies, it supports retrieval, recommendation, and next-step prediction for interactive data analysis.
- Contributions: 139k Jupyter notebooks from approximately 100k repositories spanning 2015 to 2025 form PandasCorpus, a structured resource for studying real-world Pandas usage.The corpus supports analyses of notebook executability trends, commonly used operations, and recurring operation compositions.
- Research directions: PandasCorpus preserves workflow structure rather than isolated functions, allowing retrieval models to account for operation sequences and contextual dependencies.This design distinguishes the corpus from traditional code datasets focused on isolated functions.
- Research directions: Its workflow structure supports code search, workflow recommendation, and next-step prediction in interactive data analysis environments.These applications are enabled by modeling sequences of operations and their contextual dependencies.