Source-linked AI summary
An Open-Source, Event-Driven Pipeline for Cryptocurrency Market Data: Ingestion, Forecasting, and On-Chain Fraud Detection
Basil Sajid Shaikh, Melrick Mascarenhas, Nuzhat Faiz Shaikh
TL;DR
The paper addresses how small teams can reproduce cloud-native cryptocurrency data infrastructure without managed streaming and warehousing services. It builds an open-source Kafka, filesystem-watcher, Spark, and PostgreSQL pipeline, then evaluates forecasting and fraud-detection tasks, reporting strong fraud-classification metrics while qualifying both evaluation settings.
Problem
Commercial-grade streaming and warehousing infrastructure is costly or inaccessible for small research groups, startups, and student projects working with cryptocurrency data.
Method
The paper builds a single-machine open-source pipeline using Kafka events, independent audit and ETL consumers, Spark transformations, PostgreSQL schemas, and asset-specific marts.
Results
Gradient Boosting reached accuracy 0.99 and ROC-AUC 0.9994 on the Ethereum fraud benchmark, with fraud precision 1.00 and recall 0.98.
Takeaways & Limitations
The pipeline provides a reproducible integration of infrastructure and downstream modeling, while the reported forecasting pattern aligns with prior work and the fraud result is not presented as benchmark-beating.
Takeaways & Limitations
The ARIMA-LSTM comparison uses mismatched resampling frequencies and forecast horizons, so it should be interpreted qualitatively rather than as a controlled experiment.
Abstract
from arXiv · showhide
Cryptocurrency markets generate high-frequency, multi-source data that is expensive to work with unless a team already has commercial-grade streaming and warehousing infrastructure in place. This paper describes a fully open-source pipeline that reproduces the behavior of a cloud-native, event-driven system -- file arrival triggering a message, a message triggering compute -- entirely on commodity hardware, using Apache Kafka and a filesystem-watching poller in place of managed cloud triggers. The pipeline partitions historical Gemini exchange data into hourly and minutely files, ingests them asynchronously through two independently grouped Kafka consumers (one for audit logging, one for Spark-triggered ETL), and lands cleaned output in a PostgreSQL warehouse with historical and aggregated schemas plus asset-specific data marts. We use the resulting Bitcoin data mart to compare a seasonal ARIMA model against a single-layer LSTM network for price forecasting, and separately apply Random Forest and Gradient Boosting classifiers, with additional engineered features, to the public Ethereum fraud detection benchmark introduced by Farrugia et al. We report the architecture, the modeling methodology, and the resulting metrics, and we are explicit about the limitations of comparing forecasts issued at different horizons and of evaluating fraud detection on a static, already-labeled dataset.
1. Introduction
The paper presents an open-source, single-machine pipeline that reproduces cloud-native event-driven ingestion and supports two downstream modeling exercises. It contributes a decoupled Kafka-based architecture, PostgreSQL warehouse and marts, Bitcoin forecasting comparison, and Ethereum fraud-classification extension.
- Architecture: The pipeline reproduces the storage-trigger-compute pattern of managed cloud infrastructure using open-source components on a single machine.It uses Apache Kafka, a filesystem watcher, Apache Spark, and PostgreSQL.
- Architecture: Two independent Kafka consumer groups separate audit logging from Spark-triggered computation while reading the same event stream.This decouples operational visibility from the compute path.
- Data layer: A two-schema PostgreSQL warehouse and per-asset data marts support dashboarding and downstream modeling from shared cleaned data.The mart layer avoids rebuilding transformation logic for each consumer.
- Modeling exercises: The study compares seasonal ARIMA with LSTM for Bitcoin price forecasting using data produced by the pipeline.The paper explicitly accounts for the unequal footing of the two evaluations.
- Modeling exercises: The fraud study replicates and extends the Farrugia et al. Ethereum benchmark with eight engineered features and Random Forest versus Gradient Boosting.The benchmark is public and distinct from the ingestion pipeline's own output.
2. Related Work
Related work covers Kafka-based decoupled ingestion, Spark-based parallel data processing, cryptocurrency forecasting, and Ethereum fraud detection. The paper positions its contribution as documenting both systems and modeling within one pipeline while keeping their data relationship explicit.
- Systems foundations: The ingestion design follows Kafka's partition-based log model, while Spark's resilient distributed datasets support parallel merging, cleaning, and aggregation of many files.The local implementation substitutes a filesystem watcher for managed storage-trigger services.
- Forecasting: Prior cryptocurrency forecasting studies generally report that LSTM and other nonlinear sequence models track short-horizon price movement more closely than ARIMA.ARIMA remains a fast, interpretable baseline.
- Fraud detection: The Farrugia et al. Ethereum benchmark uses transactional and behavioral account features and reported XGBoost accuracy of 96.3% and AUC of 99.4%.The dataset pairs community-flagged illicit addresses with regular accounts.
- Positioning: Unlike write-ups focused only on systems or modeling, this paper documents both halves while identifying the Ethereum dataset as an external benchmark.This avoids overstating integration between ingestion and fraud detection.
3. System Architecture
The architecture converts historical exchange files into event-driven Kafka notifications, independently consumed audit and Spark paths, cleaned warehouse outputs, and asset-specific marts. The Ethereum fraud dataset appears as an external benchmark rather than an ingestion product.
- Data preparation: Historical Gemini OHLCV files are split into daily arrivals, with invalid timestamps, prices, and volumes discarded and retained timestamps normalized to ISO 8601.The source covers multiple assets and shares a common CSV schema.
- Event-driven ingestion: A Watchdog filesystem poller detects file creation or modification through native operating-system events and publishes metadata to Kafka.The message records mode, file type, path, and detection time rather than file contents.
- Event-driven ingestion: Metadata-only Kafka events keep messages small and allow consumers to read files independently while asset-based partitioning enables parallel processing.This locally reproduces the decoupling of object-storage triggers from compute.
- Two independent Kafka consumers: Independent consumer groups audit every event and trigger Spark after configurable batches, with separate offsets supporting restart without reprocessing completed files.Additional consumers can be added without changing the existing groups.
- Transformation and storage: Spark cleans and types batch data, computes daily price and volume summaries, and uses DataFrame logic that can move from local mode to a multi-node cluster.The transformations include mean, median, minimum, maximum, and total volume.
- Data marts and dashboards: An asset-specific SQL mart and dashboards provide downstream views, while the Ethereum fraud dataset is drawn independently from the pipeline.The figure distinguishes the external benchmark from the ingestion flow.
4. Architectural Advantages: Scalability and Extensibility
The pipeline separates ingestion, transformation, and storage into independently scalable and recoverable components. Asset-specific marts let downstream models and dashboards reuse cleaned data without duplicated processing.
- Scalability: Separating ingestion, transformation, and storage allows each subsystem to scale independently as workload demands change.Kafka consumers can scale through repartitioning, while Spark can move from local mode to a multi-node cluster.
- Fault isolation and recovery: Independent Kafka consumer groups isolate audit logging from Spark-triggered warehouse production, so failures remain confined to one path.Each stage also commits its own progress, enabling recovery by restarting only the affected stage.
- Extensibility: Adding a tradable asset requires no new ingestion or transformation code; only an asset-specific data mart must be added for dedicated outputs.The existing watcher, Kafka topic, and Spark job use the asset symbol carried by file paths and event payloads.
- Data reuse: Historical, aggregated, and per-asset schemas let dashboards and forecasting models consume shared cleaned data without re-deriving it from raw files.The Bitcoin mart supports both forecasting experiments and the dashboard layer without duplicated cleaning logic.
- Design comparison: The architecture makes trade-offs explicit by comparing its ingestion strategy with a bulk-load ETL script and a managed cloud pipeline.The comparison covers alternatives such as S3 notifications, Lambda, and a managed warehouse.
5. Forecasting Bitcoin Prices
The study evaluates seasonal ARIMA and LSTM forecasting on Bitcoin data prepared at different resolutions and horizons. Both produce usable forecasts, while the LSTM follows short-term movement more tightly, but the setups do not support a controlled quantitative comparison.
- Study design: The study compares seasonal ARIMA as an interpretable baseline with LSTM for capturing non-linear Bitcoin price dynamics.The models were developed from the pipeline's Bitcoin warehouse data.
- SARIMAX preparation: ADF testing found the raw monthly series non-stationary at p = 0.998, while transformation and differencing reduced the value to 0.045.The workflow applied a Box-Cox transform followed by seasonal and first-order differencing before treating the series as stationary.
- SARIMAX model: SARIMAX(1,1,0)×(0,1,1,12) was selected by AIC grid search with AIC = 231.86, and its residuals showed no significant remaining autocorrelation.The residual ADF p-value was under 0.001.
- SARIMAX results: The six-month SARIMAX forecast tracks the broader seasonal cycle and upward trend but misses the sharp late-sample price acceleration.Forecasts were transformed back to the original price scale for comparison with actual prices.
- LSTM method: LSTM uses daily scaled prices for one-step-ahead prediction with a single four-unit layer and a dense output layer.The network was trained with Adam and mean squared error for 100 epochs.
- Comparison and limitation: LSTM has substantially lower error on its own test window, but differing resolutions, spans, and horizons make the comparison qualitative rather than controlled.The paper describes both models as usable forecasts and the non-linear model as tracking short-term movement more tightly, not as generally 41% more accurate.
6. Ethereum Wallet Fraud Detection
The fraud experiment applies engineered-feature preprocessing and tuned tree classifiers to a labeled Ethereum wallet benchmark. Gradient Boosting performs strongest, reaching high fraud-class precision and recall alongside ROC-AUC of 0.9994.
- Dataset: The benchmark contains 9,841 Ethereum wallet addresses described by 51 transactional and behavioral features, with 22.14% labeled fraudulent.Features include timing, volume, ERC-20 activity, and smart-contract interaction measures.
- Feature engineering: Preprocessing expanded the representation to 817 dimensions through imputation and one-hot encoding, while eight engineered ratios, diversity measures, and activity features sharpened the fraud signal.Identifier columns were removed and missing values were imputed with zero.
- Modeling procedure: Random Forest and Gradient Boosting used scaled pipelines, GridSearchCV tuning, and a stratified 75/25 split; SMOTE was implemented but not activated.The roughly 1:3.5 fraud-to-legitimate ratio was considered manageable without oversampling.
- Results: Gradient Boosting was stronger on both F1 score and ROC-AUC, reaching accuracy 0.99, fraud precision 1.00, fraud recall 0.98, and ROC-AUC 0.9994.The classification report covers 2,461 held-out addresses.
- Feature interpretation: The strongest Gradient Boosting features include ERC-20 token activity, received-address diversity, and account lifetime, consistent with the benchmark's reported fraud-signal patterns.The trained model was serialized with joblib for reuse.
7. Discussion
The paper's contribution is integration rather than algorithmic novelty: one cheaply reproducible pipeline supports two unrelated downstream tasks while decoupling audit logging from computation. Fraud results align with the established benchmark, whereas the forecasting comparison supports only a qualitative reading.
- The modeling components are standard techniques; the paper's main contribution is integrating them within one cheaply reproducible pipeline.The warehouse and mart layer is exercised by forecasting and fraud-detection tasks.
- Two independent Kafka consumer groups read one topic, keeping audit logging decoupled from the compute path without coordination overhead.
- Accuracy 0.99 and AUC 0.9994 from Gradient Boosting are comparable to, and marginally above, Farrugia et al.'s 96.3% accuracy and 99.4% AUC.The paper attributes the difference, if anything, to eight engineered features and a different train/test split, not a fundamentally stronger model.
- LSTM tracks short-term price movement more closely than ARIMA, matching the broader literature, but the forecasting setup is not a rigorous head-to-head benchmark.
8. Limitations
The evaluation is bounded by local, task-specific, and non-controlled settings. The system was not tested at real streaming volumes, forecasting used mismatched horizons, and fraud detection used a static labeled dataset without live on-chain integration.
- The pipeline was evaluated as a single-node local simulation, without load testing at real streaming volumes or partial-failure testing.
- Forecasting was restricted to Bitcoin, while the multi-asset framing appears only in ingestion, warehouse, and dashboard layers.
- The ARIMA–LSTM comparison used mismatched resampling frequencies and forecast horizons, so it should be read qualitatively rather than as a controlled experiment.
- Fraud detection used a static, already-labeled 9,841-row public dataset, a random rather than temporal split, and no live on-chain integration.
- No cost or throughput benchmarking against a managed-cloud deployment was performed, leaving the cost-effectiveness argument architectural rather than measured.
9. Conclusion
The paper concludes that a single-machine, open-source pipeline can reproduce a cloud-native ingestion pattern and support independent forecasting and fraud-detection exercises. Results were usable but unsurprising, and future work should broaden assets, control evaluations, and measure operational performance.
- Watchdog-triggered Kafka events, independent consumer groups, Spark ETL, PostgreSQL schemas, and per-asset marts reproduce cloud-native ingestion on one machine.
- LSTM outperforms ARIMA on short-horizon price tracking, while Gradient Boosting performs in line with the established Ethereum fraud benchmark.
- Future work should extend both experiments to additional assets under a shared controlled evaluation protocol.
- The pipeline's actual throughput and cost should be measured against a managed-cloud equivalent rather than argued from architecture alone.
Data and Code Availability
The paper provides public access to dashboards, the Ethereum fraud dataset, and the underlying Gemini exchange history through external services.
- Two Tableau dashboards are published on Tableau Public: a single-asset performance dashboard and a multi-asset comparison dashboard.
- The Ethereum fraud-detection dataset is publicly available on Kaggle, and Gemini exchange history is available through CryptoDataDownload.