Source-linked AI summary
Web Price Extraction: State of the Art and an Adaptive Browserless Implementation
Evgeniia Kositsyna, Jorge Lloret-Gazo
TL;DR
Reliable web price extraction must balance adaptability, accuracy, scalability, and computational cost across changing website structures. The paper develops an adaptive browserless system that combines rule-based fragmentation with Bayesian weight updates and genetic-algorithm optimisation. The hybrid scheme raises precision from 77.2% to 87.3% and reduces average per-page processing time by approximately 14% relative to the baseline.
Problem
Existing price-extraction approaches trade off accuracy, adaptability, scalability, and resource use across classical, browser-based, browserless, and AI/ML methods.
Method
The paper builds a browserless rule-based fragmentation pipeline and extends it with Bayesian rule-weight updates and a genetic algorithm for global parameter tuning.
Results
13.1% higher extraction precision and approximately 14% lower average per-page processing time were achieved by the combined Genetic Algorithm and Bayesian weighting scheme relative to the baseline.
Takeaways & Limitations
The hybrid approach is presented as a competitive alternative to manually tuned browserless extractors and more resource-intensive browser- or large-language-model-based methods.
Takeaways & Limitations
The evaluation used approximately 200 test records, so the reported validation remains preliminary and should be expanded for narrower confidence intervals.
Abstract
from arXiv · showhide
Price extraction from websites is a key task for market monitoring, price comparison, and business analytics in e-commerce. Existing approaches can be broadly divided into four groups, and understanding their trade-offs in accuracy and scalability is essential for selecting suitable extraction strategies. Classical methods rely on manually written wrappers and rule induction from labeled pages, offering high accuracy but adapting poorly to structural changes and requiring considerable maintenance effort. Browser-based methods, using tools such as Selenium and Puppeteer, handle dynamic JavaScript content but consume large computational resources and scale poorly. Browserless approaches retrieve HTML directly via HTTP requests, offering significant gains in speed and cost, but rely on rules calibrated for specific sites. Methods based on machine learning and large language models offer adaptability but require training data and substantial computation. Our main contribution is an adaptive browserless price extraction system that improves robustness to structural differences between websites. We implemented a baseline architecture combining HTML page fragmentation with syntactic, semantic, and frequency rules, and extended it in two ways: a Bayesian approach that dynamically updates rule weights, and a genetic algorithm that optimizes the system's global parameters. This hybrid scheme increased precision from 77.2% to 87.3% and reduced average per-page processing time by approximately 14% relative to the baseline, confirming it as a competitive alternative to manually tuned browserless solutions and to more resource-intensive browser- or LLM-based methods, offering high extraction accuracy at low computational cost.
1 Introduction
The paper compares four web-scraping approaches and develops an adaptive browserless price-extraction system to address structural changes while retaining low resource requirements. Its system combines rule-based extraction with Bayesian and genetic-algorithm adaptations and evaluates three configurations comparatively.
- Price extraction supports competitor monitoring, market-trend detection, and automated pricing in e-commerce.
- Constantly changing HTML structures, JavaScript-generated content, and growing collection demands make reliable web extraction challenging.
- Classical wrappers are usually precise but adapt poorly to layout changes and require substantial manual maintenance.
- Browser-based methods handle dynamic interfaces but consume considerable resources and scale poorly, whereas browserless methods are faster and cheaper but struggle with client-side rendering.
- AI and machine-learning methods can adapt to changing structures but introduce training, data, computational, and generalisation challenges.
- The proposed system combines HTML fragmentation with syntactic and semantic rules, then adds Bayesian rule-weight updates and genetic-algorithm parameter optimisation.
- The study compares baseline, Bayesian-enhanced, and genetic-algorithm-optimised configurations using precision, coverage, and processing time.
2 State of the art
Web extraction methods trade off adaptability, accuracy, resource use, and scalability. Classical and browserless approaches can be efficient or accurate under specific conditions, while browser simulation and AI-based methods address dynamic or changing structures with different computational and data requirements.
- Methods based on artificial intelligence: Machine-learning extraction can adapt to changing page structures but requires substantial labelled data containing both positive and negative examples.The cited discussion identifies training complexity, dataset availability, computational cost, and generalisation as associated challenges.
- Classical methods: Classical XPath and DOM methods achieved up to 100% F-measure while requiring minimal computational resources.The reported results used 2011-era hardware.
- Browser simulation methods: Browser rendering allocates approximately 13% of execution time to initialization, 85% to rendering, and 2% to extraction.Initialization and rendering together require roughly 50 times more time than data retrieval, while 96.8% of visual-wrapper traffic is irrelevant to target-data acquisition.
- Browserless methods: Browserless extraction achieved 81% average precision and 97.61% average specificity across 735 sites, with 0.8 seconds average processing time per page.Perfect precision and specificity were reported for 577 of 735 sites.
- Methods based on artificial intelligence: AutoScraper achieved an 88.69% F1-score on SWDE with GPT-4-Turbo, while zero-shot operation outperformed five supervised baselines.The evaluation covered 80 websites across 8 domains, and its unexecutable rate was 4.06%.
3 Build a scraper browserless
The paper develops an adaptive browserless price-extraction system that combines rule-based HTML processing with Bayesian confidence updates and genetic parameter optimisation. Across experiments, the hybrid configuration improves precision, reduces processing time, and maintains stable scalability and memory behaviour.
- 3.1 General structure: The adaptive system processes HTML through browserless retrieval, fragmentation, rule application, and extraction, while an adaptation module updates rule weights and optimises parameters.The system was developed and tested on more than 250 marketplaces and is designed to use fewer resources than full browser simulation.
- 3.3 Bayesian weight update: Bayesian updating assigns each rule an individual confidence coefficient that adapts from observed extraction outcomes instead of treating all rules as equally reliable.Initial counters use Laplace smoothing, starting at one correct and one false discard, corresponding to confidence 0.5.
- 3.3 Bayesian weight update: Weak rules cannot discard fragments independently; several low-confidence rules must fire together, protecting against isolated false positives.The discard decision depends on the accumulated confidence-weighted penalty and the discard threshold.
- 3.5 Experimental results: 98.75% coverage was achieved by all configurations, while precision increased from 77.2% for the baseline to 83.5% with Bayesian training and 87.3% with Bayesian updating plus the Genetic Algorithm.The hybrid configuration also reduced average processing time from 0.620 seconds per page to 0.533 seconds.
- 3.5 Experimental results: The Genetic Algorithm increased the frequency-rule activation minimum from 3 to 9 fragments and lowered the discard threshold from 0.500 to 0.404.These changes reduce erroneous discarding on pages with few price elements and reduce candidates reaching final price selection.
- 3.5 Experimental results: Processing time scales linearly with URL count, while peak memory remains nearly independent of sample size and execution mode through streaming page processing.At 200 URLs, sequential execution used approximately 465 MB peak memory versus 472 MB for parallel-3.
- 3.6 Future work: The evaluation remains limited by an approximately 1,000-record dataset and aggregate analysis that does not distinguish product categories or their page structures.Future category-specific evaluation is proposed to identify strengths and weaknesses across product types.
4 Conclusions
The paper reviews four web price-extraction approaches and presents a browserless system augmented with Bayesian weighting and genetic parameter optimisation. On a common test set, the hybrid configuration improved precision and processing efficiency, while remaining a preliminary validation.
- Practical contribution: The study combines a rule-based browserless fragmentation pipeline with Bayesian rule-weight updates and genetic optimisation of extraction thresholds.The genetic algorithm tunes global parameters after Bayesian calibration of rule weights.
- Theoretical conclusion: No single extraction approach is universally optimal; suitability depends on data volume, computational resources, content dynamism, and website-structure change frequency.The review covers classical, browser-based, browserless, and AI/ML-based methods.
- Evaluation results: 13.1% higher extraction precision and 14% lower average per-page processing time were achieved by the hybrid scheme relative to the baseline configuration.Bayesian weighting alone increased precision by approximately 8.2% relative to the manually tuned baseline.
- Overall implication: The hybrid approach is presented as an alternative to manually tuned browserless extractors and more resource-intensive browser- or LLM-based methods.The authors describe the results as competitive accuracy at low resource cost, based on approximately 200 test records.