PaperlayerView exact revision on arXiv

Source-linked AI summary

Web Price Extraction: State of the Art and an Adaptive Browserless Implementation

Evgeniia Kositsyna, Jorge Lloret-Gazo

arXiv:2609.01030v1cs.IRcs.LGcs.NE

TL;DR

Price extraction must remain accurate across structurally varied websites, a practical challenge for e-commerce monitoring. This paper combines Bayesian rule-weight updates with genetic-algorithm tuning in a browserless system, increasing precision by approximately 13.1% and reducing average per-page processing time by approximately 14% versus baseline.

  • Problem

    Price extraction methods need to preserve accuracy across structurally variable websites, a practical challenge for monitoring hundreds of marketplaces.

    Source: §2.7 · p. 27

  • Method

    The system combines a rule-based fragmentation pipeline with Bayesian rule-weight updates and a genetic algorithm for tuning extraction thresholds.

    Source: §1 · p. 3, §4 · p. 45

  • Results

    Approximately 13.1% relative precision increase and 14% lower average per-page processing time were reported for the combined Genetic Algorithm and Bayesian weighting scheme versus baseline.

    Source: §4 · p. 46

  • Takeaways & Limitations

    The hybrid approach is presented as an alternative to manually tuned browserless extractors and more resource-intensive browser- or LLM-based methods, with competitive accuracy at low resource cost.

    Source: §4 · p. 46

    The test sample contained approximately 200 records, and the authors identify expanding it as a direction for future work.

    Source: §3.6 · p. 43

Abstract

from arXiv · show

Price extraction from websites is a key task for market monitoring, price comparison, and business analytics in e-commerce. Existing approaches can be broadly divided into four groups, and understanding their trade-offs in accuracy and scalability is essential for selecting suitable extraction strategies. Classical methods rely on manually written wrappers and rule induction from labeled pages, offering high accuracy but adapting poorly to structural changes and requiring considerable maintenance effort. Browser-based methods, using tools such as Selenium and Puppeteer, handle dynamic JavaScript content but consume large computational resources and scale poorly. Browserless approaches retrieve HTML directly via HTTP requests, offering significant gains in speed and cost, but rely on rules calibrated for specific sites. Methods based on machine learning and large language models offer adaptability but require training data and substantial computation. Our main contribution is an adaptive browserless price extraction system that improves robustness to structural differences between websites. We implemented a baseline architecture combining HTML page fragmentation with syntactic, semantic, and frequency rules, and extended it in two ways: a Bayesian approach that dynamically updates rule weights, and a genetic algorithm that optimizes the system's global parameters. This hybrid scheme increased precision from 77.2% to 87.3% and reduced average per-page processing time by approximately 14% relative to the baseline, confirming it as a competitive alternative to manually tuned browserless solutions and to more resource-intensive browser- or LLM-based methods, offering high extraction accuracy at low computational cost.

1 Introduction

The paper compares web price-extraction approaches and develops a browserless system intended to balance adaptive extraction with low resource use. It examines a rule-based baseline, Bayesian rule-weight updates, and genetic-algorithm parameter optimization.

1 Introduction

  • Automated competitor-price monitoring supports real-time pricing decisions, and algorithm adoption raised margins in the German gas-station study.The competitor-monitoring market was estimated at USD 1.2 billion in 2024 and projected to reach USD 2.5 billion by 2033; in the study, margins rose 9% in non-monopoly markets and 28% when both duopolists adopted algorithmic pricing.Source: §1 · p. 2
  • The work compares classical, browser-based, browserless, and AI-driven methods, focusing on whether lightweight browserless systems can compete with machine learning at lower computational and implementation cost.Reliable extraction is challenged by changing HTML, JavaScript-generated content, and anti-scraping mechanisms.Source: §1 · pp. 2–3
  • Browserless tools reduce resource use and improve speed for large-scale extraction, whereas browser-based tools handle JavaScript content but can be resource-intensive and scale poorly.Browserless methods may struggle with highly dynamic sites or client-side rendering; AI and machine-learning approaches can adapt to changing structures but raise challenges involving training, data availability, computation, and generalization.Source: §1 · p. 3
  • The Python browserless baseline extracts prices using syntactic and semantic rules.Traditional site-specific wrappers can be precise but require prior HTML analysis and are poorly adaptable to layout changes, with high maintenance costs and manual effort.Source: §1 · pp. 2–3
  • A Bayesian extension calibrates each rule’s confidence from observed extraction outcomes.This mechanism individually updates rule weights.Source: §1 · p. 3
  • A genetic algorithm further tunes the fragment-discard threshold and minimum fragment count used by the frequency rule.Source: §1 · p. 3

2 State of the art

This section reviews web-extraction concepts and implementation families, comparing browser-based, browserless, and AI/ML methods. It examines their trade-offs in handling dynamic content and structural variation, robustness and accuracy, and computational and resource costs.

2.1 Definitions and basic concepts

  • Web information extraction identifies target items, while web data extraction retrieves structured records; scraping must also determine whether values represent prices, weights, ratings, or quantities.HTML is semi-structured, and AJAX-loaded content is absent from the initial HTTP response until JavaScript injects it into the DOM.Source: §2.1 · pp. 4–5

2.2 General classification of approaches

  • The four approaches are classical rules, browser simulation, browserless HTTP wrappers, and AI/ML; accurate ML extraction requires many labeled pages with positive and negative examples.The review notes that each approach has distinct trade-offs and none is universal.Source: §2.2 · pp. 6–8

2.3 Classical methods

  • F-measure reached 100% for Nokia 1200 and Canon G10, while extraction accuracy was 93.81% on www.dienthoaididong.com.vn and 58.5% on www.trananh.vn.The lower result reflects XPath patterns’ fragility when a site uses different HTML templates across product categories.Source: §2.3 · p. 12

2.4 Methods that simulate browsing

  • Browser simulation handles JavaScript- and AJAX-generated content and simplifies wrapper creation, but rendering takes approximately 85% of execution time and 96.8% of visual-wrapper traffic is irrelevant.Browser initialization accounts for about 13% of execution time and extraction 2%; initialization and rendering together take roughly 50 times longer than data retrieval.Source: §2.4 · pp. 12–13
  • In a 24-hour test, Selenium achieved 100% uptime while Playwright achieved 99.72%, and comparisons across tools showed no simultaneous winner in speed, CPU use, and memory.Playwright used the least CPU in most scenarios, while Selenium consistently used the least RAM; the passage cautions that tool choice should depend on the resource or stability priority.Source: §2.4 · p. 16
  • Browser-simulation tools face active blocking: a Selenium system with anti-blocking modules avoided blocking during its test, while browser fingerprinting can persist despite IP and User-Agent rotation.Without the modules, blocking began by the fifth iteration; with them, average iteration time rose from 5.479 to 6.862 seconds. Fingerprints changed within a day in 45.2% of cases, yet the hybrid method linked fingerprints for an average of 54.48 days.Source: §2.4 · pp. 16–17

2.5 Browserless methods

  • Browserless methods retrieve raw HTML or API responses directly, offering performance, low overhead, and scalability, but struggle with JavaScript-generated content and anti-bot defenses.Hand-writing an HTTP wrapper can take experienced developers several days for a single site.Source: §2.5 · pp. 17–18
  • A multiagent system distributes scraping among specialized agents and supports parallel, scalable collection, but bypassing blocks added about 25% to average iteration time in one study.The architecture includes collection, processing and coordination, and storage and access layers; agent roles include crawling, parsing, data handling, and anti-blocking.Source: §2.5 · pp. 18–19
  • Bayesian rule weighting can update with observations, while genetic algorithms evolve candidate regexes; examples support combining these approaches to adapt extraction rules.For phishing detection, Bayesian thresholding improved HSBC CCR from 99.22% to 99.70% and reduced missed pages from 34/226 to 9/226; genetic algorithms also performed well in image extraction and neural-network tuning.Source: §2.5 · pp. 19–20
  • 81% average precision and 97.61% average specificity were reported across 735 sites for a browserless price-extraction architecture, with 100% precision and specificity on 577 sites.The architecture also achieved 100% specificity on 630 sites, and its 735-page experiment took 10 minutes, or 0.8 seconds per page on average.Source: §2.5 · p. 21

2.6 Methods based on artificial intelligence

  • AI/ML methods learn extraction patterns from labeled examples and can adapt to structural changes, but require substantial training data and computation and can be complex and hard to interpret.One LSTM-based system reported 99% failure-prediction accuracy and 96.9% information-extraction accuracy after page-structure changes.Source: §2.6 · pp. 22–23
  • AutoScraper [37] generates XPath extraction scripts through top-down path refinement, bounded step-back retries, and synthesis that selects the sequence performing best across randomly chosen seed pages.The retry limit is dmax, and synthesis evaluates sequences from ns seed webpages.Source: §2.6 · p. 23
  • 88.69% F1 was achieved by zero-shot AutoScraper across 80 sites and 8 domains, outperforming all five supervised baselines [37].Its unexecutable rate was 4.06%, compared with 14.37% for COT; the benchmark compares supervised baselines with AutoScraper.Source: Table 3 · p. 24, §2.6 · p. 24
  • As seed pages increase from 1 to 5, correct extraction rises from approximately 65% to 73% for GPT-4-Turbo and 44% to 57% for GPT-3.5-Turbo, with diminishing gains.Unexecutable rates also decrease as seed counts rise.Source: §2.6 · p. 24
  • WebArena’s best GPT-4 agent succeeds on 14.41% of end-to-end tasks versus 78.24% for humans, while Mind2Web step success falls from 52.0% cross-task to 38.9% cross-website and 39.6% cross-domain.Even in Mind2Web’s simplest scenario, only 5.2% of tasks were completed without any step error.Source: §2.6 · pp. 24–25
  • WebVoyager achieves 59.1% task success using screenshots and text, versus 30.8% for text-based GPT-4 and 40.1% for its own text-only variant.The benchmark covered real-world tasks on 15 popular websites.Source: §2.6 · p. 25
  • VIPS segments pages using visual cues and achieved 97% overall accuracy on 140 pages, but narrows the search space rather than identifying prices or other target values.Its three-stage process extracts visual blocks, identifies weighted separators, and merges blocks until the coherence threshold is reached.Source: §2.6 · pp. 25–26

2.7 Comparative analysis of all instruments

  • Classical accuracy drops from 100% to 58.5% on heterogeneous data, while AI retains 88.69% F1 across 8 domains and browserless combines 81% accuracy with 0.8 s/page.For monitoring hundreds of heterogeneous marketplaces under limited resources, the review selects browserless extraction; dynamic JavaScript content and anti-bot defenses remain limitations.Source: Table 4 · p. 26, §2.7 · pp. 28–29
  • Structural variability is the review’s comparison metric: classical rules falter on changing layouts, browser simulation handles dynamic content but incurs rendering overhead, browserless is fast but misses client-rendered data, and AI generalizes without manual rules but costs more [17].Ringer completed 25 of 34 sites versus 6 for CoScripter, while rendering consumed around 85% of its execution time; browserless averaged 81% precision and 0.8 seconds per page across 735 sites. AI’s efficiency is weaker for small collections and under budget constraints [37].Source: §2.7 · pp. 27–28

3 Build a scraper browserless

This section presents an adaptive browserless price-extraction system that analyzes HTML and combines Bayesian rule-weight updating with genetic optimization. It describes the system’s reported precision, coverage, and processing-time outcomes, along with parameter tuning, behavior under load, and planned extensions.

3 Build a scraper browserless

  • The system extends browserless price extraction with Bayesian rule-weight updating and genetic optimization to improve robustness to marketplace and website-structure differences.It was developed and tested on more than 250 marketplaces; the browserless approach uses fewer computational resources than full browser simulation.Source: §3 · p. 29

3.1 General structure

  • The baseline pipeline identifies price-bearing HTML fragments, applies syntactic, semantic, and frequency rules, and selects a surviving price candidate through staged heuristics.Rules accumulate penalties; the system tries a unique surviving candidate, then semantic current-price markers and parent classes, and finally the most frequent remaining value.Source: §3.1 · pp. 30–32

3.2 Implementation tools

  • The implementation uses Python, requests, BeautifulSoup, and pandas, with the Bayesian and genetic modules built from standard language features rather than external machine-learning frameworks.Requests retrieves HTML without JavaScript rendering, while BeautifulSoup parses it and locates fragments around price clues.Source: §3.2 · p. 33

3.3 Bayesian weight update

  • Bayesian training assigns each rule an individual confidence that changes according to correct and false discards, replacing the classical method’s fixed weight of one for every rule.Counters start at one apiece, yielding confidence 0.5 through Laplace smoothing; weak rules cannot discard a fragment alone, helping guard against isolated false positives.Source: §3.3 · pp. 33–35

3.4 Parameter optimisation with a genetic algorithm

  • The genetic algorithm tunes the minimum fragment count for activating frequency rules and the discard threshold to maximize extraction precision on the training set.These global parameters affect the precision–recall balance; elitism preserves the best chromosome, so the best fitness does not decrease across generations.Source: §3.4 · pp. 37, 40
  • The genetic algorithm tunes global parameters after Bayesian training, using the already calibrated, dataset-specific rule weights.The final module combines those Bayesian weights with the optimized parameter values before test-set evaluation.Source: §3.4 · p. 40

3.5 Experimental results

  • 87.3% precision at 98.75% coverage and 0.533 s/page outperformed Bayesian-only (83.5%, 0.546 s) and baseline (77.2%, 0.620 s).Coverage was 98.75% for all three configurations; the combined system reduced average per-page time by approximately 14% relative to baseline.Source: §3.5 · p. 41, Table 10 · p. 41
  • The genetic algorithm raised the frequency-rule activation minimum from 3 to 9 fragments and lowered the discard threshold from 0.500 to 0.404.The higher minimum prevents frequency rules from firing on pages with few candidates; the lower threshold discards fragments with smaller accumulated penalties, reducing candidates reaching final selection.Source: §3.5 · pp. 41–42, Table 11 · p. 41
  • At N=200, sequential, parallel-2, and parallel-3 processing took approximately 279, 284, and 307 s, respectively, with near-linear time growth.The close timings indicate extraction’s CPU stage, rather than network waits, primarily determined runtime; linear scaling made processing time predictable as URL volume increased.Source: §3.5 · p. 42, Figure 6 · p. 43
  • Peak memory depended weakly on URL count and execution mode; at N=200, parallel-3 peaked at 472 MB versus 465 MB for sequential.Streaming page processing avoids accumulating the whole batch in memory; the small mode differences were attributed to per-worker buffers.Source: §3.5 · p. 42, Figure 7 · p. 44

3.6 Future work

  • The current evaluation uses about 200 test records from approximately 1,000 records; future work plans a test sample of approximately 2,000 and category-specific evaluation.The aggregate evaluation does not distinguish product types; the proposed categories include electronics, clothing, books, and food, with separate precision, coverage, and time measures.Source: §3.6 · pp. 43–44
  • Future category-specific Bayesian weights could address the current system’s shared weights, which average rule reliability across different page types.Separate weights by category are proposed to improve accuracy on heterogeneous content and clarify which rules drive decisions within each category.Source: §3.6 · pp. 44–45
  • An online Bayesian update is proposed to adjust rule posteriors from newly processed pages with confirmed labels, adapting to changing website structures without full retraining.In the current implementation, training-stage Bayesian weights remain fixed during exploitation.Source: §3.6 · p. 45

4 Conclusions

The paper reviews four approaches to web price extraction and presents a browserless system combining rule-based fragmentation with Bayesian rule-weight updates and genetic parameter tuning. Its evaluation reports improved precision and processing time, while the conclusions emphasize that method choice depends on task conditions and that the results remain preliminary.

4 Conclusions

  • The practical system combines a rule-based fragmentation pipeline with Bayesian updates of rule weights and a genetic algorithm for tuning extraction thresholds.The paper also reviews classical, browser-based, browserless, and AI/ML-based approaches to web price extraction.Source: §4 · p. 45
  • No single extraction approach is universally optimal; the choice depends on data volume, available computational resources, content dynamism, and how often website structures change.The review concludes that method selection should follow task requirements.Source: §4 · p. 45
  • Bayesian weighting alone raised precision approximately 8.2%, while the combined Genetic Algorithm and Bayesian scheme raised it approximately 13.1% and cut average per-page processing time approximately 14%, versus the manually tuned baseline configuration.The three configurations were compared on the same test set.Source: §4 · p. 46
  • Processing time scaled linearly with URL count, while peak memory stayed within a narrow range across sample sizes and execution modes.The paper describes this behavior as suitable for predictable batch processing.Source: §4 · p. 46
  • The results from approximately 200 records are preliminary; proposed next steps include a doubled test sample, per-category evaluation and weights, online weight updates, and comparisons with alternative parsers.The implementation, experimental scripts, and dataset are publicly available.Source: §4 · p. 46

Related papers

Loading 2609.01030v1…