Scraping infrastructure with 50+ Python/PHP parsers, BeautifulSoup and Selenium, MySQL storage, cron scheduling, and proxy management.
The infrastructure is built on more than 50 parsers developed in Python (BeautifulSoup for static HTML) and PHP. Selenium comes in for sites that load their prices dynamically in JavaScript. Each parser is specialized for a target site and handles the specifics of its structure: pagination, lazy loading, product variants.
Proxy management is the invisible but crucial part of the system. Sending hundreds of requests from the same IP triggers anti-bot protections. A pool of rotating proxies spreads the requests out, and an automatic detection system identifies blocked proxies and replaces them without interruption.
Cron jobs orchestrate the collection cycles at frequencies suited to each site (some change their prices daily, others weekly). MySQL stores the price history with timestamps for trend tracking. Cleaning scripts handle false positives and formatting anomalies.
50+ Python parsers (BeautifulSoup) for extracting structured data.
Selenium for sites with dynamic JavaScript rendering.
Additional PHP parsers run as cronjobs.
MySQL storage for products, prices, history and alerts.
Proxy management and rotation to avoid blocks.
100+ parser files across the Python and PHP implementations.
BeautifulSoup + Selenium
Scheduled execution
Price and stock history
A brief, a repo, or a quick call - whatever works.