Scrapers for Monitoring Competitor Prices: How It Works and What It Costs

A business usually learns that a competitor cut prices from its own client: "theirs is 15% cheaper, will you give a discount?" By that point your managers have been losing deals for a week without understanding why. A competitor price scraper removes this blind spot: every morning at 7:00 you have a fresh snapshot of prices, availability, and promotions across all your key competitors — in a spreadsheet, a dashboard, or a Telegram alert. Let's break down how it's built technically, what it costs, and where scraping projects usually stumble.

Who needs price monitoring and why

Typical situations from our projects where competitor price monitoring delivers measurable impact:

• Online stores. Repricing: keeping your price within a corridor relative to 3–5 competitors across the top 200 SKUs. Losing the price position on a high-runner item means a 20–40% drop in that item's sales before you even notice.
• Manufacturers and distributors. MSRP control: which dealers are dumping and breaking the market. Manually checking 50 dealer sites is a full day of a manager's work; the scraper does it every night.
• Marketplace sellers. Tracking prices and search positions on Rozetka, Prom, Allo: who's entering your niche, with what assortment and prices.
• Service businesses. Monitoring competitors' price lists — clinics, auto service centers, training courses — where prices are openly published on the sites.

The common denominator across all these cases is frequency. A one-off price analysis can be done by hand in a day, and you don't need a scraper for that. The value appears when you need the snapshot daily or weekly: then a one-time build replaces the ongoing manual work that, realistically, nobody does regularly.

How a competitor price scraper works: the architecture

Collecting data from sites is a five-step pipeline:

1. Crawling. On a schedule, the script opens the target pages: catalogs, product cards, price lists. For simple sites, HTTP requests are enough; for sites with dynamic rendering, a headless browser is used.
2. Extraction. The required fields are pulled from the HTML: name, price, old price, availability, SKU, rating. A separate rule set is written for each site.
3. Normalization. "1,299 UAH," "1299.00 UAH," and "from 1299₴" are brought to a single number; sizes, colors, and packaging to a single format.
4. Product matching. The hardest stage: understanding that your "Bosch GSB 13 RE Drill" and the competitor's "BOSCH GSB13RE Professional" are the same product. It works through a combination of SKUs and barcodes where they exist, plus AI matching by name and specs where they don't.
5. Delivering the result. The data lands in a database, and from there into Google Sheets, a BI dashboard, your CRM via API, or a messenger alert on the trigger "competitor dropped the price by more than 5%."

Why matching is half the project

Collecting prices is technically not hard. What's hard is matching catalogs correctly when you have 3,000 SKUs, the competitor has 5,000, names are written differently, and SKUs are given hit or miss. The quality of matching is exactly what separates a working tool from a spreadsheet nobody trusts. On projects we start with the top 100–300 positions that drive the bulk of revenue, and bring matching accuracy to 95%+ before expanding coverage. Then expansion goes in waves of 200–300 positions — with a manual spot-check of each wave.

What else you can collect besides prices

Custom web scraping rarely stops at prices. The same pipeline collects:

• availability and delivery times — a competitor may be cheaper but out of stock;
• promotions, promo codes, and banners — what their marketing activity is made of;
• new catalog positions — where the competitor is expanding their assortment;
• reviews and ratings — a ready source for analyzing a rival product's weak spots;
• competitors' job openings — an indirect but telling signal about growth plans.

What a scraper costs: real numbers

Tier	What's included	Cost
Basic	1–3 sites, up to 500 positions, scheduled export to Google Sheets	from $600
Standard	5–15 sites, product matching, dashboard, Telegram alerts	$1,600–3,000
Extended	15+ sources, anti-bot bypass, API, price history, AI catalog matching	from $4,000

Plus maintenance — $100–400 a month: sites change their markup, and extraction rules have to be updated. This isn't optional, it's part of reality: a scraper without maintenance degrades in 2–4 months. Details and examples are on the scrapers page.

A ready-made monitoring service or a custom scraper

There are subscription price-monitoring services on the market — and for typical e-commerce cases they work fine. The line is drawn like this:

• A ready-made service makes sense when your competitors are large stores and marketplaces the service already covers, and you only need prices for standard categories. You pay $200–600 a month for the subscription — continuously, as long as you use it.
• A custom scraper wins when the sources are non-standard: niche sites, price lists in PDF, regional competitors, B2B portals, specific fields like delivery times or configurations. A one-time build from $600 — and the tool is yours, with no monthly fee per source.

Over a year, a custom scraper for 5–10 sources usually comes out cheaper than a subscription, and more importantly, it collects exactly what you need, not what's in the service's catalog.

Pitfalls

• Anti-bot protection. Cloudflare, captchas, request limits. Solved with proxy rotation, human-like bypass patterns, and a smart schedule — but this goes into the architecture from the start, not "added later."
• Markup changes. The competitor updated the design, the extraction rules broke. That's why a proper scraper has self-health monitoring: if 12 prices come in instead of 500, the owner gets an alert before the business sees a hole in the data.
• Dynamic content. Prices loaded by scripts, regional prices, prices behind a login — each case is solvable but affects complexity and cost.
• The legal side. Collecting publicly available data is normal global practice, the basis of entire price-analytics industries. The red lines: personal data, content behind a login and password, and direct copying of someone else's content for publication. Prices and availability are public market information.

Where to send the data so it actually gets used

The most common cause of a monitoring project's death: data gets collected, but nobody looks at it. Working delivery formats, in order of maturity:

1. Google Sheets with conditional formatting — enough to start: red highlights where you're pricier than the market.
2. Telegram alerts on triggers: a price change over a threshold, a new product appearing, a competitor's position disappearing.
3. A BI dashboard with history: price dynamics over six months, the share of positions where you're in the corridor, a map of dealer dumping.
4. AI analytics on top of the collected data. The next step is a model that doesn't just show data but writes weekly conclusions: where you're systematically losing price position and what it means for revenue. How it's built — on the AI analytics page.

A rule we impose on clients: the data must have one owner — a person who looks at the snapshot weekly and makes pricing decisions. The scraper supplies facts, but the decision on repricing, a promotion, or a talk with a dumping dealer is made by a person with authority.

A typical project by weeks

In practice, a price-monitoring scraper launches quickly: week 1 — we lock the list of competitor sites and priority positions and check the technical accessibility of the sources; week 2 — we write the collection and extraction, run the first full cycles, and verify accuracy by hand; week 3 — catalog matching, setting up data delivery and alerts, handover. Basic configurations for 1–3 sites we deliver in 5–7 working days.

Want to understand exactly what can be collected on your competitors and what it would cost? Book a free 30-minute AI audit — we'll look at your sources, assess the complexity, and give you an honest budget range: maxicolabs.com/contact.