Competitor Price Tracking: Building Offers, Sellers and Price History From Retailers' Own Pages

Competitor price tracking is the repeated collection of competitors' publicly displayed prices for the same products, stored as a time series so you can see who is cheaper today, when each price changed and how deep and how long each promotion ran. A single price check tells you where you stand. A tracked history tells you how your market behaves.

This post builds the core of a tracker: what to track, how to collect it responsibly, a data model that keeps history intact, change detection that turns observations into events, and queries that answer the questions a pricing team actually asks. All of the code below was run against sample data.

What the dataset needs

Every competitor price tracker, however large, comes down to three things:

• Offers: one seller's listing for one product, identified by GTIN.
• Sellers: the retailers making those offers.
• Observations: repeated readings of each offer's price, original price, delivery charge and availability over time.

Get these right and everything else (dashboards, alerts, repricing rules) sits on top.

Step 1: decide what to track

Scope is where most trackers go wrong, usually by trying to track everything. Settle three lists before writing any code:

• Products: start with the lines that drive revenue or traffic, with a GTIN for each. GTINs let you match the same product across retailers that name it differently.
• Competitors: the retailers your customers genuinely compare you with, not every site that stocks the product.
• Cadence: daily suits most categories; fast-moving ones may need several readings a day. Match it to how quickly you can act.

Step 2: collect responsibly from retailers' own pages

Each retailer's product page is the original source of its price. Collect from public product pages only, identify your crawler honestly, keep request rates low, respect each site's terms, and check robots.txt before every fetch:

from urllib import robotparser
from urllib.parse import urlsplit

_parsers: dict[str, robotparser.RobotFileParser] = {}
USER_AGENT = "acme-price-monitor/1.0 (+https://example.com/bot)"

def allowed(url: str) -> bool:
    """Check a retailer's robots.txt before fetching one of its product pages."""
    parts = urlsplit(url)
    root = f"{parts.scheme}://{parts.netloc}"
    rp = _parsers.get(root)
    if rp is None:
        rp = robotparser.RobotFileParser(root + "/robots.txt")
        rp.read()
        _parsers[root] = rp
    return rp.can_fetch(USER_AGENT, url)

f a path is disallowed, skip it and log it. A tracker that depends on ignoring a site owner's instructions is not one you want under pricing decisions. For a fuller checklist of polite crawling habits, see these web scraping best practices.

Step 3: a data model that keeps history

Three tables cover offers, sellers and observations. The examples use SQLite so you can run them locally; the same shape works in any SQL warehouse.

CREATE TABLE sellers (
  seller_id   INTEGER PRIMARY KEY,
  name        TEXT NOT NULL,          -- storefront name as displayed
  domain      TEXT NOT NULL UNIQUE
);

CREATE TABLE offers (
  offer_id    INTEGER PRIMARY KEY,
  gtin        TEXT NOT NULL,          -- the join key across sellers
  seller_id   INTEGER NOT NULL REFERENCES sellers(seller_id),
  url         TEXT NOT NULL UNIQUE,
  first_seen  TEXT NOT NULL
);

CREATE TABLE price_observations (
  offer_id     INTEGER NOT NULL REFERENCES offers(offer_id),
  observed_at  TEXT NOT NULL,         -- UTC, 'YYYY-MM-DD HH:MM:SS'
  price        NUMERIC NOT NULL,
  list_price   NUMERIC,               -- original price, if a reduction is shown
  shipping     NUMERIC,
  currency     TEXT NOT NULL,
  availability TEXT NOT NULL,
  PRIMARY KEY (offer_id, observed_at)
);

Two rules matter. price_observations is append-only: never update a price in place, or you destroy the history that makes the data valuable. And GTIN is the join key: titles vary between retailers, GTINs do not.

Step 4: turn observations into events

Raw observations are hard to act on. A pricing team wants to know what changed: a competitor dropped a price, started a promotion or ran out of stock. Compare each new observation with the previous one for the same offer:

from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class Observation:
    offer_id: int
    price: Decimal
    list_price: Decimal | None
    availability: str          # in_stock, out_of_stock, ...

def detect_changes(prev: Observation | None, curr: Observation,
                   min_move: Decimal = Decimal("0.01")) -> list[str]:
    """Compare two observations of the same offer and name what changed."""
    if prev is None:
        return ["new_offer"]
    events = []
    move = (curr.price - prev.price) / prev.price
    if move <= -min_move:
        events.append(f"price_drop:{move:.1%}")
    elif move >= min_move:
        events.append(f"price_rise:{move:.1%}")
    if prev.list_price is None and curr.list_price is not None:
        events.append("promotion_started")
    elif prev.list_price is not None and curr.list_price is None:
        events.append("promotion_ended")
    if prev.availability != "in_stock" and curr.availability == "in_stock":
        events.append("back_in_stock")
    elif prev.availability == "in_stock" and curr.availability != "in_stock":
        events.append("went_out_of_stock")
    return events

A drop from 219.00 to 199.00 with a struck-through 229.00 returns ['price_drop:-9.1%', 'promotion_started']. A move of 50p on a 219.00 item returns nothing, because min_move filters out noise. Tune it per category, then route events to alerts, and keep the observations themselves for analysis.

Step 5: queries that answer real questions

A view for the latest observation per offer, including the delivered price shoppers actually pay:

CREATE VIEW latest_prices AS
SELECT o.gtin, s.name AS seller, p.price, p.shipping,
       p.price + COALESCE(p.shipping, 0) AS delivered_price,
       p.currency, p.availability, p.observed_at
FROM price_observations p
JOIN offers o  ON o.offer_id = p.offer_id
JOIN sellers s ON s.seller_id = o.seller_id
WHERE p.observed_at = (SELECT MAX(observed_at)
                       FROM price_observations
                       WHERE offer_id = p.offer_id);

The market spread for each product, counting in-stock offers only:

SELECT gtin,
       COUNT(*)                        AS sellers,
       MIN(delivered_price)            AS cheapest,
       MAX(delivered_price)            AS dearest,
       ROUND(AVG(delivered_price), 2)  AS average
FROM latest_prices
WHERE availability = 'in_stock'
GROUP BY gtin;

The lowest price per offer over the previous 30 days:

SELECT o.gtin, s.name AS seller, MIN(p.price) AS low_30d
FROM price_observations p
JOIN offers o  ON o.offer_id = p.offer_id
JOIN sellers s ON s.seller_id = o.seller_id
WHERE p.observed_at >= datetime('now', '-30 days')
GROUP BY o.offer_id;

That last one is more useful than it looks. A "40% off" label means little until you know what the product sold for over the previous month, and in the EU, retailers announcing a price reduction must show the lowest price applied in at least the 30 days before it. A 30-day low lets you read competitors' promotions accurately and check your own.

Practical notes

1. Compare delivered prices. A cheaper shelf price plus a delivery charge is often not cheaper.
2. Filter on availability. An out-of-stock offer at a low price is not competition today.
3. Store currency on every row. Never infer it from a domain.
4. Keep raw page captures for a sample of observations, so disputed figures can be checked.
5. Watch the run, not just the rows. A sudden fall in observations from one retailer usually means its site changed, not that it stopped selling.

Wrapping up

Competitor price tracking is a time-series problem as much as a collection problem. Scope it to the products and competitors that matter, collect from retailers' own public pages within their rules, store every observation in an append-only, GTIN-keyed model, and turn changes into events your pricing team can act on. Build that core well and the dashboards and repricing rules on top become the easy part.

FAQ

What is competitor price tracking?

Competitor price tracking is the repeated collection of competitors' publicly displayed prices for the same products, stored over time so a business can compare price positions, detect price changes and promotions, and understand how its market behaves.

How often should you track competitor prices?

Daily collection suits most retail categories. Fast-moving categories may need several readings a day, while slower categories and strategic reviews can work with weekly data. Match the cadence to how often prices change and how quickly you can act.

How should competitor price history be stored?

In an append-only table with one row per offer per observation, holding price, any original price, delivery charge, currency, availability and a UTC timestamp, linked to products by GTIN so the same item can be compared across retailers.