Your scraper will not throw an error: designing page extraction that fails loudly, one field at a time

A scraper almost never fails the way a program is supposed to fail. It does not raise. It returns a number.

Last week a price list came back as 0. The page had been redesigned: the price was still on it, in a <span> that no longer carried the class the extractor looked for. The extractor found nothing, applied .get_text() to nothing, and the pipeline wrote 0 into a table, where 0 is a legal price. Nobody saw the problem until a quote went out at zero.

That is the failure mode worth engineering against, and it is not the one most scraping guides prepare you for. An exception is a gift: it stops, it names a line, it leaves a hole you notice. The quiet failures — the empty string, the plausible wrong number, the field that came from the wrong element on the same page — are the ones that travel.

This is about extraction that fails loudly, per field, and about the small set of decisions that make a scraper survive the page it reads: where you anchor, what you do when a field is absent, what counts as a change, and how you enumerate a list that is being written while you read it.

Three ways an extraction fails, and only one of them is obvious

Of the three, the loud one is the cheapest and the plausible one is the only genuinely dangerous one. A design that treats all three the same way — "log it and move on" — optimises for the cheapest failure and hides the expensive one.

The concrete shapes of plausible are worth naming, because each needs a different guard:

• A sibling's value. .parent().find("span") returns the first span, and after a layout change the first span is the unit ("per person") instead of the amount.
• A default. The code has price or 0, or capacity or 4, because some page somewhere was missing the field. The default outlives the reason it was added.
• A stale value. The page is a cached copy, or the extraction ran against yesterday's fixture.
• A wrong item. The selector matched the first result rather than the requested one.
• A unit change. 12,000 → 12.000 after a locale switch, parsed as 12.

None of those are visible in an exit code of 0.

Keep the page before you parse it

The single decision that makes every later question answerable costs almost nothing: store the raw response, with the URL, the fetch time, the status code, and a content hash, before any parsing happens.

def fetch(url, session):
    r = session.get(url, timeout=20)
    body = r.text
    rec = {
        "url": url,
        "status": r.status_code,
        "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "sha256": hashlib.sha256(body.encode("utf-8")).hexdigest(),
        "body": body,
    }
    store_raw(rec)          # append-only, content-addressed
    return rec

Two things become possible once this exists, and neither is possible without it.

First, any parse can be replayed. When a field comes back wrong, you do not re-fetch a page that may have changed again; you re-run the extractor against the exact bytes that produced the bad row. That is the difference between a bug you can debug and a bug you can only reproduce by luck.

Second, the hash is a free change signal. Identical bytes mean the page did not change, so the extractor's output should be identical; if the output differs, your parser is non-deterministic (iteration order, locale, clock) and that is a real bug. Different bytes with identical output is the normal case for a page carrying a timestamp or a session token — which is why page-level hashing is a poor alert and per-field hashing is a good one.

Storage is cheap at this scale: a 120 KB page per fetch, a few hundred fetches a day, is nothing against the cost of reconstructing a page you no longer have.

Anchor on the fact, not on the layout

Every field should be extracted by a ladder of anchors, in order, and the ladder should be explicit in the code rather than implied by whichever selector happened to work when you wrote it.

And one anchor that should never appear: the positional path. div > div:nth-child(3) > span:nth-child(2) is not an anchor, it is a snapshot of yesterday's layout. It is also the shape that produces plausible wrong values most reliably, because after a redesign it still matches something.

The same rule applies to generated class names. css-1x2y3z is a build artefact; it changes when the site's bundler changes, which is more often than the design does. Prefer the attribute that describes the fact over the attribute that describes the styling.

A practical check on any selector you are about to commit: if the page's designer changed the visual layout tomorrow without changing the content, would this still match? If the answer is no for a field that matters, the field is one tier too low.

Three tiers per field, and a record of which one fired

The ladder is only useful if the extractor reports which rung it landed on. A field that normally comes from tier 1 and suddenly comes from tier 4 is a page that has started to drift, visible before it breaks.

def extract_field(doc, spec):
    """spec: {"tier1": fn, "tier2": fn, "tier3": fn, "tier4": fn} -> Field"""
    for tier in (1, 2, 3, 4):
        fn = spec.get(f"tier{tier}")
        if not fn:
            continue
        raw = fn(doc)
        if raw is None:
            continue
        if isinstance(raw, list) and len(raw) > 1:
            return Field(raw[0], "ambiguous", tier=tier, candidates=raw)
        return Field(raw, "found", tier=tier)
    return Field(None, "missing", tier=None)

Three details in that small function matter more than they look.

Tier order is a preference, not a fallback chain. Tier 1 should be tried first even when it is slower, because it is the tier whose meaning is stated by the site rather than inferred by you. If tier 1 fires 100% of the time, tiers 2–4 are dead code that will quietly rot; if it fires 60%, you have a working page and a drift problem at the same time, which is the useful thing to know.

Ambiguity is its own outcome. Two candidates is not "pick the first one". It is a field whose meaning on that page is unclear, and the honest handling is to record both and mark the field, not to gamble. In a price or a date, the first of two candidates is a coin flip that a person will read as fact.

missing is not an error and not a value. It is a state, and it should be spelled out — which is the next section.

Absent is a value. Zero is a lie

The most expensive line of code in a scraper is the one that makes a hole look like data:

price = parse_price(node) or 0        # never

or 0 collapses "the page said zero", "the page said nothing", "the parser failed", and "the field was not on this page" into one number. Downstream, a filter sorts by price, a quote totals it, and a person signs it.

The fix is a small typed result, carried all the way to storage:

@dataclass
class Field:
    value: Any
    status: str           # found | missing | ambiguous | malformed | blocked
    tier: int | None = None
    candidates: list | None = None
    note: str = ""        # why it is not found

def parse_price(node):
    if node is None:
        return Field(None, "missing", note="no price element")
    text = norm(node.get_text())            # collapse whitespace, unify spaces
    m = re.fullmatch(r"([\d.,]+)\s*(원|KRW|USD|\$)?", text)
    if not m:
        return Field(None, "malformed", note=text[:40])
    return Field(to_decimal(m.group(1), m.group(2)), "found")

Now 0 means zero, and missing means the page did not say. A row with price.status = "missing" is a row that a person can act on: re-fetch, open the page, ask the client. A row with price = 0 is a row that nobody looks at.

Two rules that keep this honest in practice:

• missing must be visibly missing in whatever the human sees — a dash, not a blank cell and not a zero. Blank cells read as "we didn't get to it"; a typographic dash reads as "the page did not say", which is the truth.
• The status must survive the write. If the database column is price DECIMAL NOT NULL, the status has nowhere to live and will be flattened to 0 by the first INSERT. Either the column is nullable, or there is a companion price_status column. The schema is where this decision is actually made.

Two sources on one page: reconcile, don't pick

Many pages state the same fact twice — once for humans in the markup, once for machines in JSON-LD, itemprop, or a hydration payload. That redundancy is free verification: extract both and compare.

a = jsonld_field(doc, "price")     # typed
b = dom_field(doc, "price")        # rendered
if a and b and a.value != b.value:
    record("disagreement", field="price", typed=a.value, rendered=b.value)

Three outcomes, three different meanings:

The disagreement rate is one of the best single numbers a scraping pipeline can report, because it catches the class of failure that no exception ever will.

Normalise before you hash

Change detection that compares raw strings alerts on everything and is therefore read by nobody. Hash normalised values, per field:

• collapse whitespace, including the non-breaking space that appears in prices and never in your regex;
• unify full-width and half-width characters;
• resolve relative URLs against the page URL before comparing;
• normalise numbers to a canonical decimal with the unit kept separately (12000 + KRW, not "12,000원");
• convert dates to ISO with the page's stated timezone, or record "timezone unknown" explicitly;
• sort lists whose order carries no meaning, and do not sort ones where order is the fact (a menu's course order, a schedule's sequence).

Then one hash per field, and drift is a field-level event: hours changed, price unchanged. That is a sentence a person can act on, unlike "the page changed".

A useful companion metric is tier drift: the share of runs where a field was found below its usual tier. A field normally at tier 1 that spends a week at tier 3 is telling you the site is mid-migration, and the ladder is already carrying the weight. That is the moment to add the tier-2 anchor, not after it goes missing.

Enumerating a list that is being written

The subtle one: reading a paginated collection while the collection changes underneath you. Offset pagination over a list sorted newest-found-first loses and duplicates items, and it loses them silently.

The arithmetic is simple. Page 1 gives items 1–20. Before you request page 2 with offset=20, two new items arrive at the top. Everything shifts down by two, so page 2's window now starts at what used to be item 19: you see 19 and 20 again, and the two oldest items you were about to reach fall off the end of your walk. Reported counts look right; two items were never read, and two were read twice.

Two fixes, in increasing order of correctness:

Key cursor instead of offset. Ask for "items after this one" using a total order the server guarantees, usually (published_at, id):

GET /items?before=<published_at>,<id>&limit=20

The cursor is a position in the ordering, not a count from the top, so insertions at the top cannot shift it. This requires that the sort key is stable and unique — which is why (timestamp, id) is the usual pair: timestamps collide, ids do not.

Deduplicate by identity anyway. Even with a cursor, an item can be edited and reordered while you walk, so keep a set of ids seen in the run and drop repeats. The count of dropped repeats is itself a metric: rising duplicates mean the ordering is not as stable as you assumed.

And a second watermark for updates. If items can be edited rather than just added, "new since last run" misses them, because they are old. That needs a separate pass keyed on updated_at, with its own cursor, and an explicit rule for what happens when a run overlaps the previous one: the later run wins, and the earlier run's rows for the same ids are superseded, not appended.

The general principle: a scrape over a moving list needs a position, not a count. Any API that only offers offsets over a frequently-written collection is telling you it will be approximate — say so in the output rather than pretending otherwise.

When the page pushes back

A 403 or 429 is not a parser failure, and it is not a reason to try harder. It is the other side's answer, and the only correct responses are to slow down, to stop, and to record it as a state.

if r.status_code in (403, 429):
    return Field(None, "blocked", note=f"HTTP {r.status_code}")

Practical rules that kept my own runs out of trouble:

• A per-host budget, not a global one: N requests per minute per host, plus a cap per run. Concurrency multiplies your request rate in ways that a sleep in one worker does not fix.
• Honour Retry-After when it is given, and treat its absence as a reason to back off exponentially rather than to retry immediately.
• Never rotate identity to get past a block. Changing user agent, proxy, or account to keep reading a page that refused you is not engineering, it is defeating a decision the operator made — and it is the thing that turns a missing data point into a banned account or a legal problem. The block is information: the page is not yours to read at this rate.
• Record the refusal, so a gap in the data has a cause in the log: blocked is a state, and a report that says "hours unknown for 3 of 40 shops — rate limited" is honest, while 3 shops with hours = "" are not.

Respect for the crawl budget is also self-interest: a scraper that reads politely can run for years, which is worth far more than the hour it saved by hammering.

Fixtures are the regression suite, one incident at a time

Every time a field breaks, you already have the page — because you stored it. Copy that response into a fixtures directory, name it for the incident, and add one assertion:

fixtures/
  shop-1234-2026-11-09-price-removed.html
  shop-1234-2026-11-09.expected.json

def test_price_removed_still_reports_missing():
    page = load("shop-1234-2026-11-09-price-removed.html")
    out = extract(page, SPEC)
    assert out["price"].status == "missing"
    assert out["name"].value == "…"

Two properties make this suite worth more than its size. It is offline, so it runs in milliseconds and never depends on the live site. And it is documentation: the directory listing is the history of every way this site has surprised you, in the order it happened.

The assertion to write in almost every case is not "the value is X" but "the status is what it should be" — that a removed element produces missing rather than 0, that a duplicated element produces ambiguous rather than the first candidate, that a blocked response produces blocked rather than an exception that crashes the batch.

The numbers that tell you it is rotting

A scraper decays from the day it is written. Four metrics make the decay visible before it costs anything:

All four are cheap to compute from rows you are already storing, and all four are meaningless at the aggregate level: per field and per site, or they say nothing. "94% success" is a number that hides the only field anyone cares about.

What this does not fix

• A page whose meaning changed while its shape stayed. The digits are in the same <span> and now mean the price per month instead of per item. No tier, no hash, no fixture catches that; only reading the page does.
• A source that is wrong about itself. Extracting a stale price accurately is still a wrong price. Extraction quality is not truth.
• Permission. A page you are allowed to read and a page you should read are different questions, and the second one is not technical.
• A site that deliberately changes shape per request. Obfuscated markup, rotating class names, and decoys exist to make extraction expensive. The honest response is a cheaper source — an API, a feed, a phone call — not a longer selector.
• A missing decision about what to do with a gap. The pipeline can report missing perfectly and still have nowhere to put it. Someone has to decide whether a row with a missing field is written, held, or dropped.

Checklist

• The raw response is stored before parsing, content-addressed, with URL, status and fetch time.
• Every field has an explicit anchor ladder: typed data → identity by meaning → label adjacency → normalised text.
• No field is extracted by positional path or generated class name.
• The extractor records which tier fired, per field.
• Ambiguity is a status with the candidates kept, not a coin flip.
• Absent fields are null with a reason, never 0, "", or a default.
• The storage schema can hold the status; nothing forces a hole into a zero.
• Where a page states a fact twice, both copies are extracted and disagreements are recorded.
• Comparison happens on normalised values, hashed per field.
• Lists are walked by key cursor, deduplicated by id, with a separate watermark for updates.
• A 403 or 429 is recorded as blocked; identity is never rotated to defeat it.
• Each incident leaves a fixture, and the assertions are about statuses as much as values.
• Null rate, tier-fire rate, disagreement rate, and page size are reported per field and per site.
• Gaps in the output are visible as gaps to the person reading it.

None of this makes extraction correct. It makes extraction legible: for any row you can say which element the value came from, how sure the rung says you should be, what the page did when it had nothing to say, and when it last changed its mind. That is the difference between a scraper that fails and a scraper that lies, and only one of those is recoverable.

Disclosure: I build small automation tools and write about the parts that break. I also maintain an English guide for an AI assistant app whose invite system I use as a concrete example of these ideas: Pickle invite code — what the app does and how to enter it and the invite link pickle.com/i/RDKXK77G9ZRT. If someone joins through that link I receive in-app usage credit; it costs the person joining nothing. Nothing in this article depends on that product, and every pattern above is generic.