Retry Is Not Resume: Building a Web-to-CSV Pipeline That Survives Interruptions

The first run is the easy part.

The real test starts after record 70 of 100 succeeds, the process stops, and you
run it again. If the pipeline fetches the first 70 pages again, appends duplicate
rows, or trusts a half-written state file, it is not resumable. It is only
restartable.

While building a small Python + Playwright source kit, I separated retry from
resume and gave the output data, progress state, and run evidence different
responsibilities. This post walks through the decisions that made the biggest
difference.

Retry and resume solve different failures

A retry handles a temporary failure inside one run:

• a request times out;
• a page returns incomplete markup;
• a required field is temporarily missing.

Resume handles a different event: the run itself ends and a later process must
continue from durable evidence.

Putting both concerns inside one retry loop hides the distinction. Increasing
the retry count does nothing after a terminal closes or a machine restarts.
Instead, I used an item-level retry loop and wrote a completed-record map only
after parsing succeeded.

The essential flow is small:

completed = read_state(config_fingerprint)

for item in input_rows:
    if item.record_id in completed:
        log_skip(item.record_id, reason="completed")
        continue

    parsed = retry(lambda: parse(fetch(item.url)), max_retries=2)
    write_result(item.record_id, item.url, parsed)
    completed[item.record_id] = hash_output(parsed)

write_state_atomically(completed, config_fingerprint)

Retry decides whether the current item gets another attempt. Resume decides
whether the item should be fetched at all in a future run.

The CSV is not the progress database

It is tempting to inspect the last CSV row and treat it as a cursor. That works
until input order changes, a blank line appears, or the output schema changes.
The data file then has two jobs: representing results and controlling execution.

I kept them separate:

• normalized.csv contains the current result rows;
• state.json stores completed record IDs and output hashes;
• run_log.jsonl records succeeded, failed, retrying, and skipped events;
• report.html gives a human-readable result.

The input contract requires record_id,url. A seen set catches duplicate IDs
inside the current input. The durable completed map catches IDs that succeeded
in an earlier run.

This also makes a repeated run understandable. A row is not silently absent; the
log says whether it was skipped because it was a duplicate or because it had
already completed.

Old state must not silently match a new configuration

Resume state becomes dangerous when extraction rules change.

Suppose the first run extracted title and price. Later, the configuration
adds availability. If the old completed IDs are accepted without question,
the pipeline skips pages that have never been processed with the new rule.

The kit hashes a canonical representation of the field configuration:

fingerprint = sha256(
    json.dumps(
        canonical_fields,
        sort_keys=True,
        separators=(",", ":"),
    ).encode("utf-8")
).hexdigest()

That fingerprint is written beside the completed map. When a later run points
to the same output directory with a different configuration, the pipeline
raises an explicit error instead of combining incompatible results.

The fix is intentionally boring: use a new output directory for a new
extraction contract. Quietly guessing would be more convenient and less safe.

Atomic writes protect the mechanism that enables recovery

A state file is useful only if it survives the interruption it is supposed to
help recover from.

Writing directly to state.json can leave truncated JSON if the process stops
at the wrong moment. The next run then fails before it can resume. The same risk
applies to the normalized CSV.

The pipeline writes a complete temporary file and replaces the destination:

temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(value, encoding="utf-8")
temporary.replace(path)

The implementation uses the same pattern for JSON, CSV, and the append-style
run log. It does not make every filesystem failure impossible, but it removes
the common partial-overwrite window.

Evidence should be useful without leaking page content

Debug logs often become accidental data stores. Dumping raw HTML, full query
strings, or local paths would make troubleshooting easier and distribution less
safe.

The public log records the record ID, status, attempt number, and exception type.
It does not record page HTML or local filesystem paths. The HTML report escapes
output values before rendering them.

That boundary matters for a reusable source kit: the person running it controls
the permitted target pages, while the library keeps operational evidence
focused on the run.

I tested the extracted delivery archive, not only the development folder

A passing development environment is not proof that a buyer's ZIP contains the
same working pieces.

I extracted the delivery archive into a clean directory and ran its included
test suite. The result was 9/9 tests passed. Then I ran the offline fixture
demo twice:

• 2 records succeeded;
• 0 records failed;
• 1 duplicate ID was skipped safely;
• the next run read the completed state instead of fetching finished records again.

The fixture demo uses local HTML and requires no browser, account, proxy, cloud
service, or paid API.

Where Playwright fits

The state, retry, parsing, reporting, and fixture paths do not import Playwright.
The browser dependency is loaded only for permitted http:// or https://
pages.

That keeps the offline demo deterministic and lets the same pipeline accept a
real browser fetcher when JavaScript-rendered pages are in scope.

This is not a universal scraper. Selectors are deliberately simple, and the
operator must define them for pages they own or are allowed to automate. The kit
does not bypass login, CAPTCHA, access controls, rate limits, robots rules, or
website terms.

The finished source kit

I packaged the implementation as the
Resumable Web-to-CSV Pipeline Kit.
It includes the full Python source, tests, configuration examples, offline
fixtures, and run documentation.

• Personal: $24 for one user
• Team: $59 for up to five users in one legal entity

The public
delivery repository
shows the actual product screens and verification evidence before purchase.
Custom selectors, custom development, remote setup, ongoing support, and future
updates are not included.

The larger lesson is independent of this kit: retries recover an attempt;
durable state recovers a workflow. Treating those as separate responsibilities
turns "run it again" from a gamble into a defined operation.