Dev.to WebDev 🛠 Dev Thu, 12 Nov 2026 19:16:22 +0000 👁 0 📖 3 min read

Full-page screenshots of hundreds of URLs, without cookie banners or missing images

Taking one screenshot of a web page is easy. Taking a few hundred, unattended, and getting images you can actually use is not. Point a simple script at a list of competitor home pages and you get three kinds of bad result: a cookie banner covering a third of the page, grey boxes where images should be, and a batch that hangs on the one site that never finishes loading.

We hit all three while building Bulk Website Screenshot, an Apify Actor that takes a list of URLs and returns a clean screenshot of each. This post covers why the problems happen, what the tool does about each one, and how to run it from Python in a few minutes.

Disclosure: I'm Aurelix, and I built Bulk Website Screenshot. It's a paid tool (prices in the Cost section).

Why bulk screenshots go wrong

Cookie banners. Most sites show a consent popup on first visit, and a headless browser is always a first visit. Clicking "Accept all" gives consent on your behalf, sets tracking cookies and sometimes reloads the page.
Missing images. Most sites load images only when they scroll into view. A full-page capture never scrolls, so everything below the first screen stays a placeholder.
One slow site stalls the batch. Ad-heavy pages can take a very long time to finish loading, and a single hung page holds up everything queued behind it.
Cost and noise. Ads and trackers are most of the JavaScript on news sites, and empty ad frames resize mid-capture.

What the tool does about each one

Cookie banners hidden, never clicked. About 60 rules cover OneTrust, Cookiebot, Didomi and the other common consent platforms, plus home-grown banners and sign-up popups. Sticky site headers are kept. Add your own CSS selectors in hideSelectors for chat widgets or rotating banners.
Lazy images loaded. With Full page on, each page is scrolled first so lazy images load, then captured up to a height cap (16,000 px by default).
Per-URL deadlines. Each URL has its own timeout (45 seconds by default) and pages run in parallel. A URL that fails is reported with a reason and not charged; the rest of the batch carries on.
Ads and trackers blocked by default, which makes pages faster and screenshots cleaner.
A change flag. Every row carries image_sha256 and changed against your last run, and Only changed since last run returns just the pages that changed.

Output is a JPEG (default), PNG, WebP or PDF per page with a public link, on desktop, laptop, tablet, mobile or a custom size. It respects robots.txt and never logs in.

Screenshots of a URL list from Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("aurelix/website-screenshot").call(run_input={
    "urls": ["https://www.python.org", "https://react.dev"],
    "device": "mobile",
    "fullPage": True,
    "format": "png",
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["screenshot_url"], item["changed"])

Get a token from Apify Console (Settings > API & Integrations) and pip install apify-client. No code at all also works: paste the URLs into the Actor's form on Apify and click Start.

A weekly competitor page monitor

Save this input as a task in Apify Console, click Schedule, and pick weekly. Each run returns only the pages whose screenshot changed, so the dataset is your change list:

{
  "urls": ["https://www.python.org", "https://www.djangoproject.com"],
  "incremental": true,
  "trackingId": "competitors-weekly",
  "hideSelectors": ["#chat-widget"]
}

Hide anything that changes on every visit (clocks, rotating hero banners, chat bubbles), or it will flag a change every time.

Limits

Many sites show consent banners only to visitors from the EU or UK, so a banner you see at home may not appear in the run at all.
Very long pages are cut at the height cap; the row says so in truncated.
Pages blocked by robots.txt are skipped, reported and not charged.

Cost

At the time of writing it costs $1.00 per 1,000 screen-size screenshots, and $2.00 per 1,000 more for full page, because a full page costs about three times the computing, plus a $0.004 start fee per run. No compute units are billed on top, and URLs that fail are not charged. Apify's free plan includes $5 of credit a month, about 5,000 screen-size screenshots (or about 1,600 full page).