Dev.to WebDev 🛠 Dev 👁 0 📖 6 min read

Why Your Image Scraper Gets 403 on Images That Load Fine in the Browser

Here is a failure mode that wastes more engineering hours than it should. You open a product page, watch the gallery thumbnails render perfectly, then run your download loop and get a folder full of 403 Forbidden respons

Why Your Image Scraper Gets 403 on Images That Load Fine in the Browser

Here is a failure mode that wastes more engineering hours than it should. You open a product page, watch the gallery thumbnails render perfectly, then run your download loop and get a folder full of 403 Forbidden responses, or worse, a set of 32×32 grey placeholders. Nothing is broken. The server is answering honestly. Your request simply is not the request the browser made.

Two mechanisms cause most of it: the Referer header, and signed CDN URLs. Both are easy to bypass by accident when testing manually, and both are brutal when you scale to a few thousand images.

Mechanism one: the server wants to know where you came from

Hotlink protection is the oldest anti-leech trick on the web, and it is still everywhere. The rule is usually something like: serve the image only if Referer starts with one of the site's own hosts. No header, wrong host, or an empty value — respond 403, or respond 200 with a placeholder.

The Referer header is not something a client can invent. Browsers derive it from the document that initiated the request and then apply a Referrer-Policy to decide how much of it survives:

Policy Sent across origins
no-referrer nothing at all
origin origin only, e.g. https://shop.example/
strict-origin-when-cross-origin origin only, and nothing on HTTPS→HTTP downgrade
unsafe-url the full path including query string

Modern browsers default to strict-origin-when-cross-origin, which means a cross-origin image request normally carries only the origin, not the path. Sites with stricter settings send nothing, and a no-referrer page will produce image requests with an empty Referer — which is exactly the request that hotlink rules reject.

This has a direct consequence for anyone calling fetch from a browser extension. In fetch, the referrer option can only shorten what the browser would send; it cannot lengthen it, and it cannot be set to an arbitrary foreign origin. There is no supported way to fabricate someone else's referrer. So if your downloader runs in the extension's service worker, the request carries chrome-extension://<id> as its origin, the server sees a stranger, and you get a wall of 403s.

The fix is architectural, not a header hack: run the request in the page's own context. A content script executes inside the page's origin, so a fetch it issues carries the page's referrer and the page's cookies, which is precisely the request that already worked when the thumbnail rendered. Moving the download path from the background worker into a content script resolves a surprising share of "this image refuses to download" reports. Where the request must stay in the worker — because of CORS, credentials, or size — mirror the page's policy deliberately: read document.referrer, read the document's referrer-policy meta or header, and send the narrowest value that still satisfies the host's whitelist rather than guessing.

Mechanism two: the URL was a temporary ticket

The second mechanism looks like nothing at all, because the URL in the DOM is usually valid at the moment you read it:

https://cdn.example.com/photos/hero.jpg?OSSAccessKeyId=LTAI...&Expires=1767225600&Signature=abc123%3D

That is an Alibaba Cloud OSS presigned URL: an access key id, an expiry timestamp, and an HMAC signature calculated over the object path plus the expiry. Tencent COS and Qiniu use the same pattern with different parameter names — Qiniu uses e= and token=, CloudFront uses Policy=, Signature= and Key-Pair-Id=. AWS S3 presigned URLs put the signature in X-Amz-Signature with X-Amz-Expires controlling the window.

Once the expiry passes, the signature is dead and the object returns 403 AccessDenied regardless of who you are. And here is the trap: the signature is computed over the exact path and query, so any "helpful" normalisation on your side invalidates it. Strip the query string to get a "clean" URL and you have deleted the ticket. Rewrite the path to a sibling directory and the HMAC no longer matches. Append a transform parameter for a different size and the server hashes something else. The signature is not a decoration you can tidy up — it is the whole authentication.

So the correct handling of a presigned URL is embarrassingly simple and easy to get wrong: use the string you found, byte for byte, and treat it as a short-lived value rather than a durable identifier. Cache the bytes, not the URL. If you must store a reference, store it alongside its expiry so a later re-download can fail fast instead of wasting a request on a dead ticket.

What this means for a crawler

Put the two mechanisms together and the practical rules fall out:

  1. Take URLs from the DOM, never construct them. Read img.currentSrc, srcset candidates, data-src attributes, and the resolved URL of CSS backgrounds. Reconstruction is where signatures die and where lazy-load placeholders get mistaken for originals.
  2. Tag every URL with the context that produced it. Page URL, page-level referrer policy, whether the URL was presigned, whether the request needs cookies. A URL without its context is half a request.
  3. Issue the request from the right realm. Page context for referrer- and cookie-dependent resources; worker context only when you have established the host does not care.
  4. Classify the failure properly. 403 with an XML error body is usually a dead signature; 403 with an HTML body is usually hotlink; 200 with a 1×1 GIF is a tracking pixel or an anti-leech placeholder. Three different bugs that all look like "download failed" in a log.
  5. Fail in a bounded way. A dead signature cannot be re-signed by you. Retry once with the page's original referrer, then mark the asset as unrecoverable instead of burning the queue on it.

None of this is conceptually hard. All of it is detail work that has to survive contact with thirty different CDNs, three image-serving proxies, and a page that renders its gallery through a <picture> element with six srcset candidates.

That is the part OmniPic AI Studio Pro handles for you.

OmniPic AI Studio Pro configuration and engine settings

How OmniPic deals with it

OmniPic AI Studio Pro is a Chrome extension for site-wide image inspection and collection, built around the assumption that the URL you see in the DOM is the only trustworthy one:

  • Site-wide crawl with DOM-accurate discovery — it reads images from <img>, <picture>, srcset/currentSrc, CSS backgrounds, inline data-* attributes and lazy-load payloads, taking the exact string the page used rather than a rebuilt guess.
  • CDN original recovery — it recognises the transform conventions used by Alibaba Cloud OSS, Tencent COS and Qiniu, and walks a thumbnail back to the full-resolution object while keeping any signature attached to the URL.
  • On-device AI reverse image search — MobileNet embeddings (1024 dimensions) accelerated with WebGPU, so "find more like this one" runs in your browser.
  • pHash deduplication — 64-bit perceptual hashes and Hamming-distance matching collapse repeated assets across pages.
  • Batch export — thousands of images packed through a streaming ZIP, or written straight into Eagle or Billfish.
  • 100% local processing — decoding, hashing, scoring and packing happen on your machine. No uploads, no server-side copy.

It runs on Chrome, Edge and Firefox.

If your current workflow involves a Python script, a for loop and a folder of 403 files, the interesting question is not how to get around the block — it is whether you were requesting the right URL from the right context in the first place.

Install OmniPic AI Studio Pro from the Chrome Web Store

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.