Fetching user-supplied URLs from a Cloudflare Worker without getting burned

AI disclosure: this post was written with AI assistance (drafting, code and editing). The code comes from a project whose unit and integration tests I ran on October 28, 2026.

Any API that takes a URL and fetches it (link previews, metadata extraction, uptime checks, feed readers) is a request-forgery machine unless you put rules around it. Running on Cloudflare Workers removes some risks, since there is no cloud metadata service sitting on localhost, but not all of them. Your Worker can still be pointed at internal hostnames, at origins that only trust Cloudflare IPs, or at a 5 GB file.

These are the checks I ended up with, in the order a request hits them. The snippets are trimmed; bad, blocked and tooLarge are small helpers that build error responses.

1. Validate the URL before you touch the network

if (u.protocol !== 'http:' && u.protocol !== 'https:') throw bad('Only http and https URLs are supported.');
if (u.username || u.password) throw bad('URLs with credentials are not allowed.');
if (u.port && u.port !== '80' && u.port !== '443') throw blocked('Only ports 80 and 443 are allowed.');

Then reject hostnames that can never be public: localhost, single-label names, and suffixes such as .local, .internal, .lan, .home.arpa, .test, .invalid and .onion. Strip a trailing dot first. localhost. is a valid way to write localhost.

2. Resolve the name and check every address

A hostname can point anywhere, so check where it resolves. Workers have no dns.lookup, but DNS over HTTPS works with a plain fetch:

async function doh(name: string, type: 'A' | 'AAAA'): Promise<string[]> {
  const r = await fetch(`https://cloudflare-dns.com/dns-query?name=${encodeURIComponent(name)}&type=${type}`, {
    headers: { accept: 'application/dns-json' },
    signal: AbortSignal.timeout(5000),
  });
  const j: any = await r.json();
  const want = type === 'A' ? 1 : 28;
  return (j.Answer || []).filter((a: any) => a.type === want).map((a: any) => String(a.data));
}

Query A and AAAA in parallel and reject the request if any answer is non-public. "Any", not "the first one": an attacker controls the DNS records and can mix a public and a private address.

The IPv4 blocklist is the usual set: 0/8, 10/8, 100.64/10, 127/8, 169.254/16, 172.16/12, 192.168/16, 192.0.0/24, 192.88.99/24, 198.18/15, the three documentation ranges, multicast and 240/4. IPv6 is where most hand-rolled checks leak, because IPv4 can hide inside it. Decode and re-check the embedded IPv4 address for:

• IPv4-mapped ::ffff:127.0.0.1
• the deprecated IPv4-compatible form ::127.0.0.1
• NAT64 64:ff9b::/96
• 6to4 2002::/16, where the IPv4 address sits in the second and third groups

Then block ::, ::1, fc00::/7, fe80::/10, fec0::/10, ff00::/8, 2001:db8::/32 and 2001::/23. Write unit tests for each form. Parsing :: compression and embedded dotted quads is easy to get subtly wrong.

Treat anything you cannot parse as unsafe.

3. Follow redirects yourself

fetch follows redirects by default, which skips every check above for the second hop. Turn that off and loop:

const seen = new Set<string>([url.href]);
for (let hop = 0; ; hop++) {
  await assertPublicUrl(url);            // steps 1 and 2, on every hop
  const res = await fetch(url, { redirect: 'manual', signal: AbortSignal.timeout(timeoutMs) });
  if (res.status < 300 || res.status > 399 || !res.headers.get('location')) return res;
  if (hop >= maxRedirects) throw new Error('too many redirects');
  url = new URL(res.headers.get('location')!, url);   // Location may be relative
  if (seen.has(url.href)) throw new Error('redirect loop');
  seen.add(url.href);
}

Return the redirect chain to the caller as well. It makes debugging much easier for API users.

There is a gap I have not closed: DNS rebinding. The Worker checks the name, then fetch resolves it again, and a hostile DNS server with a zero TTL can answer differently the second time. Workers do not let you pin the connection to the IP you checked, and a fetch to a bare IP breaks TLS and virtual hosting. If your threat model is serious, put the fetcher behind an egress proxy that does the resolution and the check in one step.

4. Cap time and bytes

content-length is a hint, not a promise. Reject early when it is present and too large, then enforce the cap while streaming:

const reader = res.body!.getReader();
let total = 0;
const chunks: Uint8Array[] = [];
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  if (total + value.byteLength > maxBytes) { await reader.cancel(); throw tooLarge(maxBytes); }
  chunks.push(value);
  total += value.byteLength;
}

Gzipped sitemaps need the same treatment after decompression (DecompressionStream('gzip')), or a few kilobytes of input can expand into gigabytes. Pass every fetch an AbortSignal.timeout() too.

Also mind the plan limits. On the Workers Free plan a request gets 10 ms of CPU time and 50 subrequests, according to the pricing and limits pages. Two DoH lookups per hop count against the subrequest limit, and parsing a large PDF or sitemap does not fit in 10 ms. A small in-memory DNS cache with a 60-second lifetime helps with the first problem. The second needs the paid plan.

5. Respect robots.txt properly

If your API crawls on behalf of users, honor robots.txt. RFC 9309 standardized the rules in 2022, and a few of them differ from what older parsers do:

• Matching: the most specific rule (the longest pattern) wins. When an Allow and a Disallow are equally specific, Allow should win. * matches any sequence and a trailing $ anchors the end.
• Groups: use the group whose user-agent matches your product token, and fall back to * only when none does. Several groups for the same agent are merged.
• Fetching robots.txt: a 4xx response means no restrictions. A 5xx response or a network failure means you must assume everything is disallowed. Follow at least five redirects, and parse at least the first 500 KiB.
• /robots.txt itself is always allowed.

The matcher is short. Compile each pattern into an anchored regex, and keep the raw pattern length to compare specificity:

function compile(pattern: string): RegExp {
  const anchored = pattern.endsWith('$');
  const body = (anchored ? pattern.slice(0, -1) : pattern)
    .split('*').map((p) => p.replace(/[.+?^${}()|[\]\\]/g, '\\$&')).join('.*');
  return new RegExp('^' + body + (anchored ? '$' : ''));
}

canFetch(token: string, url: string): boolean {
  if (this.disallowAll) return false;          // robots.txt returned 5xx or was unreachable
  const u = new URL(url);
  if (u.pathname === '/robots.txt') return true;
  const path = decodeURIComponent(u.pathname) + u.search;
  let best = -1, allow = true;
  for (const r of this.rulesFor(token)) {
    if (r.rx.test(path) && (r.pattern.length > best || (r.pattern.length === best && r.allow))) {
      best = r.pattern.length;
      allow = r.allow;
    }
  }
  return allow;
}

Ignore empty Disallow: lines (they mean "allow everything"), and wrap decodeURIComponent in a try/catch, because real-world paths contain stray % signs.

Testing

Three layers were worth writing:

• Pure unit tests for the robots matcher: group selection by product token, longest match, an Allow/Disallow tie, * and $ patterns, an empty Disallow, and percent-encoded paths.
• Unit tests with a stubbed fetch for the network logic: a public URL that 302s to 169.254.169.254, a 301 to http://[::ffff:7f00:1]/ (127.0.0.1 written as IPv4-mapped IPv6 in hex), a robots.txt that returns 503, a robots.txt behind a redirect, and a redirect loop.
• Integration tests against wrangler dev with real public URLs and blocked targets, to confirm the checks stop what they should without blocking legitimate sites.

One gap I would add next: a test for a body that is larger than its content-length header claims.

This article was created with the help of AI. The code was tested by me before publishing.