Your Firewall Is Eating Your Link Previews: Bot Protection, Geo-Blocks and Social Crawlers

You did everything right. og:title is set, og:image is an absolute HTTPS URL to a 1200×630 JPG, and "view source" shows the tags in the first HTML response. Then you paste the link into LinkedIn and get a bare URL. Facebook shows the domain and nothing else. Slack shows a title but no image.

When the tags are correct and the preview is still empty, the problem is usually not your markup. Something in front of your app is refusing to talk to the crawler. This post is a checklist for that layer: CDNs, WAFs, bot protection, geo rules and hotlink protection.

Preview crawlers are bots, and your security stack hates bots

When someone shares a link, the platform (not the person) fetches your page. That request looks nothing like a browser visit:

These requests run no JavaScript, keep no cookies, never solve a CAPTCHA, often come from another country, and sometimes arrive in bursts when a post spreads. To a modern bot-protection product, that looks a lot like a scraper. So it gets a challenge page, a 403, or a 429, and the platform quietly gives up.

The platform never tells the person sharing the link. They just see an ugly card and assume your site is broken.

Step 1: Reproduce it with curl, one crawler at a time

Before touching any dashboard, find out which crawlers fail. This loop fetches the page and the OG image the way each crawler would, and prints the status code and content type:

#!/usr/bin/env bash
# check-preview-crawlers.sh <page-url> <og-image-url>
PAGE="$1"
IMG="$2"

UAS=(
  "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"
  "LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)"
  "Twitterbot/1.0"
  "Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)"
  "Discordbot/2.0; +https://discordapp.com"
  "TelegramBot (like TwitterBot)"
  "WhatsApp/2.23.20.0 A"
)

for ua in "${UAS[@]}"; do
  page=$(curl -s -o /dev/null -w "%{http_code} %{content_type}" -A "$ua" -L "$PAGE")
  img=$(curl -s -o /dev/null -w "%{http_code} %{content_type} %{size_download}B" -A "$ua" -L "$IMG")
  printf "%-28s page: %-32s image: %s\n" "${ua%% *}" "$page" "$img"
done

What you want to see for every row: 200 text/html for the page and 200 image/jpeg (or png/webp) for the image, with a sensible size.

What the bad rows usually look like:

• 403 text/html: a WAF rule or bot-fight mode blocked it.
• 503 or 200 text/html on the image URL: a JavaScript challenge or "checking your browser" page was served instead of the file. The 200 is a lie. The crawler got HTML, not a JPG.
• 429: rate limiting.
• 301 loops or a login page: an auth wall or a redirect that only browsers survive.

One important limit: curl from your laptop is not curl from Meta's network. A rule that blocks by country, ASN or IP reputation will let your request through and block the real crawler. If curl looks fine but the platform still fails, go to Step 2.

Step 2: Ask the platforms what they actually received

Each big platform has an official tool that shows its own fetch result:

• Facebook Sharing Debugger (developers.facebook.com/tools/debug) shows the response code Meta got, the final URL after redirects, and which tags it read. If it says "Response Code: 403" or "Could not scrape URL", that is your firewall talking. The Facebook link preview guide on OG Scope explains how the cache behaves once the fetch works, which matters because Facebook keeps the broken scrape until you ask for a new one.
• LinkedIn Post Inspector (linkedin.com/post-inspector) is blunter than the post composer. If its crawler cannot fetch the image file, it tells you, while the composer just drops the picture. The LinkedIn preview notes cover which tags LinkedIn reads and why it can keep showing an older card.

There is also a useful middle step between curl and the platform tools. A third-party checker such as OG Scope fetches your page from its own server, without running JavaScript, and draws estimated cards for Facebook, LinkedIn, X, Slack, WhatsApp, Discord and Telegram. If OG Scope can read your tags but Facebook cannot, the block is specific to Meta's network or user agent. If neither can, the block is broader, or the tags only exist after JavaScript runs.

The usual suspects

1. Bot-fight modes and "under attack" challenges

CDN bot-protection features (Cloudflare Bot Fight Mode, "I'm Under Attack" mode, similar settings at other providers) challenge clients that do not behave like browsers. Preview crawlers never pass a JavaScript challenge. If you turned on an aggressive mode during a traffic spike and forgot about it, your previews have been broken since that day.

2. Custom WAF rules that match on user agent

Rules like "block anything that is not Chrome, Safari or Firefox" or "block user agents containing bot" are common copy-paste snippets. They take out every preview crawler in one line.

3. Geo-blocking

If you block all countries except your own (very common for local stores, banks, and sites that got hammered by foreign traffic), remember that Meta, LinkedIn, X and Slack mostly fetch from data centres in the US and Europe. Your Pakistani or Indonesian visitors can see the page fine and still get a blank card in their group chats.

WhatsApp is the exception. Its preview is built on the sender's phone, so a geo rule usually does not break WhatsApp, but a user-agent rule or a "browsers only" challenge will.

4. Rate limiting

A popular post can make a platform re-fetch the URL many times in a short window. A strict per-IP limit on / or on your image path can start returning 429s at exactly the moment the post is getting attention.

5. Hotlink protection on the image path

Hotlink protection rules decide whether to serve an image based on the Referer header. Crawlers often send none, or send their own. If your rule only serves images when the referer is your own domain, the HTML fetch succeeds and the image fetch fails, which is the classic "title but no picture" card. The OG image checker on OG Scope is handy here because it reports whether the image file itself loaded, separately from the tags.

6. Signed or expiring image URLs

Images served from a private bucket with a signed URL work on launch day and return 403 a week later, when the platform re-scrapes. Use a stable, public URL for og:image. When you change the image, change the URL (for example add ?v=2) instead of overwriting the file in place.

Letting preview crawlers through without opening the door to scrapers

The lazy fix, "allow every request whose user agent contains facebookexternalhit", works, but user agents are trivial to fake. Anyone who reads this post can now send that header. Better options, roughly from best to worst:

Use your CDN's verified-bot signal

Cloudflare verifies well-known bots by IP and network, and exposes the category in the rules language. The category for link-preview crawlers is literally named Page Preview. A WAF custom rule like this lets verified preview bots skip the rest of your custom rules and your bot-fight settings:

Expression:
  (cf.verified_bot_category eq "Page Preview")

Action:
  Skip -> remaining custom rules, Super Bot Fight Mode rules

Depending on your plan, you may only have the broader cf.client.bot (known good bots) field. That includes search engines and other verified bots too, which is usually fine.

If you only want to relax things for the paths a crawler actually needs, narrow it:

(cf.verified_bot_category eq "Page Preview")
and (http.request.method eq "GET")
and (starts_with(http.request.uri.path, "/blog/")
     or starts_with(http.request.uri.path, "/og/"))

Other CDNs and hosting platforms have their own "verified bots" lists. Look for that before you write user-agent rules by hand.

Verify by network, if you run your own proxy

Meta documents that its crawler IPs belong to its own autonomous system, and you can list current ranges with a WHOIS query:

whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

If you run nginx or a self-hosted WAF, allow the preview user agents only when the request also comes from those ranges. Re-pull the list on a schedule. Hard-coding it once is how it silently goes stale.

Exempt the image path from hotlink rules

Keep hotlink protection for your big media, but serve your OG images from a path that has no referer check, for example /og/. Those files are meant to be shown on other sites. That is the whole point of them.

Give crawlers a separate rate-limit bucket

If your rate limiter supports it, put verified preview bots in their own, looser bucket instead of exempting them entirely.

A five-minute pre-launch checklist

Before you share a launch link in a big group or post:

1. Run the curl script against the live URL and the image URL. Every row should be 200 with the right content type.
2. Paste the URL into a server-side preview checker like OG Scope and confirm the image loads.
3. Run the Facebook Sharing Debugger and LinkedIn Post Inspector once. Fix any non-200 response before the first real share, because the first scrape is the one that gets cached.
4. Check your CDN dashboard for any "under attack", bot-fight or geo rule someone turned on in the last few months.
5. If you changed the image, change its URL too.

The annoying part about this class of bug is that the page works perfectly for you. Your browser passes every challenge, sends a nice referer, lives in an allowed country, and runs the JavaScript. The crawler does none of that. Test as the crawler, and the "mystery blank card" usually turns out to be a single firewall rule.