Search crawlers were adding our customers' events to their calendars

A customer messaged us because their add to calendar count looked far too high for the size of their event, and almost none of those adds came after someone had looked at the event page.

We run AddCal, which makes add to calendar links and buttons for events. The add count is the number our customers look at most, so when it's way off we need to know why.

How an add gets counted

Each event page has a button per calendar app (Google, Apple, Outlook, Office 365, Yahoo, .ics and so on). Every button is a link like /e/{event}/google. We count a hit on that link as an add, then redirect to the calendar app.

When the page renders, the JS adds an ac click token to each link, a random 14 char hex string. iOS Calendar re-fetches .ics URLs every hour, so the token lets us count the first click and skip the refreshes. Repeats of the same token within 24 hours aren't counted.

We already returned a 403 to anything with bot, spider or crawl in the user agent. It didn't help much.

What the traffic looked like

Platform wide it started on 24 September: 1,000 to 2,400 fake adds a day, across 200 to 700 events a day. The pattern:

• Straight hits on /e/{event}/{service} with no referer.
• Spread evenly over all 7 calendar apps. People mostly pick Google or Apple, nobody picks Yahoo as often as Google.
• Nearly every hit from a new IP, mostly in China, landing on events in US timezones.
• Valid ac tokens, which only our page JS creates.

Search crawlers run JavaScript now, so they render the page with real tokens in the links and then follow those links. The crawler's own request gets our 403. Then the same URL, same token, comes back from other IPs with a normal browser user agent, and that one counted.

We pulled 4 days of request logs and matched each token back to the crawler that first fetched it:

Sogou is close to one replay per token. The "browsers" come from the same Jiangsu ISP ranges as the spider, and the spider itself sometimes sends full Chrome headers. It looks like one fleet fetching under two user agents. On one IP the user agent flipped between Mac Chrome and Mac Safari within 20 seconds on the same token.

Bing replays came through Microsoft IPs as Edge, plus proxies. Baidu showed up as Chrome/133.0.6943.141 on Windows, the exact same build every time, from Baidu's own network.

Step 1: match the ASN to datacentre hosting

The first fix was flagging requests from hosting networks. Real people adding an event to their calendar are on home or mobile connections, not on AWS.

We look up each IP's ASN with MaxMind's GeoLite2-ASN database and match it, along with the IP, against public lists of datacentre and hosting networks. To pick the lists, we built a labelled sample from 14 days of logs: 40k bot hits (tokens replayed after a crawler 403, or used from 3+ IPs) and 41k human sessions (page load then an add from the same IP within 15 minutes). Then we scored a few open lists:

Most of the humans the lists flagged were on iCloud Private Relay, which exits through Cloudflare, Fastly and Akamai, so every list treats it as hosting. Apple publishes the egress ranges and we exempt them. The next biggest group was Chrome on Windows coming from Google's AS15169 over IPv6, so Google IPv6 outside the published Cloud ranges is exempt too. So are Zscaler, Forcepoint and iboss, since a lot of people browse through those at work.

Step 2: mark links handed to crawlers

About a third of the bad traffic came from home ISPs. No IP list will catch that.

We can tell when a page is being rendered for a crawler though, because the crawler names itself in its user agent. So the links a crawler gets carry a marked token. When that token comes back later from a normal looking browser, we don't count the add.

This has to happen on the server, since the crawler reads the server-rendered HTML. It also only covers links handed out after the change, not the ones already sitting in crawler queues.

Step 3: headers the browser sends by itself

Browsers add some headers that a script replaying a URL usually forgets:

• Chromium 89+ sends sec-ch-ua on every HTTPS request, including Edge, Opera, Brave, Samsung Internet and Android WebView.
• Chromium, Firefox 90+ and Safari 16.4+ send Sec-Fetch-Mode, Sec-Fetch-Site and Sec-Fetch-Dest.

We logged these for 4 days and scored rules against the labelled samples:

The first two catch scripts. They miss the Sogou replays completely, because those run a real Chrome and send every header.

The third one catches those. Sec-Fetch-Site: none means the navigation didn't come from a page: it was typed, pasted, bookmarked or opened from another app. A person clicking our button sends same-origin, or cross-site from an embed. Email links don't carry a token, since only our page JS creates one, so someone clicking an add link in a newsletter is never caught by it.

Edge cases worth testing if you do this: Chrome on iOS (CriOS) is WebKit and sends no client hints. Firefox sends no client hints but does send Sec-Fetch-*. Safari 16 and older send neither. iOS in-app browsers (Facebook, Instagram) don't put a Version/ in the user agent at all.

We're running these as log-only for now, to measure them on live traffic before they flag anything.

Flagged adds still work

Flagged adds still redirect to the calendar app. We store them with is_server = 1, skip the counters, and filter them out of every report.

If we're wrong about someone, they still get the event in their calendar. And we can change our mind later, since nothing is thrown away.

Stop handing crawlers the links

Everything above deals with the links after crawlers have them. We also stopped handing them out:

• Every add and subscribe button is now rel="nofollow".
• robots.txt blocks Bing, Sogou and Baidu from the add and subscribe paths. Event pages stay crawlable for everyone.

We left Google allowed. The add links send X-Robots-Tag: noindex, and Google only sees that header on URLs it's allowed to fetch. Blocking it would leave any already-indexed add links stuck in the index, to stop about 40 replays in 4 days.

robots.txt only reaches the crawlers themselves. Sogou's browser-like fleet probably ignores it, so we're keeping the token and header checks.

Where it landed

We watch the human rate most. What's left there is mostly people on commercial VPNs (M247, Datacamp, Clouvider), which run on hosting networks.

If you count clicks on links crawlers can see, the quickest check is whether a token or ID that only your page hands out is turning up from IPs that never loaded the page. An even spread across options people don't pick evenly is another good sign something's off.

If you want add to calendar links where someone else deals with this, that's AddCal. You can make one in a few seconds with the free link generator, or put an add to calendar button on your site.