Do AI chat engines block scrapers, and how do providers get around it?
Yes, AI chat engines block scrapers, and how much they block varies by engine. ChatGPT, Gemini, Google AI Mode and Bing AI Mode still answer a logged-out browser. Perplexity, the Copilot app and Grok now stop it at a log
Yes, AI chat engines block scrapers, and how much they block varies by engine. ChatGPT, Gemini, Google AI Mode and Bing AI Mode still answer a logged-out browser. Perplexity, the Copilot app and Grok now stop it at a login or sign-up wall, and Claude, Meta AI and DeepSeek never let it in.
- One provider dropped three engines in four weeks. Grok on 15 October 2026 (a sign-up wall before the first answer), the Copilot app on 7 November and Perplexity on 12 November, each because it blocked logged-out sessions. For Perplexity the changelog says each workaround "stopped working within days".
- A block can be regional and temporary. From 30 June to 24 September 2026, ChatGPT met every logged-out request from a Czech IP address with a login wall, while other countries still got answers.
- The page changes as often as the gate. By 25 August 2026, ChatGPT served 98% of logged-out answers in a mobile-web format that carries no model name and no search queries. No request was blocked, and every parser that read those fields had to change.
Do AI chat engines block scrapers, and how do providers get around it?
They do, in three ways: bot detection at the network edge, a login or sign-up wall, and changes to the page that break a parser. Providers get around the first and the third with real browsers, proxies in the target country, retries and parsers that follow each UI change. They do not get around the second without an account, and a provider that will not use accounts drops the engine.
The table shows what a logged-out browser gets from each engine on 16 November 2026. The evidence is the changelog and the providers guide of cloro, an API that runs these engines in logged-out browser sessions and records each change in access.
| Engine | Logged-out browser | Evidence |
|---|---|---|
| ChatGPT | Answers, behind Cloudflare bot checks. Login wall for Czech IP addresses from 30 June to 24 September 2026 | Changelog, 30 June and 24 September 2026; scraper README |
| Gemini | Answers | Supported endpoint |
| Google AI Mode, AI Overview | Answers | Supported endpoints |
| Bing AI Mode (Copilot Search in Bing) | Answers, except in CN, KR and RU, where it gives no answer |
Changelog, 7 November 2026 |
| Copilot app | Blocks logged-out sessions | Changelog, 7 November 2026 |
| Perplexity | Blocks logged-out sessions | Changelog, 12 November 2026 |
| Grok | Sign-up wall before the first answer | Providers guide, 15 October 2026 |
| Claude, Meta AI, DeepSeek | Login wall | Providers guide |
Bing AI Mode is Copilot Search inside Bing results. It is a different page from the Copilot app, and the two behave differently: the first still answers a logged-out visitor, the second does not.
How does ChatGPT detect a scraper?
The README of cloro's open ChatGPT scraper repo says that reading ChatGPT's web UI means getting past two Cloudflare checks: JA4 TLS fingerprinting and Turnstile.
Cloudflare documents both. A JA3 or JA4 fingerprint identifies a client by how it opens a TLS connection, and the fingerprint "acts as a stable identifier across different destination IPs, ports, and certificates". A site owner can write a rule that challenges or blocks a fingerprint. So a new IP address does not hide an HTTP library that opens connections differently from a browser. Turnstile runs "a series of small non-interactive JavaScript challenges" in the page, such as proof-of-work and probes of web APIs, and looks for browser quirks. A client that does not run JavaScript cannot complete it.
Cloudflare's Bot Management also gives each request a bot score from 1 to 99 for how likely it came from a bot. Heuristics against known fingerprints, a machine-learning model and JavaScript detections that find headless browsers all feed it.
The edge is the first gate. The second is the account. On 30 June 2026 the changelog records that ChatGPT met every logged-out request from Czech IP addresses with "a login wall, which scrapers cannot pass through", so the country came off the supported list. By 24 September ChatGPT no longer showed that wall, and CZ came back.
The third change broke parsers and left access alone. On 21 July 2026 OpenAI began to serve part of the logged-out traffic in a mobile-web format that streams HTML fragments instead of an event stream. By 25 August it served 98% of responses that way. In that format the model name and the search queries ChatGPT ran are not in the page, so a scraper that read them gets empty fields. The answer text and the inline citations did not change.
Why did Perplexity, Copilot and Grok stop working?
Each engine moved from bot checks to a wall, and the changelog shows the steps.
Grok. On 10 May 2026 the changelog says Grok's anti-bot defenses "have grown materially more complex over recent weeks, raising the proxy, fingerprinting, and retry cost of every successful extraction". By 15 October grok.com showed a sign-up wall to logged-out visitors before the first answer, and the endpoint was discontinued.
Perplexity. On 25 August 2026, Perplexity "continues to restrict access", and the cost to serve the endpoint had risen 70%. In the weeks before 16 September it showed log-in walls "on nearly all requests", until a workaround brought success rates back. On 12 November the endpoint was discontinued: "Each workaround we found stopped working within days."
Copilot. The Copilot app blocked logged-out sessions, and the endpoint was discontinued on 7 November 2026. Bing AI Mode, launched the same day, covers Bing's AI answers in search.
On Grok and Perplexity, the proxy, fingerprint and retry costs rose for months before either engine asked for an account.
How do scraping APIs get around the blocks?
With four methods, each of which makes the request look like what it is: one browser visit from one country.
- Real browser sessions. A full browser opens TLS connections like a browser and runs Turnstile's JavaScript like a browser, because it is one. The providers guide says cloro retrieves answers "using real browser sessions routed through proxy infrastructure", without the engines' own APIs.
- Proxies in the target country. Each request goes out through a proxy in the country the caller asks for, and for ChatGPT, Gemini and Bing AI Mode also a US state. The engine then answers as it would for a visitor in that place. This is also why a regional login wall, like ChatGPT's in Czechia, makes requests from that country fail while other countries still work.
- Retries. The Grok entry counts retries in the cost of each successful extraction. The provider tries again inside the same request, and cloro bills only successful results, so those internal retries cost the caller nothing.
- Parsers that follow the UI. The ChatGPT scraper README lists "CSS class names that change between deploys" as part of the work. When Perplexity moved its search queries to a new place in its response around 15 July 2026, the field came back empty on most responses until the parser was updated.
That is the method at the level a buyer needs to judge a provider. None of it works once an engine adds a wall.
Where does a provider stop?
At the login. A wall that asks for an account can be passed only with an account, and that is a different product with different terms. The providers guide lists Claude, Meta AI, DeepSeek and Amazon Rufus as not supported because each is "behind a login wall" and access "conflicts with" the vendor's terms of service. It also states the rule: the service "will not scrape services that require authentication".
There are three practical results for a buyer:
- Answers reflect a logged-out visitor. With no account of any kind, free or paid, the answer is what an unauthenticated person in that country sees. A ChatGPT Plus subscriber can see a different answer.
-
The model is the engine's choice. ChatGPT decides which model answers a logged-out session, for example
gpt-5-3orgpt-5-3-mini. No logged-out scraper can choose. - Coverage can shrink overnight. A provider that refuses to log in loses an engine the day the engine adds a wall. Three engines left that list in four weeks this autumn.
Ask any provider how it handles a login wall. If the answer is "with accounts", the data reflects those accounts.
Is this the same as websites blocking AI crawlers?
No. Many pages that answer this question cover the reverse case: a website that blocks the crawlers AI companies use to train models or to fetch pages for an answer. Cloudflare has a setting for that, AI Crawler and Agent access policies, with separate actions for search, agent and training traffic. That setting decides who may read your site. This post is about who may read the AI engine's answer page.
How was this checked?
Every date and quote above comes from public sources read on 16 November 2026: the changelog and providers guide in cloro's docs, and the README of its ChatGPT scraper. The Cloudflare statements come from Cloudflare's own documentation, linked where they are used.
Limits: the table records what one provider saw in logged-out browser sessions. A browser with a logged-in account, a different country or a different provider can get a different result on the same day. Engines change access often, so check the date on each row.
This post is published by cloro, which sells an API for the engines in the "Answers" rows above.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.