Stop Getting Blocked: A Deep Dive into Hybrid Web Scraping Architecture

If you've ever tried to scrape Amazon or Flipkart, you know the pain. You write a perfect BeautifulSoup script, it works for two days, and then—bam. Cloudflare blocks your IP, or the site changes a CSS class, and your scraper crashes.

I got tired of maintaining broken scrapers, so I built a Self-Healing E-Commerce Scraper that refuses to break. It uses a hybrid architecture, a 6-tier extraction cascade, and AI fallbacks to survive even the most heavily fortified websites.

Here is a deep dive into how the architecture works, and how you can use it. Pillar 1: Hybrid Fetching (Cost-Per-Success)

Most people make the mistake of launching a heavy Selenium/Playwright browser for every request. It's slow and eats gigabytes of RAM.

Instead, I built a dual-path system:

The Fast Path: The scraper first tries a standard HTTP request using curl_cffi. This library spoofs the TLS/JA3 fingerprint of a real Chrome browser. It fetches static sites in milliseconds.

The Heavy Path: If the fast path fails (403 Forbidden) OR the site requires JavaScript (like Amazon), it automatically drops into Camoufox—a modified Firefox fork that spoofs fingerprints at the C++ engine level.  Pillar 2: The 6-Tier Extraction Cascade

When the HTML arrives, the engine doesn't just rely on one CSS selector. It tries 6 different strategies in order, stopping as soon as it succeeds:

JSON-LD: Scans for application/ld+json (the most resilient method).

Embedded JSON: Extracts Next.js (__NEXT_DATA__) or Shopify blobs.

CSS Selectors: Uses BeautifulSoup, but prioritizes selectors that worked in previous runs (self-healing memory).

Regex Fallback: Pairs product names with their nearest price tag.

Fuzzy Matcher: Scans raw text for currency symbols (₹, $) and scores nearby text blocks.

LLM Healer: Sends cleaned HTML to Google Gemini or Qwen3.6 AI to extract the data semantically. Bypassing the Login Wall

Amazon's biggest defense is checking session history. A fresh browser with zero cookies is instantly banned.

To fix this, I added a warm_amazon.py script. You run it once, log in manually, and it saves your valid cookies. The scraper then injects these cookies into the stealth browser, making it look exactly like you browsing normally.

GitHub Repo: https://github.com/uyg7x/Self-Healing-Web-Scraper