Dev.to WebDev 🛠 Dev Fri, 06 Nov 2026 03:02:23 +0000 👁 0 📖 9 min read

Detect a website's tech stack for B2B lead research (Python + Node.js)

If you sell a Shopify app, a WordPress plugin or migration services, the most useful thing to know about a prospect is what their website runs on. The same goes for agencies doing audits and for anyone cleaning a lead li

This post covers:

A standard-library Python script that fingerprints the CMS, shop system and CDN from the HTML and headers, and the email provider from MX records.
The pitfalls: false positives from naive rules, things you can't see without a browser, bot protection, and the legal side of using this for outreach.
A hosted detector with fingerprints for 324 technologies and a lead filter, used from Python and Node.js.

Disclosure: I built the Tech Stack Detector Actor used in part 3. Parts 1 and 2 don't need it.

1. A minimal fingerprinting script (Python standard library)

Most technologies leave traces in one of three places: response headers (server: cloudflare, x-served-by: cache-… from Fastly), the HTML (/wp-content/, cdn.shopify.com, __NEXT_DATA__), and DNS (MX records pointing to …mail.protection.outlook.com mean Microsoft 365). The script below checks a small set of rules. Google's DNS-over-HTTPS JSON API handles the MX lookup, so no DNS library is needed.

"""Guess a site's CMS, shop system, CDN and email provider with the standard library only.

Usage: python techstack_stdlib.py shopware.com wordpress.org allbirds.com
"""
import json
import re
import sys
import urllib.request

UA = {"User-Agent": "Mozilla/5.0 (techstack-demo; +https://example.com/contact)"}

# (technology, where to look, regex) - a tiny subset of real fingerprint sets
RULES = [
    ("WordPress", "html", r"/wp-content/|/wp-includes/|<meta[^>]+generator[^>]+WordPress"),
    ("WooCommerce", "html", r"woocommerce"),
    ("Shopify", "html", r"cdn\.shopify\.com|Shopify\.theme"),
    ("Shopware", "html", r"/bundles/storefront/|sw-plugin|Shopware"),
    ("Contentful", "html", r"ctfassets\.net"),
    ("Next.js", "html", r"/_next/static/|__NEXT_DATA__"),
    ("Google Tag Manager", "html", r"googletagmanager\.com/gtm\.js"),
    ("Cloudflare", "headers", r"^server: cloudflare|^cf-ray:"),
    ("Fastly", "headers", r"^x-served-by: cache-|^fastly-"),
    ("Amazon CloudFront", "headers", r"^x-amz-cf-id:|^via: .*cloudfront"),
    ("nginx", "headers", r"^server: nginx"),
]
MX = {"google.com": "Google Workspace", "googlemail.com": "Google Workspace", "outlook.com": "Microsoft 365",
      "pphosted.com": "Proofpoint", "mimecast.com": "Mimecast", "zoho": "Zoho Mail"}


def fetch(domain):
    req = urllib.request.Request(f"https://{domain}/", headers=UA)
    with urllib.request.urlopen(req, timeout=20) as r:
        headers = "\n".join(f"{k.lower()}: {v.lower()}" for k, v in r.headers.items())
        return headers, r.read(3_000_000).decode("utf-8", "replace")


def mail_provider(domain):
    url = f"https://dns.google/resolve?name={domain}&type=MX"  # DNS over HTTPS, no extra library
    answers = json.load(urllib.request.urlopen(url, timeout=10)).get("Answer", [])
    hosts = [a["data"].split()[-1].rstrip(".").lower() for a in answers if a.get("type") == 15]
    for h in hosts:
        for key, name in MX.items():
            if key in h:
                return name
    return "self-hosted / other" if hosts else "none"


for domain in sys.argv[1:]:
    try:
        headers, html = fetch(domain)
    except Exception as exc:  # noqa: BLE001 - one bad site must not stop the list
        print(f"{domain:<16} ERROR {exc}")
        continue
    found = [t for t, where, rx in RULES if re.search(rx, headers if where == "headers" else html, re.I | re.M)]
    print(f"{domain:<16} {', '.join(found) or '-'} | mail: {mail_provider(domain)}")

Output (03.11.2026):

$ python techstack_stdlib.py shopware.com wordpress.org allbirds.com
shopware.com     Shopware, Contentful, Fastly, nginx | mail: Microsoft 365
wordpress.org    WordPress, WooCommerce, Google Tag Manager, nginx | mail: self-hosted / other
allbirds.com     Shopify, Google Tag Manager, Cloudflare | mail: Microsoft 365

It looks plausible, but two of these results are wrong. That's what part 2 is about.

2. Pitfalls

Mentions aren't installations. shopware.com is Shopware's marketing site. It's built on Contentful and Nuxt, but the word "Shopware" appears all over the page, so the rule fires. wordpress.org isn't a WooCommerce shop, it just links to WooCommerce. Good fingerprints target things only an installation produces: asset paths (/bundles/storefront/), cookies (_shopify_y), meta generator tags, specific headers or JavaScript globals. They don't match brand names in the page text.
You only see what the server sends. Tags injected by Google Tag Manager, or loaded only after cookie consent, aren't in the initial HTML. A plain HTTP fetch will miss many analytics and marketing tools. A headless browser sees more, but it's slower and more expensive per site. For CMS, shop system, CDN and email provider, the initial HTML and DNS are almost always enough.
Bot protection returns a different page. Cloudflare or Akamai challenges send a "checking your browser" page with status 403 or 503. Detect that and report it as "blocked". Don't report "no technologies found".
Versions are rarely exposed. <meta name="generator" content="WordPress 6.6"> is getting rare, because hiding versions is a common hardening step. Don't build a pitch on version numbers unless the page actually shows them.
The legal side of outreach. Detecting technology on a public website is fine. What you do with the result is a different matter. In the EU, cold email to businesses still falls under the GDPR and national rules (in Germany, § 7 UWG generally requires prior consent for advertising emails, even B2B). Use tech-stack data to qualify and personalise, and check your own legal basis before you contact anyone.
Be polite at scale. Fetch one page per domain, set a descriptive User-Agent, run a few requests in parallel at most, and cache results. A site's stack doesn't change every day.

3. Hosted option: Tech Stack Detector on Apify

The Tech Stack Detector takes domains or URLs and returns one record per site. Each record includes:

The CMS, shop system, JS frameworks, analytics, tag managers, consent managers, CDN, hosting, web server, payments, live chat and marketing tools it finds
The email provider from MX records
An evidence string for every detection (which header, cookie, script URL or HTML snippet matched), so you can check false positives yourself
A warning when the site returned an error status or almost no HTML, which is typical of bot protection and JavaScript-only sites

filterTechnologies turns it into a lead filter: only sites with at least one of the listed technologies are returned (and charged).

Set APIFY_TOKEN from a free Apify account (Console → Settings → API & Integrations).

Python (`pip install apify-client`)

"""Detect the tech stack of a list of websites and save CMS, shop system and more to techstack.csv.

Actor: https://apify.com/paperlinenstudio/tech-stack-detector
Usage: APIFY_TOKEN=... python tech_stack_leads.py shopware.com wordpress.org shopify.com
Lead filter: ONLY=Shopify,WooCommerce python tech_stack_leads.py ... keeps only matching sites (others are not charged).
"""
import csv
import os
import sys

from apify_client import ApifyClient

domains = sys.argv[1:] or ["shopware.com", "wordpress.org", "shopify.com"]
ONLY = [t.strip() for t in os.environ.get("ONLY", "").split(",") if t.strip()]
client = ApifyClient(os.environ["APIFY_TOKEN"])

run = client.actor("paperlinenstudio/tech-stack-detector").call(
    run_input={"urls": domains, "filterTechnologies": ONLY, "includeEvidence": False}
)
rows = list(client.dataset(run.default_dataset_id).iterate_items())
cols = ["domain", "cms", "ecommerce", "jsFrameworks", "analytics", "consentManagers", "cdn", "hosting", "emailProvider", "technologyCount", "error"]
with open("techstack.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f)
    w.writerow(cols)
    for r in rows:
        w.writerow([", ".join(v) if isinstance(v, list) else v for v in (r.get(c) for c in cols)])
for r in rows:
    show = lambda v: (", ".join(v) if isinstance(v, list) else v) or "-"  # noqa: E731
    print(f"{r['domain']:<20} cms={show(r.get('cms'))}  shop={show(r.get('ecommerce'))}  cdn={show(r.get('cdn'))}  mail={show(r.get('emailProvider'))}")
print(f"Saved {len(rows)} rows to techstack.csv")

Output (03.11.2026), with the lead filter set to Shopify or WooCommerce:

$ ONLY=Shopify,WooCommerce python tech_stack_leads.py allbirds.com gymshark.com wordpress.org shopware.com woocommerce.com
woocommerce.com      cms=WordPress  shop=WooCommerce  cdn=-  mail=Self-hosted / other
gymshark.com         cms=-  shop=Shopify  cdn=Cloudflare  mail=Proofpoint
allbirds.com         cms=-  shop=Shopify  cdn=Cloudflare  mail=Microsoft 365
Saved 3 rows to techstack.csv

wordpress.org and shopware.com were dropped, because neither runs a shop system. Compare this with part 1: here shopware.com comes back as Contentful (not Shopware), and wordpress.org isn't tagged as WooCommerce.

Node.js (`npm i apify-client`)

// Detect the tech stack of a list of websites and save CMS, shop system and more to techstack.csv.
// Usage: APIFY_TOKEN=... node tech_stack_leads.mjs shopware.com wordpress.org shopify.com
// Lead filter: ONLY=Shopify,WooCommerce node tech_stack_leads.mjs ... keeps only matching sites (others are not charged).
import { writeFileSync } from 'node:fs';
import { ApifyClient } from 'apify-client';

const domains = process.argv.slice(2).length ? process.argv.slice(2) : ['shopware.com', 'wordpress.org', 'shopify.com'];
const ONLY = (process.env.ONLY ?? '').split(',').map((t) => t.trim()).filter(Boolean);
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('paperlinenstudio/tech-stack-detector').call({ urls: domains, filterTechnologies: ONLY, includeEvidence: false });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
const cols = ['domain', 'cms', 'ecommerce', 'jsFrameworks', 'analytics', 'consentManagers', 'cdn', 'hosting', 'emailProvider', 'technologyCount', 'error'];
const cell = (v) => { const s = Array.isArray(v) ? v.join(', ') : (v ?? ''); return `"${String(s).replaceAll('"', '""')}"`; };
writeFileSync('techstack.csv', [cols.join(','), ...items.map((r) => cols.map((c) => cell(r[c])).join(','))].join('\n') + '\n');
const show = (v) => (Array.isArray(v) ? v.join(', ') : v) || '-';
for (const r of items) console.log(`${r.domain.padEnd(20)} cms=${show(r.cms)}  shop=${show(r.ecommerce)}  cdn=${show(r.cdn)}  mail=${show(r.emailProvider)}`);
console.log(`Saved ${items.length} rows to techstack.csv`);

$ node tech_stack_leads.mjs
wordpress.org        cms=WordPress  shop=-  cdn=-  mail=Self-hosted / other
shopify.com          cms=-  shop=Shopify  cdn=Cloudflare  mail=Google Workspace
shopware.com         cms=Contentful  shop=-  cdn=Fastly  mail=Microsoft 365
Saved 3 rows to techstack.csv

A single record (real, from 03.11.2026; technologies list shortened to two entries):

{
  "url": "https://shopware.com/",
  "domain": "shopware.com",
  "finalUrl": "https://www.shopware.com/en/",
  "statusCode": 200,
  "redirects": 2,
  "https": true,
  "title": "The ecommerce platform to power your online business | Shopware",
  "language": "en",
  "generator": null,
  "cms": "Contentful",
  "ecommerce": null,
  "jsFrameworks": [
    "Nuxt",
    "Vue.js"
  ],
  "analytics": [],
  "tagManagers": [
    "Google Tag Manager"
  ],
  "advertising": [],
  "consentManagers": [
    "Usercentrics"
  ],
  "cdn": [
    "Fastly"
  ],
  "hosting": [],
  "webServers": [
    "Nginx"
  ],
  "backend": [],
  "liveChat": [],
  "marketingAutomation": [],
  "payments": [],
  "emailProvider": "Microsoft 365",
  "mxRecords": [
    "shopware-com.mail.protection.outlook.com"
  ],
  "technologyCount": 14,
  "technologyNames": [
    "Fastly",
    "Contentful",
    "Varnish",
    "Usercentrics",
    "Nuxt",
    "Vue.js",
    "PWA (Web App Manifest)",
    "Open Graph",
    "Schema.org JSON-LD",
    "Friendly Captcha",
    "Content Security Policy",
    "HSTS",
    "Google Tag Manager",
    "Nginx"
  ],
  "categories": {
    "CDN": [
      "Fastly"
    ],
    "CMS": [
      "Contentful"
    ],
    "Cache": [
      "Varnish"
    ],
    "Consent management": [
      "Usercentrics"
    ],
    "JavaScript framework": [
      "Nuxt",
      "Vue.js"
    ],
    "Progressive Web App": [
      "PWA (Web App Manifest)"
    ],
    "SEO": [
      "Open Graph",
      "Schema.org JSON-LD"
    ],
    "Security / captcha": [
      "Friendly Captcha"
    ],
    "Security header": [
      "Content Security Policy",
      "HSTS"
    ],
    "Tag manager": [
      "Google Tag Manager"
    ],
    "Web server": [
      "Nginx"
    ]
  },
  "technologies": [
    {
      "name": "Fastly",
      "category": "CDN",
      "version": null,
      "evidence": "header x-served-by: cache-iad-kcgs7200129-IAD"
    },
    {
      "name": "Contentful",
      "category": "CMS",
      "version": null,
      "evidence": "html …el=\"preconnect\" href=\"https://images.ctfassets.net\"><link nonce=\"L24q8x3WvkmBXzf…"
    }
  ],
  "warning": null,
  "error": null,
  "responseTimeMs": 3152,
  "checkedAt": "2026-11-03T14:04:31+00:00"
}

Inputs you'll probably change

{
  "urls": ["example-shop.de", "https://www.example.com/"],
  "filterTechnologies": ["Shopware", "WooCommerce", "Magento"],
  "detectEmailProvider": true,
  "includeEvidence": true,
  "maxConcurrency": 10
}

Keep includeEvidence on while you build a list, so you can spot-check why a site was tagged. Switch it off for large exports.

Wrap-up

A quick look at a few sites: headers, HTML and an MX lookup with the standard library. Only use rules that match traces of an installation, not brand names.
Lead lists of hundreds or thousands of domains, with evidence and a technology filter: Tech Stack Detector.

Feedback and feature requests are welcome, either in the comments or on the Actor's Issues tab.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.

← News 🛠 Dev 🏠 Home