What 235 real domains taught us about bulk website technology lookups
We ran the same 235 domains through StoreTechScan three times. In the first run, 69 were analyzable. In the paced run, 218 were. On a later pass with other changes, coverage dropped to 59. An enrichment job that reports
We ran the same 235 domains through StoreTechScan three times. In the first run, 69 were analyzable. In the paced run, 218 were. On a later pass with other changes, coverage dropped to 59. An enrichment job that reports only the rows it can classify can make these runs look much more similar than they actually were. Benchmark and CC BY 4.0 data
I work on StoreTechScan. These measurements are from our scanner. The comparisons below describe the workflows each tool documents.
Measure access before comparing detections
The fixed list contained 202 domains from Shopify case studies, nine from WooCommerce showcases or case studies, and 24 non-store controls. We kept the same list through all three runs. Historical case studies were the platform reference. A store could have changed platforms by the time we requested its homepage. Benchmark CSV
| Run | Analyzable | HTTP 429 | WAF blocks |
|---|---|---|---|
| Initial run with 16 workers | 69/235 | 155 | 11 |
| Four workers with 250 ms between starts | 218/235 | 1 | 14 |
| Later repeat with extraction and memory changes | 59/235 | 164 | 11 |
The runs varied in timing and other settings, so the comparison measures variation in coverage under changing conditions. In the paced run, 209 of the 218 analyzable domains agreed with their historical platform reference. That 95.9% agreement applies to this reachable, Shopify-heavy cohort. The other 17 domains in the input should still appear in the report with their access status. Measurement notes
DNS can add evidence when the page is quiet
In a separate November 19 pilot, we checked public MX, TXT, NS, www CNAME and DMARC records for 71 Shopify stores, nine WooCommerce stores and 20 B2B SaaS sites. We found at least one recognized signal for 97 of 100 domains. Google Workspace MX appeared on 52 and Shopify www CNAME on 41. A TXT verification record can persist after a service has been removed. Keep that evidence alongside the HTTP or markup signal so a downstream reviewer can decide how much weight it deserves.
The median DNS lookup took 345 ms and the 90th percentile hit the 1.5 s deadline. That pilot used a longer deadline than the production lookup. Put DNS alongside the page request and give it a deadline of its own so an unresponsive nameserver does not hold up the result.
Choose the lookup that matches your input
| Tool | Where it fits |
|---|---|
| Wappalyzer | A supplied URL list that needs cached technology profiles or a live recursive crawl. The documented API supports both modes and uses callbacks for live recursive lookups. |
| BuiltWith | A supplied list for Bulk Domain API lookups, or discovery of websites using a technology through the Lists API. |
| StoreTechScan | A supplied list that needs a synchronous batch of domain results with observed HTTP and DNS evidence, including explicit failure rows. |
Wappalyzer Technology Lookup ยท BuiltWith API ยท StoreTechScan API
Here's a two-domain request you can run without a key while your one-time trial has allowance.
curl -sS --fail-with-body https://storetechscan.com/api/v1/inspect \
-H 'Content-Type: application/json' \
--data '{"domains":["allbirds.com","gymshark.com"],"profile":"tech_stack","include_evidence":true}' \
| jq '.results[] | {domain, status, billable, dns_signals, evidence}'
Keep status and billable with the technology columns in your CSV or warehouse. A blank technology field on an accessible page and a blank field on a blocked page deserve different follow-up. StoreTechScan OpenAPI specification
AI tools assisted with the preparation of this article.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.