Dev.to AI 🤖 Ai 👁 0 📖 7 min read

How I Audited 50 Popular Tech Blogs for Rogue Tracking Scripts Using Go and Headless Browsers

I curl'd a tech blog's homepage, grepped the HTML for <script> tags, and counted four. I opened the same page in a real browser, opened DevTools → Network, and counted 31 third-party requests — including a consent cookie

I curl'd a tech blog's homepage, grepped the HTML for <script> tags, and counted four. I opened the same page in a real browser, opened DevTools → Network, and counted 31 third-party requests — including a consent cookie that was already set before I'd clicked anything.

That gap is the whole reason this project exists. Static HTML is a lie about what a page actually does. Modern tag managers, dynamically injected pixels, and shadow scripts only reveal themselves once JavaScript has executed in a real browser engine.

So I built Pageguard — a Go-based auditor that crawls sites, runs their client-side tags in a headless browser, and validates the network requests, DOM mutations, and cookies it observes against a plugin-based rule engine. Then I pointed it at 50 popular developer/tech blogs and let it run.

Here's what I learned about the engineering and what was hiding in plain sight.

1. Why Static HTML Parsing Fails

curl and regex-based scrapers give you the server-rendered shell. That's it.

The things that actually matter for a tracking audit are almost never in that shell:

  • Tag managers (GTM, GTAG, etc.) load a small bootstrap script, then fetch and execute dozens of further tags at runtime.
  • Pixels and beacons are injected via document.createElement after hydration, so they don't appear in the source.
  • Consent-state logic decides which cookies to drop — and that decision happens in JS, not in HTML.
  • Shadow scripts (analytics heartbeats, session replay, A/B test SDKs) often fire on a timer, well after the page "loads."

Takeaway: if your audit tool isn't executing JavaScript in a real engine, you're auditing a screenshot of the page, not the page itself.

2. The Architecture (Building the Auditor in Go)

Pageguard is deliberately split into three layers so the heavy lifting stays isolated:

Layer Responsibility
Orchestrator BFS crawl, scope rules, robots.txt respect, worker pool
Browser controller Drives a headless Chromium (Playwright stdio worker) per page, captures network + DOM + cookies
Rule engine Declarative, plugin-based validation: allowed domains, unauthorized cookies, unexpected beacons

The core is dependency-light (stdlib only) so it builds and tests offline; the browser and store sit behind interfaces. The interesting part for this post is the concurrent worker pool — I needed to audit many pages in parallel without letting 50 Chromium instances melt my RAM.

Here's the worker pool, capped with a semaphore and errgroup so a single hung page can't wedge the whole run:

// audit.go
package audit

import (
    "context"
    "sync"

    "golang.org/x/sync/errgroup"
)

type Job struct {
    URL  string
    Mode string // "consent-pending" | "consent-accepted"
}

// auditSite runs one page through the headless browser and returns findings.
func auditSite(ctx context.Context, b *Browser, job Job) (*Report, error) {
    page, err := b.NewPage(ctx, job.URL, job.Mode)
    if err != nil {
        return nil, err
    }
    defer page.Close()
    return page.Validate(ctx)
}

// Run audits a batch of URLs with a bounded number of concurrent browsers.
// The semaphore is what keeps memory from spiking when N is large.
func Run(ctx context.Context, b *Browser, jobs []Job, concurrency int) (<-chan *Report, error) {
    out := make(chan *Report)

    g, ctx := errgroup.WithContext(ctx)
    sem := make(chan struct{}, concurrency) // <-- the actual limiter

    go func() {
        defer close(out)
        g.Wait()
    }()

    for _, job := range jobs {
        job := job
        g.Go(func() error {
            select {
            case sem <- struct{}{}:
            case <-ctx.Done():
                return ctx.Err()
            }
            defer func() { <-sem }()

            rep, err := auditSite(ctx, b, job)
            if err != nil {
                return err
            }
            select {
            case out <- rep:
            case <-ctx.Done():
                return ctx.Err()
            }
            return nil
        })
    }

    return out, nil
}

Two details matter here:

  • The semaphore, not the goroutine count, is the memory control. errgroup fans out, but sem is what bounds how many Chromium tabs are alive at once.
  • Mode is the audit's whole point. I run every page twice — once with consent pending (fresh context, no cookies) and once with consent accepted — so I can see exactly what a site drops before you've said yes.

3. Intercepting the Network & Validating Against Rules

The browser controller listens to the DevTools Protocol network stream and feeds every request into the rule engine. With chromedp this looks like:

// browser.go
package browser

import (
    "context"
    "net/url"

    "github.com/chromedp/cdproto/network"
    "github.com/chromedp/chromedp"
)

// observe wires up network + cookie capture for a single page and hands
// each observed request to the rule engine as it happens.
func (p *Page) observe(ctx context.Context, engine *RuleEngine) {
    chromedp.ListenTarget(ctx, func(ev interface{}) {
        switch e := ev.(type) {
        case *network.EventRequestWillBeSent:
            u, err := url.Parse(e.Request.URL)
            if err != nil {
                return
            }
            // Each rule inspects the host + path and emits a finding.
            engine.CheckRequest(ctx, e.Request.Method, u)

        case *network.EventResponseReceived:
            // Set-Cookie headers are where "pre-consent" tracking lives.
            for _, h := range e.Response.Headers {
                if h.Name == "Set-Cookie" {
                    engine.CheckCookie(ctx, h.Value, u.Host)
                }
            }
        }
    })
}

The rule engine is plugin-based and declarative. A policy is just data (YAML/JSON), so a team can encode its own allow-list without touching Go:

# policy.yaml
allowed_domains:
  - "example.com"
  - "assets.example.com"

forbidden_cookies_before_consent:
  - "ga_client"
  - "gtm"
  - "_fbp"

max_third_party_requests: 12

Every observed request and cookie is matched against these rules; violations become findings with the URL, cookie name, and consent state attached as evidence.

4. The Experiment & Surprising Findings (50 Tech Blogs)

I audited 50 popular developer/tech blogs. Each was run in both consent states. Site names are anonymized (Blog #1, #14, etc.) on purpose — the goal is to surface engineering patterns, not to shame specific teams.

Metric (across 50 blogs) Result
Median third-party requests per page 19
Max third-party requests observed (single page) 47
Blogs that dropped a tracking cookie before consent 34 / 50 (68%)
Blogs still firing a legacy/abandoned analytics tag 12 / 50 (24%)
Blogs whose tracking payload exceeded 120 KB 9 / 50 (18%)
Median time-to-networkidle 6.4 s

Four findings stood out:

  1. Cookies set before "Accept." 68% of blogs dropped at least one analytics or tag-manager cookie in the pending state. A handful did it via a first-party Set-Cookie on the document response itself — invisible in the JS, visible only in the response headers. This is the single most common GDPR/CCPA foot-gun I found.

  2. Zombie analytics tags. ~24% of blogs were still firing a tag for a product they'd clearly migrated off (a deprecated analytics endpoint, an old A/B test SDK). The beacon returned a 200 and kept polling on a timer, so networkidle never fired cleanly on those pages.

  3. The tag manager is the black box. On the heaviest pages, the first third-party script was the tag manager, and everything after it was opaque. You literally cannot know the final request count by reading the HTML — the manager decides at runtime.

  4. Tracking is a real perf tax. The 18% of pages with >120 KB of tracking payload added a measurable chunk to load time. On a content blog, that's the reader's attention being spent on someone else's SDK.

5. The Hardest Engineering Bottlenecks

Running 50 headless browser audits concurrently in Go is where the fun turned into pain. Three problems dominated:

  • Chromium RAM spikes & zombie processes. Each tab is a real process tree. I tracked RSS per instance and force-recycled the browser once it crossed a threshold, rather than relying on a fixed "N pages per browser" rotation — some pages leak far more than others. I also had to make teardown survive SIGKILL so a hung run didn't leave orphaned Chrome processes eating memory.

  • networkidle is a trap. networkidle means "no network connections for 500 ms." A page with a 30-second analytics heartbeat, a WebSocket, or a long-polling support widget never goes idle, so a naive wait hangs until timeout and fails a page that was usable 12 seconds ago. My fix: wait on domcontentloaded + an explicit readiness predicate (no DOM mutations for ~500 ms), with a wall-clock cap — and mark a timeout rather than fail the page.

  • Aggressive bot protection & rate limits. Several blogs sit behind WAFs that fingerprint headless Chromium. I had to randomize viewport/UA, throttle per-domain, and back off on 403/429 — and accept that a small number of sites simply refuse to be audited headlessly.

6. Key Takeaways

If you want to audit your own site, here are three rules that actually hold up:

  1. Audit in a real browser, in both consent states. Run every page with consent pending and accepted; the diff is where your compliance risk lives. Static HTML will not show you this.
  2. Wait on a readiness predicate, not networkidle. Use domcontentloaded + a DOM-stability check with a hard timeout, and treat a timeout as a finding, not a crash.
  3. Make it a CI gate, not a one-off. Encode your allow-list and cookie policy as declarative data, run it on every deploy, and fail the build when a new unauthorized domain or pre-consent cookie appears.

I've open-sourced all of this as Pageguard — the Go orchestrator, the headless-browser controller, the plugin-based rule engine, and the consent-state simulation. If you build governance or martech tooling, I'd love a review or a PR.

👉 https://github.com/bhkbdbhatt/pageguard

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.