Your Own Workers Are the Bot Farm: Session Arbitration for Parallel Scrapers

The incident report that had no attacker

We were six months into a price-monitoring pipeline when the alerts started looking embarrassing. Sporadic 429s after the login step. A CAPTCHA interstitial on a site that had never served one. p95 latency that doubled between 02:00 and 04:00 UTC for no reason anyone could name.

The team did what teams do. We blamed the proxy pool. We swapped providers, then swapped back. We lowered concurrency globally, which made the pipeline slow and the 429s rarer but still present, which is the worst possible outcome because now you have paid twice for the same bug.

What actually fixed it was a twenty-line logging change. We started recording, for every request, the exit IP (via a simple echo endpoint), the session token we had used, and the wall-clock window the request occupied. Then we plotted, per exit IP, how many requests were in flight at the same instant.

The answer was uncomfortable: us. At any given moment, two or three of our own workers were sitting on the same residential exit IP at the same time, talking to the same host family, with completely different TLS sessions and cookie jars. We had built a distributed bot farm. The farm had our logo on it.

What a "session" actually is on a gateway endpoint

Rotating residential services usually hand you one hostname and one port - a gateway - and encode all of your identity and intent into the username. Thordata works this way; its docs show an endpoint like:

# Shape from the Thordata docs (User & Pass auth guide).
# Generate your real username string in the dashboard's endpoint
# generator; this is the format it emits:
curl -x "http://td-<youruser>-sessid-123456-sesstime-90:<password>@t.pr.thordata.net:9999" \
     "https://ipinfo.thordata.com"

The sessid is the session identifier and sesstime pins the exit IP for that session for 1 to 90 minutes. Two requests carrying the same sessid within that window leave from the same household IP; change the id and you get a fresh lottery draw. Most rotating residential gateways work this way in some flavor - a session token embedded in the username pins the exit IP for a bounded window. The mechanism, not the vendor, is the point.

Now here is where the trouble hides. Marketing pages sell "unlimited ports" or "unlimited concurrent connections", and teams read that as "run as many workers as you like". What the gateway actually enforces is per-session concurrency: connections sharing one session id compete with each other and eventually queue or get rejected. And what the target site sees is per-IP request overlap, which it treats as a strong signal - a real residential line usually carries one family, so two simultaneous, unrelated TLS handshakes from your exit IP looks like a shared credential or a datacenter wearing a costume.

So there are two collision surfaces: your provider's session limits, and every target's IP-window heuristics. Both punish the same mistake, and neither one tells you in the response body that self-collision is what happened.

Three ways to hand out sessions

Random rotation per request. Never reuse a session id; every request gets a new IP. Kills self-collision on paper, but destroys anything stateful: logins, add-to-cart flows, pagination that expects the same A/B bucket, or a page that served you a CSRF token from one IP and validates it from another. It also wastes the TLS and cache warmth you paid for. Fine for dumb bulk SERP fetches; terrible for multi-step workflows.

One fixed session per worker. Thread 3 always uses sessid-883412. Simple, sticky, and one bad restart away from chaos: nothing stops process B from reusing a sessid that process A picked, especially when ids are generated from timestamps or random digits and survive in logs. When it breaks, it breaks the whole run, and a flagged sticky IP stays in the rotation you can't see.

A lease pool with a TTL. One arbiter process hands out session ids, guarantees at most one holder at a time, expires leases that go silent, and burns ids after errors. This is the version worth the twenty lines.

The arbiter, in code

Standard library only. The core promise: acquire() never returns the same session id to two holders simultaneously, and any lease older than its TTL is reclaimed even if its holder died.

import collections
import random
import threading
import time
from contextlib import contextmanager


class SessionLeasePool:
    """Hand out proxy session ids so that no id has two live holders."""

    def __init__(self, ttl_seconds, max_sessions):
        self.ttl = ttl_seconds
        self.max_sessions = max_sessions
        self._cond = threading.Condition()
        self._alive = {}    # session_id -> expiry (time.monotonic)
        self._holder = {}   # session_id -> holder name

    def acquire(self, holder, timeout=30.0):
        deadline = time.monotonic() + timeout
        with self._cond:
            while True:
                self._reap()
                for sid in self._alive:
                    if sid not in self._holder:
                        self._holder[sid] = holder
                        return sid
                if len(self._alive) < self.max_sessions:
                    sid = str(random.randint(100000, 999999))
                    self._alive[sid] = time.monotonic() + self.ttl
                    self._holder[sid] = holder
                    return sid
                remaining = deadline - time.monotonic()
                if remaining <= 0:
                    raise TimeoutError("no session lease within timeout")
                self._cond.wait(remaining)

    def release(self, sid, retire=False):
        with self._cond:
            self._holder.pop(sid, None)
            if retire:
                self._alive.pop(sid, None)
            self._cond.notify_all()

    def _reap(self):
        now = time.monotonic()
        for sid in [s for s, exp in self._alive.items()
                    if exp <= now and s not in self._holder]:
            self._alive.pop(sid, None)

Pair it with a collision counter so you can see the property holding, rather than trusting it:

class InFlightGauge:
    def __init__(self):
        self._lock = threading.Lock()
        self._counts = collections.Counter()
        self.collisions = 0

    @contextmanager
    def slot(self, sid):
        with self._lock:
            self._counts[sid] += 1
            if self._counts[sid] > 1:
                self.collisions += 1
        try:
            yield
        finally:
            with self._lock:
                self._counts[sid] -= 1

And the worker loop, where fetch_one is any function that takes the lease, performs exactly one proxied exchange, and returns the HTTP status:

pool = SessionLeasePool(ttl_seconds=50, max_sessions=6)
gauge = InFlightGauge()

def worker(name):
    while work_remaining():
        sid = pool.acquire(holder=name, timeout=60)
        try:
            with gauge.slot(sid):
                status = fetch_one(build_proxy_url(sid))
            if status in (403, 429):
                pool.release(sid, retire=True)   # burn it, mint a new id
            else:
                pool.release(sid)
        except BaseException:
            pool.release(sid, retire=True)
            raise

build_proxy_url(sid) should take the exact username template from your provider's endpoint generator and substitute only the session field. For Thordata's shape above, that means replacing the value after sessid- with sid, keeping sesstime between 1 and 90 minutes and always strictly below the lease TTL - which is why the pool example uses ttl_seconds=50 against the docs' 90-minute maximum window. A lease must expire before the provider's IP pin does, never the reverse.

Does the counter move?

Run the same task list twice: once with sessid drawn from a naive per-worker dict (or a shared timestamp, or anything without arbitration), once through the pool, and print gauge.collisions at the end. The unarbitrated mode will report a number greater than zero on any machine with more workers than session ids - that is not a measurement I need to fake for you, it is a pigeonhole. The pooled mode reports zero by construction: the gauge and the pool make the same promise, so if the gauge ever ticks after you deploy this, you have a second arbiter process you forgot about, which is exactly the bug class worth finding.

The property is structural; the value is in having a number on the dashboard that catches the day you break it - when someone shells out a retry helper from a second process, or a k8s CronJob overlaps its predecessor and now two pods run "the single" arbiter.

The traps, in order of how long they cost me

TTL clock drift. Use time.monotonic(), never time.time(). NTP stepping the wall clock mid-run will mass-expire or immortalize your leases; both are weird at 3 a.m. and invisible by morning.

Released is not reset. Retiring a lease frees the id in your process. The provider may still hold that IP warm for the remainder of its sticky window, and targets remember recent neighbors. If a sessid just drew a 429, mint a fresh random id; do not recycle the retired number and expect a clean start.

One worker, many in-flights. The lease guarantees one holder, not one request. If your "worker" is an asyncio loop with five coroutines per thread, five requests share one lease and you have rebuilt the bot farm in a smaller room. Either take a lease per concurrent in-flight exchange, or set max_sessions above your true concurrency and let acquirers wait.

Geo drift mid-workflow. A sticky session pins an IP, not a city, for as long as the household's gateway does. Mid-checkout city changes are rare but exist; if the flow matters, verify the exit country at workflow start and start over on drift. The same echo endpoint you use for the gauge (https://ipinfo.thordata.com, or any geo echo) costs one request per workflow.

Provider-side caps are not published per-plan where you'd expect. Before scaling, ramp one sessid from 1 to 32 concurrent connections and watch where latency knees or rejections begin. That number, not the "unlimited ports" line, is your real max_sessions per credential.

Pigeonhole at low session counts. With max_sessions=6 and 12 workers, half the pool waits on acquire(); your throughput is now sessions x per-session throughput, not worker count. Set workers to roughly what the cap measurement above justifies, and let the queue absorb the rest.

The boring summary

Self-collision is invisible in every log you already have: the target returns 429 like a stranger would; the gateway queues like the network would. The fix is boring - one arbiter, TTLs shorter than the provider's pin, and a counter that ticks whenever the invariant breaks. Boring is good, because the alternative is what we did: six months of blaming a pool that was doing exactly what we asked, in increasing triplicate.

On the money, briefly, since it's the usual reason teams over-concurrent: collision retries re-download pages you already had. Thordata's residential pricing page showed "starts from $0.65/GB" when I checked it today (2026-09-18), so the wasted GB read as a rounding error while the flagged sessions read as an incident. The arbitration layer cost us nothing but pride.

Disclosure: I work with Thordata (residential proxies here), so treat my vendor loyalties accordingly - though the collision math above is provider-agnostic and the pool code is yours to reuse.