Debug Expectations After Network Drop: Realtime Presence Ghosts and Scoped Leases

When realtime presence shows ghost users after a network drop, debug the server's expiry rule before blaming the transport. A device dashboard should treat presence as a short lease granted by the server, never as a fact asserted by a browser; pick the transport that fits the path, but make expiry, identity scope, and reconnect reconciliation independent of that choice.

Recommendation: issue a narrowly scoped dashboard token, derive presence from server-observed activity, expire it on a monotonic lease, and replace local state from an authoritative snapshot after every reconnect. This makes a silent network drop boring. That is the goal.

Why does realtime presence show ghost users after a network drop?

A clean close is useful evidence. It is not a reliable cleanup primitive. A laptop can sleep, a radio can lose coverage, or a middlebox can discard state without delivering a final message. WebRTC exposes connection-state transitions, but even a reported disconnected state can be temporary; failed is a stronger signal and still does not prove what the application should display. Transport state and business presence are different clocks.

The practical failure is usually an immortal boolean. The server receives online: true, stores it, and waits for offline. When the second message never arrives, the dashboard shows a ghost. Reconnect can make it worse: a new session appears while the stale session remains under another connection identifier. Now picture the debugging sequence: the socket list contains only the replacement connection, the device table still contains the old boolean, and the dashboard cache has both. Restarting the client appears to fix nothing because the incorrect authority lives elsewhere. Trace the identity and expiry attached to each layer before tuning a heartbeat.

Ghosts linger.

Do not ask a disconnect callback to prove absence. Ask whether the server has observed valid activity inside a bounded interval. For a device status dashboard, keep at least three values distinct: device identity, connection or session identity, and lease expiry. A device may have two sessions during handoff. Only the server decides whether either session still holds a live lease.

Token scope and client trust are the decision criteria

Presence data looks harmless until it reveals working hours, site activity, or fleet outages. A dashboard token should name the tenant and the rooms or device groups it may read. It should not inherit device-write authority, and a browser-supplied tenant identifier should never widen its scope. Verify authorization when the subscription starts and again when policy or membership changes. The same rule applies to publishing. A device credential may renew presence for its own device identity. It must not be able to renew another device by changing a field in the payload. Bind the authenticated principal to the server-side record, then ignore identity claims that merely arrived in JSON. This costs a little glue in the authorization layer. I would still pay it. A generic subscribe(room) API is quick to demo, but time-to-first-call is the wrong benchmark if the second call can cross a tenant boundary. The useful DX metric is how many concepts an integrator must configure before least privilege is the default. Aim for tenant, audience, allowed action, and expiry. Stop there.

Trust less.

The browser may render presence; it should not adjudicate it. Treat heartbeat timestamps, reconnect counters, and claimed status from the client as inputs subject to server validation. Use the server's clock for lease expiry. Otherwise a skewed or manipulated clock can stretch a session beyond policy.

Choose the lease interval from measured network behavior and product tolerance, not folklore. A 30-second expiry may be acceptable for a coarse fleet view and irritating for devices that routinely pause longer. A five-second expiry detects absence faster but multiplies renewal traffic and false offline transitions. Benchmark the actual reconnect distribution, then document the maximum stale window as part of the UI contract. There is no magic number.

Render uncertainty honestly. online, offline, and unknown are often more accurate than a binary dot. During snapshot recovery, unknown prevents cached green indicators from masquerading as fresh server state.

Measure both.

A small lease core

Keep the state machine smaller than the transport adapter. The following TypeScript sketch accepts only identities established by the authentication layer, extends a session lease, and folds multiple sessions into one device view. The numbers are examples for testing the mechanism, not operational defaults.

type SessionKey = `${string}:${string}`;

type Lease = {
  tenantId: string;
  deviceId: string;
  sessionId: string;
  expiresAtMs: number;
};

const leases = new Map<SessionKey, Lease>();

function renewLease(
  principal: { tenantId: string; deviceId: string },
  sessionId: string,
  nowMs: number,
  ttlMs = 30_000,
): Lease {
  const key: SessionKey = `${principal.deviceId}:${sessionId}`;
  const lease = {
    tenantId: principal.tenantId,
    deviceId: principal.deviceId,
    sessionId,
    expiresAtMs: nowMs + ttlMs,
  };
  leases.set(key, lease);
  return lease;
}

function snapshot(tenantId: string, nowMs: number): Set<string> {
  const online = new Set<string>();

  for (const [key, lease] of leases) {
    if (lease.expiresAtMs <= nowMs) {
      leases.delete(key);
      continue;
    }
    if (lease.tenantId === tenantId) online.add(lease.deviceId);
  }

  return online;
}

Production storage needs an atomic expiry strategy and a single rule for concurrent renewals. The important invariant is portable: only an unexpired server record contributes to the snapshot. A close event may delete a lease early, but correctness cannot depend on receiving it.

On reconnect, discard the local presence map and install a scoped snapshot before applying subsequent deltas. Sequence those deltas per subscription so a delayed pre-snapshot update cannot overwrite newer state. If the sequence has a gap, fetch another snapshot. This is less clever than merging arbitrary caches. It is also easier to test.

Test the ugly transitions: renewal followed by total packet loss, sleep longer than the lease, two simultaneous sessions for one device, token expiry during reconnect, a changed tenant membership, duplicated messages, and a delta arriving before snapshot installation. Assert the stale-time bound with a fake clock. Wall-clock tests are slow and flaky.

When is the runner-up enough?

The lease approach has limitations. It adds renewal traffic, state, expiry work, and a bounded period in which an absent device can still look online. It is not suitable as the sole signal for an emergency stop or another action that demands immediate, positive confirmation. In those cases, require a fresh command acknowledgment over an authenticated channel; presence remains context, not proof.

Disconnect-driven cleanup is reasonable when presence is decorative, users tolerate stale indicators, and the system already has a bounded fallback sweep. It can remove cleanly closed sessions faster than lease expiry. Keep the lease anyway; the disconnect path is an optimization. The trade-off is explicit: less state means weaker guarantees after silent failure.

A transport-native connection state is also useful in a peer-to-peer collaboration surface because it tells the local UI about path health. The WebRTC specification defines new, connecting, connected, disconnected, failed, and closed states for the peer connection. Those states describe that connection. They do not authorize a tenant subscription, reconcile a dashboard snapshot, or establish durable device presence.

For a tiny internal dashboard with no cross-tenant boundary, the runner-up may save implementation time. Write down the accepted stale window and add an expiry sweep before calling it done. Once presence triggers paging, automation, or access decisions, move authority back to a scoped server lease.

Ship the invariant, then measure it

Instrument lease renewals, expirations, reconnect snapshot latency, sequence gaps, and the count of sessions folded into each device. Do not log bearer tokens or full credentials. Alert on distributions and sustained anomalies rather than one missed heartbeat.

The deployment test is direct: cut the path without sending a close frame, advance beyond the configured lease, and verify that the authoritative snapshot drops the session. Restore the path and verify that a fresh authenticated session replaces local state without duplicating the device. Repeat with an expired token and a token scoped to another tenant; both subscriptions must fail closed.

The decision rule stays compact: transport events improve responsiveness, while scoped leases and snapshot reconciliation provide correctness. Keep client claims weak, keep configuration short, and benchmark the stale window users actually experience.

References

• https://www.w3.org/TR/webrtc/