Your Healthcare Voice AI Agent Needs a Compliance Gate (Before It Costs You $50,000)

An $800 voice agent cost a healthcare founder $50,000.

I heard this directly from him last week. The agent worked. The voice was natural and the integration was clean. It had no rules for how often it could call the same patient or during which hours. It called the same people repeatedly, some calls landed outside permitted hours, and one complaint turned into a $50,000 bill.

Nothing crashed, and no model misbehaved. This was an engineering omission, and it's the kind we can prevent.

Here's how to design voice agents so that doesn't happen.

The core principle: the model never decides if a call is allowed

LLMs are good at conversation. They are a bad place to enforce legal constraints. A prompt that says "don't call more than once a day" is a suggestion, not a control.

Compliance has to live in deterministic code that runs before the agent is ever invoked.

Trigger (form abandoned / checkout dropped)
        │
        ▼
 ┌──────────────────┐
 │  Call scheduler  │  queue + delay (e.g. 15-20 min)
 └────────┬─────────┘
          ▼
 ┌──────────────────┐
 │ COMPLIANCE GATE  │  consent, DNC, time window,
 │ (deterministic)  │  frequency caps, call type
 └────────┬─────────┘
     deny │ allow
   ┌──────┘   │
   ▼          ▼
 requeue   Voice agent ──► tools (CRM/EHR, SMS, human handoff)
 or drop        │
                ▼
          Audit log (every decision, every call)

If the gate says no, the call is requeued for the next valid window or dropped. The agent never gets a vote.

The gate

from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

MAX_CALLS_PER_DAY = 1
MAX_CALLS_PER_WEEK = 3
# Conservative window. Federal rules set the outer limit and
# some states are stricter. Confirm with counsel.
WINDOW_START, WINDOW_END = 9, 19  # recipient-local hours

def can_call(patient, call_log, call_type, now_utc=None):
    now_utc = now_utc or datetime.now(ZoneInfo("UTC"))
    local = now_utc.astimezone(ZoneInfo(patient.timezone))

    if patient.do_not_call or not patient.has_consent(call_type):
        return False, "no_consent_or_dnc"

    if not (WINDOW_START <= local.hour < WINDOW_END):
        return False, "outside_calling_window"

    day_ago = now_utc - timedelta(days=1)
    week_ago = now_utc - timedelta(days=7)
    mine = [c for c in call_log if c.patient_id == patient.id]

    if sum(1 for c in mine if c.at > day_ago) >= MAX_CALLS_PER_DAY:
        return False, "daily_cap"
    if sum(1 for c in mine if c.at > week_ago) >= MAX_CALLS_PER_WEEK:
        return False, "weekly_cap"

    return True, "ok"

A few details that matter more than they look:

• Use the recipient's timezone, not the server's. A form filled at 3 a.m. their time gets queued, not called.
• Pass call_type. Marketing and transactional calls have different consent rules and should use separate numbers.
• Return a reason code. You'll want it in the audit log.

Concurrency: your caps are only as good as your locking

A naive check-then-call has a race condition. Two workers pick up the same patient at the same moment, both see zero calls, and both dial.

Make the check and the "reserve a slot" step atomic:

# Pseudocode: reserve before dialing, release on failure
with db.transaction():
    ok, reason = can_call(patient, call_log_for(patient), call_type)
    if not ok:
        audit.log(patient.id, "denied", reason)
        return
    call_log.insert(patient_id=patient.id, at=now, status="reserved")
# dial only after the reservation commits

Use a unique constraint or row-level lock per patient so duplicate triggers (webhook retries are common) can't cause duplicate calls. Make your trigger handlers idempotent.

Keep PHI out of the conversation

Personalization is what makes these agents effective. The agent that knows where the patient left off converts far better than a generic script. But the personalization layer is also where privacy leaks happen.

• Give the agent only what it needs. First name and form progress, not the full record. Don't put diagnoses in the prompt or tool results unless required.
• Add an output check. Scan agent responses for restricted content before they reach speech, and have the agent fall back to "I can't discuss that on this call" on a match.
• Redact logs and transcripts. Call recordings and transcripts are PHI too. Treat them that way in storage and access control.

Audit everything

When a complaint arrives, your answer depends on your records. Log:

• Every gate decision, with the reason code
• Consent source and timestamp
• Call start, end, outcome, and which number was used
• Handoffs to a human

If you can't show why a call was allowed, you can't defend it.

Infrastructure checklist

• BAA coverage for every component that touches patient data: LLM provider, voice platform, telephony, transcription, storage, logging. Some vendors only offer coverage on certain tiers or via add-ons, so verify each one in writing.
• Branded, verified numbers to avoid spam labeling, and separate numbers for marketing and transactional calls.
• Instant opt-out handling that updates the consent store immediately.
• Human handoff available on every call.

Does it work? A real deployment

We built this pattern for a US telehealth company that acquires patients through paid social ads. A long intake form plus a payment step meant many people dropped off before paying.

The agent calls roughly 15 to 20 minutes after drop-off. It answers questions using the patient's form progress (pulled from their EHR), sends checkout links or discounts by SMS or email, shares testimonials, and hands off to a human on request.

It recovered about 10% of abandoned checkouts and was rolled out across multiple brands. The gate, caps, privacy controls, and audit log are what let it run at that scale without incident.

Takeaways

1. Integration is easy. Compliant integration is the actual project.
2. Never let the model enforce legal rules. Put a deterministic gate in front of it.
3. Make caps race-proof with atomic reservations and idempotent triggers.
4. Minimize and redact PHI in prompts, outputs, and logs.
5. Verify BAA coverage across the whole stack, in writing.
6. Hire for domain experience, not just the lowest quote. The $800 build is rarely the real price.

I build compliant AI and telehealth systems at Enacton, with HIPAA and calling-rule controls designed in from day one. Planning a voice AI project in healthcare? Let's talk before you launch, not after.