Canaries Show What Your Prompt Actually Sends

Assume every outbound request leaves your control the moment it is serialized, because you will never audit the provider's disk. The only honest way to learn what your prompt carried is to plant harmless unique strings in your own files and then read the exit log. This walkthrough builds that rehearsal as a small repeatable artifact you can run before pointing any client at a hosted model.

A unit test on your redactor tells you what the function believes it removed. A canary tells you what the socket actually carried, including headers, retries, and the error paths you forgot to read. The gap between those two answers is where leaks live, and it is usually a concatenation nobody deliberately wrote.

Build a canary pack no real system would accept

Create a directory of small files whose contents are obviously fake but structurally similar to the things you actually care about. Keep them out of git, keep them out of your prompt templates, and give each one a label you can grep for later. The labels matter more than the contents, since three distinct canaries let you separate a stray environment file from a chatty logger.

mkdir -p .canaries
echo -n 'AKIACANARYNOTREAL00'          > .canaries/aws_like_key
echo -n 'canary.internal.example'      > .canaries/internal_host
echo -n '[email protected]' > .canaries/contact
printf '.canaries/\n' >> .gitignore

Rotate the pack for each run so an old hit can never be mistaken for a fresh one, and keep values short enough that substring matching stays unambiguous. If your stack already runs a secret scanner, point it at the canary pack and confirm it flags every file, which tells you the shapes are realistic.

Put a single door on the outbound path

You cannot log what you cannot intercept, so give your client exactly one transport and let that transport write the record. The sketch below wraps an HTTP client, hashes each body, and checks it against the canary pack before the bytes leave the process. It is deliberately dumb: no parsing, no cleverness, just one line per request and an optional hard stop.

# egress_gate.py — illustrative sketch; adapt to your client's transport API.
import json, os, hashlib, httpx

CANARY_DIR, LOG = ".canaries", "egress.jsonl"

def load_canaries(root=CANARY_DIR):
    pats = {}
    for name in sorted(os.listdir(root)):
        raw = open(os.path.join(root, name), "rb").read().strip()
        if raw:
            pats[name] = raw
    return pats

CANARIES = load_canaries()

class LoggingTransport(httpx.BaseTransport):
    def __init__(self, inner):
        self.inner = inner

    def handle_request(self, request):
        body = request.read()
        hits = sorted(k for k, v in CANARIES.items() if v in body)
        record = {
            "host": request.url.host,
            "path": request.url.path,
            "method": request.method,
            "bytes": len(body),
            "sha256": hashlib.sha256(body).hexdigest()[:16],
            "canary_hits": hits,
        }
        with open(LOG, "a") as f:
            f.write(json.dumps(record) + "\n")
        if hits and os.environ.get("STRICT") == "1":
            raise RuntimeError(f"blocked egress: {hits}")
        return self.inner.handle_request(request)

def client():
    return httpx.Client(transport=LoggingTransport(httpx.HTTPTransport()))

Run the first pass with STRICT unset, because your goal in that pass is observation rather than prevention. Turn it on for the second pass and you have an egress guard that fails closed, which is the property you want in CI rather than in a slide. The log file becomes the artifact under review, and its hash column lets you confirm that two runs sent identical bytes.

STRICT=0 python run_prompt_job.py   # observe
STRICT=1 python run_prompt_job.py   # enforce
cat egress.jsonl

Read the log like an incident, not a checklist

A useful log is boring to read and specific about one thing: which request carried which planted string. The sample below shows three requests from one job, and only the middle line deserves attention.

{"host":"api.example-model.dev","path":"/v1/chat","method":"POST","bytes":8421,"sha256":"9f1c0e77a2b34d10","canary_hits":[]}
{"host":"api.example-model.dev","path":"/v1/chat","method":"POST","bytes":9120,"sha256":"3ab8821cc0d4f7e9","canary_hits":["internal_host"]}
{"host":"api.example-model.dev","path":"/v1/chat","method":"POST","bytes":546,"sha256":"c7d0aa91b2e4f013","canary_hits":[]}

Empty hit lists are a good sign and a weak one, because they only prove that your planted strings did not cross. Requests whose byte count rises between retries deserve a second look, since retry handlers often serialize a richer error object than the original call did. The triage below is less about pass or fail than about which code path is doing the talking.

Decide what never goes out, before the run

Customer identifiers, live credentials, internal hostnames, and raw production dumps belong in the never column, and no amount of prompt engineering relocates them. Schemas, error codes, synthetic fixtures, and stack traces with absolute paths replaced stay useful and stay sendable. Write the table down, review it whenever someone adds an exporter, and treat a new exporter as a new trust boundary by default.

The decision is cheap while it is a table and expensive after an incident, so keep the table in the repository next to the canary pack. When a reviewer asks whether a prompt is safe, the honest answer references a run and a log rather than a feeling.

Where to rehearse it

If you would rather not provision a second machine just to watch an egress log, the MonkeyCode project offers free model access together with a free server option, and this walkthrough is written so it can run there without changes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the free server as a foreign room anyway: install the logging transport on it, keep the canary pack out of any synced folder, and check the project's current terms for limits, because free tiers change without notice.

Limits you should not paper over

A canary pass proves what one run sent through one transport, and it says nothing about telemetry inside an SDK you never wrapped. Native gRPC clients, pinned TLS, and vendor-specific upload paths can bypass an environment proxy entirely, so audit the client configuration instead of trusting the proxy by default. The log is sensitive in its own right, since a body hash plus a path identifies traffic, so store it with the same care as the prompts it describes.

Who should skip this

If your organisation requires a DLP appliance or an egress allowlist, this rehearsal is a complement at best and a distraction at worst. Teams that are not permitted to create realistic-looking fake secrets, even invalid ones, should ask their security group for synthetic tokens instead. And if you only ever run models on hardware you own, canaries still help, but the threat you are modelling has a different shape.

Plant the canaries, run once with the guard open and once with it closed, then keep the log. That habit tells you more about your prompt pipeline than any redaction test you never watched fail.