Redact Before You Log: A Testable Pattern for LLM Prompt and Response Logs

Logging every prompt and model response is one of the first things teams add to an LLM feature, and for good reason: when an answer goes wrong, the log is often the only way to see what the model was actually given. The catch is that prompts are full of whatever users typed, and responses can echo it back. A debug log for an AI feature can quietly become the least protected copy of your most sensitive data.

This post walks through a small, testable pattern: redact at one boundary, before anything reaches a log sink.

Why LLM logs leak more than ordinary logs

Ordinary request logs carry fairly predictable fields. LLM logs carry free text, often pasted from emails, tickets or documents. That text can include exactly the items the OWASP Logging Cheat Sheet says should usually be removed, masked, hashed or encrypted before logging: access tokens, passwords, payment card data, government identifiers and sensitive personal data such as health information.

The OWASP Top 10 for LLM Applications (LLM02:2025, Sensitive Information Disclosure) lists tokenization and redaction, using techniques like pattern matching, among its mitigations. It also notes that telling the model not to return certain data in the system prompt "may not always be honored". So the output side needs the same treatment as the input side.

Draw the boundary

The model usually needs the raw prompt to do its job. Your logs usually don't. Make that split explicit:

  user input
      |
      v
  +--------+     raw prompt      +-----------+
  |  app   | ------------------> | model API |
  |        | <------------------ |           |
  +--------+     raw output      +-----------+
      |
      v
  +----------+   redacted event   +----------+
  | redact() | -----------------> | log sink |
  +----------+                    +----------+

The rule: nothing goes to the log sink except through redact(). No print(prompt) in a handler, no logging the full provider response object "just for now".

A small redaction function

Start with patterns for the things you know are dangerous in your domain. This version (Python 3.9+) counts what it removed, which is useful later:

import re

PATTERNS = {
    "email": re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}"),
    "us_ssn": re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
    "card": re.compile(r"\b\d(?:[ -]?\d){12,15}\b"),
    "bearer": re.compile(r"(?i)\bbearer\s+[A-Za-z0-9._~+/=-]{16,}"),
}

def redact(text):
    counts = {}
    for name, pattern in PATTERNS.items():
        text, n = pattern.subn(f"[REDACTED:{name}]", text)
        if n:
            counts[name] = n
    return text, counts

Typed placeholders such as [REDACTED:email] keep the log readable. You can still tell that the user supplied an email address, which often matters when you're debugging why the model behaved a certain way.

Pseudonymize identifiers instead of dropping them

You still need to group events by user or session. A keyed hash (HMAC) gives you a stable token without storing the raw identifier. Keep the key in your secrets manager, not in code or in the log pipeline:

import hashlib
import hmac

def pseudonym(value, key):
    return hmac.new(key, value.encode(), hashlib.sha256).hexdigest()[:16]

A plain unkeyed hash of a user ID or email is easy to reverse by hashing likely values, so use the key.

Emit one structured event

Structured JSON events also help with log injection. The cheat sheet warns that unstructured text formats can let an attacker forge entries with carriage return or line feed characters.

import json

RULESET_VERSION = "redact-v3"

def log_llm_call(logger, *, user_id, model, prompt, output, key):
    safe_prompt, prompt_hits = redact(prompt)
    safe_output, output_hits = redact(output)
    event = {
        "event": "llm_call",
        "ruleset": RULESET_VERSION,
        "user": pseudonym(user_id, key),
        "model": model,
        "prompt": safe_prompt,
        "output": safe_output,
        "redactions": {"prompt": prompt_hits, "output": output_hits},
    }
    logger.info(json.dumps(event))

Two fields deserve a note:

• ruleset records which version of your redaction rules produced the event. When you add a pattern next month, you can tell which older events were processed without it.
• redactions counts give you a cheap signal. A sudden spike in card hits in prompts tells you users are pasting something you didn't expect.

Test it like any other security control

Redaction that isn't tested tends to break silently when someone edits a regex. These run under pytest:

def test_redacts_known_patterns():
    text, hits = redact(
        "Email [email protected], SSN 123-45-6789, "
        "card 4111 1111 1111 1111, Bearer abcdefghijklmnop1234"
    )
    for secret in ("[email protected]", "123-45-6789", "4111",
                   "abcdefghijklmnop1234"):
        assert secret not in text
    assert hits == {"email": 1, "us_ssn": 1, "card": 1, "bearer": 1}

def test_leaves_ordinary_text_alone():
    text, hits = redact("Order 42 shipped on 2026-11-04.")
    assert hits == {}

Add a case each time you find a miss in real traffic, using synthetic values rather than the real data.

Know the limits

Pattern matching catches well-formed identifiers. It won't catch a person's name, a diagnosis described in plain words, or an address written as a sentence. For those, teams add entity-recognition tools, allow-lists of fields that may be logged, or a policy of logging only metadata for certain routes. None of these is perfect, so also:

• restrict who can read LLM logs, following least privilege;
• set a retention period and actually delete old logs;
• make sure debug flags, crash reporters and tracing tools don't capture raw payloads around your redaction step.

Where this fits

At Obligra we build Obligra Verify™, a system of record for AI-assisted decisions, currently in public preview. It's designed to retain the evidence your application selects, which is why we think about this boundary a lot: what you choose to keep matters as much as how you keep it.

If you take one thing from this post, make it the single boundary. One redact() call that every log path must go through is much easier to test and audit than redaction scattered across handlers.

This article was generated by an AI agent (Claude) for the Obligra Verify team and checked against the linked sources.