How to check with Python that redacted text is really gone from a PDF

A redacted PDF can fail in more ways than "someone drew a black box". The box can be part of the page or an annotation on top of it, redaction marks can be added and never applied, the redaction can be applied and then saved incrementally so the old page is still in the file, and the author's name can sit in the metadata the whole time. All of these look identical on screen.

To see which ones actually leak, I built seven versions of the same one-page form and checked each with three text extractors (PyMuPDF, pypdf and pdfminer.six), a scan of every stream in the file, and the metadata. One of the seven is the file saved by the PDF Editor in the osFoundry desktop app after Find text and redact, Apply and Sanitize. The scripts are at the end; everything ran with Python 3.12, PyMuPDF 2.1, pypdf 6.21.4 and pdfminer.six 20260107 on an Apple M1.

The test form

The form is a direct-deposit authorization generated with PyMuPDF (Example data: name Jordan Ellis, account number 4471 2093 5568, phone (503) 555-0142). The account number appears twice and the name twice, and the author field in the document info is set to the name. The text uses the built-in Times font, so each line in the content stream is a plain TJ with the bytes written as a hex string:

BT
1 0 0 1 236 326 Tm
/tiro 12.5 Tf [<3434373120323039332035353638>]TJ
ET

34 34 37 31 20 32 30 39 33 ... is 4471 2093 ... in ASCII. Keep that in mind for the next section.

A drawn box changes nothing in the text

The "redacted" version most people make draws filled rectangles over the two account numbers. In the content stream that is two new objects appended after everything else:

q
232 322 89.25 17 re
h
0 0 0 RG 0 0 0 rg B
Q

The TJ above is untouched. The rectangle is painted later, so it covers the glyphs on screen and in print, but every extractor reads the text operators, not the pixels.

grep is not a test

The obvious check is to search the file for the number:

grep -c "4471 2093 5568" form.pdf    # 0, for all seven files, including the original

It returns nothing even for the unredacted original, for two reasons. The content streams are Flate-compressed, so the bytes on disk are not text. And even after decompressing, the number is stored as the hex string <3434373120...>, not as 4471 2093 5568. A stream scan has to inflate each stream and look for both spellings, which is what the checker below does. For CID fonts (most Japanese, Chinese and Korean PDFs) even that fails, because the string holds glyph IDs that only map back to characters through the font's ToUnicode table. A real text extractor is the only check that works everywhere; the stream scan is a second line for leftovers that extractors don't read.

Seven versions, three extractors

Each version, checked for the account number. "Streams" counts the streams anywhere in the file's bytes, live or orphaned, that still hold it after inflating:

The first four are the same file as far as any reader is concerned. The three extractors agreed on every row, so for this question it doesn't matter which one you use, as long as you use one.

The fourth row is the one that catches people who know about redaction tools. PDF redaction is two steps: a Redact annotation that marks an area, and an apply step that removes what is under it. A file saved between the two shows red outlines (or black boxes, depending on the viewer) and still contains everything.

The incremental save trap

The sixth row passes all three extractors and still leaks. saveIncr(), like the plain Save in many desktop PDF tools, does not rewrite the file. It appends the changed objects and a new cross-reference table after the old %%EOF, and readers follow the newest table. The old content streams are still there, just no longer referenced.

Getting them back takes one line, because the first revision of the file ends at the first %%EOF:

raw = open("pymupdf-incremental.pdf", "rb").read()
first = raw[: raw.index(b"%%EOF") + 5]
print("4471 2093 5568" in pymupdf.open(stream=first, filetype="pdf")[0].get_text())   # True

The incremental file is 11,533 bytes; its first 7,163 bytes are the original form, intact. Counting %%EOF markers is a cheap check: the original and the full saves have one, the incremental save has two.

What a real removal writes instead

The file saved by the osFoundry PDF Editor has no account number in any stream, and the remaining text didn't move. This is the same line as before, after Apply:

q
BT
1 0 0 1 236 326 Tm
/tiro 12.5 Tf
[-6500] TJ
ET
Q

The string is gone and only a number is left in the TJ array. In a TJ array a number is a horizontal adjustment in thousandths of the font size, subtracted from the position, so -6500 moves the pen 6.5 × 12.5 = 81.25 points to the right without drawing anything. That is exactly the width of 4471 2093 5568 in Times-Roman:

Keeping the advance means anything after the removed run on the same line stays where it was. Checked word by word, all 63 surviving words have the same position as in the original (0 moved), and the page went from 75 words to 63. The black boxes are a separate stream at the end, plain filled rectangles (236 323.5 81.25 12.5 re f) sized to the removed text, with nothing underneath.

Metadata is its own step

PyMuPDF's full save removed the text and left author = Jordan Ellis in the document info. Redaction works on page content, and the info dictionary isn't page content. Clearing it is two calls:

doc.set_metadata({})
doc.del_xml_metadata()
doc.save(out, garbage=4, deflate=True)

On this form that leaves every info field empty and no Jordan anywhere in the file's bytes. The editor's Sanitize document step does the same plus attachments, comments, form fields, hidden layers, bookmarks and scripts, and its saved file had no metadata at all.

A checker you can keep

Everything above, as one function that returns a list of problems and exits non-zero if there are any:

import re, sys, zlib
import pymupdf
from pypdf import PdfReader
from pdfminer.high_level import extract_text


def problems(path, secrets):
    raw = open(path, 'rb').read()
    doc = pymupdf.open(path)
    flat = lambda t: re.sub(r'\s+', ' ', t.replace('\xa0', ' '))
    texts = {
        'pymupdf': flat(''.join(p.get_text() for p in doc)),
        'pypdf': flat(''.join(p.extract_text() for p in PdfReader(path).pages)),
        'pdfminer': flat(extract_text(path)),
    }
    out = []
    for s in secrets:
        out += [f'{name}: text contains {s!r}' for name, t in texts.items() if s in t]
        for p in doc:
            for a in p.annots():
                if s in (a.info.get('content') or ''):
                    out.append(f'annotation on page {p.number + 1} contains {s!r}')
        out += [f'metadata {k} = {v!r}' for k, v in doc.metadata.items() if v and s in v]
    if raw.count(b'%%EOF') > 1:
        out.append(f'{raw.count(b"%%EOF")} revisions: earlier versions are still in the file')
    if any(p.first_annot and p.first_annot.type[1] == 'Redact' for p in doc):
        out.append('redaction annotations were never applied')
    for m in re.finditer(rb'stream\r?\n(.*?)\r?\nendstream', raw, re.S):
        try:
            body = zlib.decompress(m.group(1))
        except zlib.error:
            body = m.group(1)
        for s in secrets:
            if s.encode() in body or s.encode().hex().encode() in body.lower():
                out.append(f'stream at byte {m.start()} still holds {s!r}')
    return out


if __name__ == '__main__':
    found = problems(sys.argv[1], sys.argv[2:])
    print('\n'.join(found) or 'clean')
    sys.exit(1 if found else 0)

uvx --with pymupdf --with pypdf --with pdfminer.six python assert_redacted.py out.pdf \
    "4471 2093 5568" "Jordan Ellis" "(503) 555-0142"

On the incremental file it prints the author, 2 revisions: earlier versions are still in the file and five streams that still hold a secret. On the drawn-box file it prints 9 extractor hits, the author and five streams. On the file from the osFoundry PDF Editor it prints clean.

It has limits worth stating. It only finds the strings you give it, so a value written differently (4471-2093-5568) needs its own entry. It reads text, so a secret that is part of an image needs OCR first. And the stream scan misses CID-font text, which the extractors still catch.

The PDF Editor that produced the clean file runs locally inside the desktop app, in line with osFoundry's private, local-first design, so the form doesn't leave the machine to be redacted. The app is free from the osFoundry download page.