The Black Rectangle That Doesn't Redact

Documents keep going public with their secrets still inside. The pattern is always the same: someone draws black boxes over the sensitive lines, exports the file, and a reader selects the text under the boxes and pastes it into an email.

The mistake keeps happening because the result looks correct. A black rectangle is black. Nothing on the screen suggests that the words underneath are still there. Here is why they are, how to check your own files in a minute, and what deleting text from a PDF actually takes.

What a PDF page really is

A PDF page is not a picture with text on it. It is a small program, a content stream, that a viewer runs to paint the page. Text is drawn by one set of operators and shapes by another. Here is a page I generated with pdf-lib: one line of text, then a black rectangle over part of it.

page.drawText('Salary: CHF 142000', { x: 40, y: 100, size: 16, font })
page.drawRectangle({ x: 100, y: 92, width: 160, height: 26, color: rgb(0, 0, 0) })

The decompressed content stream comes out as this, trimmed:

BT
0 0 0 rg
/Helvetica-7098480789 16 Tf
1 0 0 1 40 100 Tm
<53616C6172793A2043484620313432303030> Tj
ET
q
0 0 0 rg
1 0 0 1 100 92 cm
0 0 m ... f
Q

The first block sets a font, positions the cursor and shows the string. The hex in the Tj line is the text; it just isn't stored as readable characters. The second block fills a path with black. The viewer paints them in order, so the rectangle lands on top of the text and the eye sees a bar.

The text was never removed. A viewer that extracts text doesn't care about paint order, only about the Tj operators, so it reads the string straight through the bar. I ran pdf.js on that file:

const doc = await pdfjs.getDocument({ data: bytes }).promise
const page = await doc.getPage(1)
const content = await page.getTextContent()
console.log(content.items.map((i) => i.str))
// [ 'Salary: CHF 142000' ]

A rectangle tool or a "black highlight" in a PDF editor typically produces exactly this: one more paint operation, with the content underneath untouched. Some editors have a real redaction command that removes the text. If you're not sure which one you used, test the result.

Check your own file in one minute

You don't need special software:

1. Open the exported PDF, select all (Ctrl/Cmd+A) and paste into a text editor. Any word from under a box is a leak.
2. Or search for a word you covered with the viewer's find function. A hit on a blacked-out page is a leak.
3. Or run the script above, or a text extractor such as poppler's pdftotext, and read the output.

Do this on the file you are about to send, not on the one you edited. Also look beyond the page text. A PDF can carry the same words in metadata, bookmarks, form fields, annotations and attachments, and a file saved incrementally can keep earlier versions of a page in its own bytes. Covering the visible text fixes none of that.

Finding the boxes automatically

We build Vellum, a set of free browser-based PDF tools, and one of them is a checker for exactly this. The way it works shows how the mistake looks from the inside. It never has to guess where text is hidden. It walks the page's operator list from pdf.js and looks for filled paths.

It keeps a running transformation matrix through every save, restore and transform operator, plus the last fill colour set. When it meets a fill operator with a dark colour (relative luminance under 0.45), it records the path's bounding box in page coordinates. It also reads annotations of the Square, Redact and Highlight kinds that have a dark interior colour.

Then come the false-positive filters, which matter more than the detection:

• A box must be at least 6 points wide and 3 points high, and cover no more than half the page. Thin rules and full-page backgrounds are not redactions.
• The page is rendered at 72 dpi, and the box only counts if at most 2% of its pixels are light. A dark title bar with white text on it is a fill too, but the text is visible, so nothing is hidden and nothing should be reported.
• Only then are the text items compared with the box, and a fragment counts as covered when at least 55% of its area falls inside.

The last filter shows where the design comes from. A checker that reports "leak" on every dark rectangle is worse than none, because people stop believing it. So it renders the page and looks at the pixels before saying anything.

It analyses at most 40 pages per document, since the render check is the expensive step, and it reports how many pages it examined. The whole thing runs in the browser tab. The black-box page runs it if you want to try a document without installing anything.

What removing the text takes

Two approaches work.

Edit the content stream. Find the Tj operators under the region and rewrite or delete them, leaving the layout intact. It is the cleanest result: the document stays searchable and the file stays small. It is also the hardest to get right. Text may be split into fragments, use custom encodings, or come from a reused form object, and the covered characters may sit in the middle of a fragment.

Flatten the page. Render each page to an image, paint the black boxes onto the pixels, and build a new PDF from the images. The words under the box stop existing, because the only thing left is a picture of the page with a black area on it.

Our redaction tool takes the second route, and the code is short. In outline (my paraphrase, not the file verbatim):

for (let i = 0; i < src.numPages; i++) {
  // pdf.js renders the page to a canvas at 300 dpi
  const canvas = await renderPage(src, i, { dpi: 300 })
  // black boxes are painted straight onto the pixels
  ctx.fillStyle = '#000'
  for (const b of boxesOnPage(i)) ctx.fillRect(b.x * w, b.y * h, b.w * w, b.h * h)
  // the output document is built from scratch: one image per page
  const page = out.addPage([pageWidthPt, pageHeightPt])
  page.drawImage(await out.embedPng(canvasToPng(canvas)), { x: 0, y: 0, width: pageWidthPt, height: pageHeightPt })
}

Three details in the real implementation are worth copying:

• The output starts from an empty document. Nothing is copied from the source PDF, so its objects cannot come along by accident.
• PNG, not JPEG. JPEG puts a halo of compression noise around every black letter on white, which is visible on text. The code falls back to JPEG at quality 0.95 only when a page's PNG exceeds 4 MB, which in practice means photo-heavy pages. The longest side is also capped at 6,000 pixels so the browser can allocate the canvas.
• Padding around detected text. When boxes come from pattern matching (email addresses, phone numbers, IBANs), the box grows by 35% of the font size horizontally and 15% vertically. A box that misses the edge of a glyph by a pixel leaves a readable sliver of a letter, and a sliver of an "@" is still an "@". Slightly too much black costs nothing.

The price of flattening is that the result is only an image. You lose text selection and search, and screen readers get nothing to read. The file is often larger too. Before you apply, the tool warns that the output becomes an image with no selectable text. Keep the original for the version you still want to edit, and send the flattened copy.

Pattern search has limits too

Flattening solves the destruction. Choosing what to cover is a separate problem, and it needs the same honesty. The pattern detectors run over text lines rebuilt from pdf.js text fragments, so:

• an IBAN is accepted only if the mod-97 checksum passes, and a Swiss social security number only if its EAN-13 check digit does, which keeps random digit runs from becoming false positives;
• a pattern split across two lines is not found, because matching works line by line;
• a page that is a scan with no text layer has nothing to match, so the tool warns that some pages look like scans and points you to OCR. A proposal that silently skipped them would look like a clean result.

Whatever proposals a detector makes, someone has to look at the page before applying it. In the redaction tool, matches are proposed on the preview and nothing is applied until you press the button.

The habit worth keeping

Never trust a redaction because it looks like one. Extract the text from the file you are about to send and search it for the words you meant to hide.