We Tested Five Ways to Black Out a Number in a PDF, and Our Own Detector Missed Three

Someone needs to send a document with their social security number hidden. They open it, cover the number with something black, export, send. The question we wanted answered was narrow: after each of the common ways of doing that, can the recipient still copy the number out of the file?

We also wanted to know whether our own leak detector would notice. It didn't, at first. This post covers both: what the five methods do to the file, and the three bugs the test exposed in code we thought was finished.

The test document

One fictitious certificate, written in Word and exported to PDF: an invented name, an invented address, and a Swiss social security number (AHV/AVS) of the form 756.0000.0000.02. No real data anywhere. Each method was applied to a fresh copy of the same file.

Two judges read every result:

• an independent text extraction with pypdf, which has no opinion about what is visible and simply dumps the text layer;
• our browser-side detector, which tries to find text sitting under opaque shapes and report it to the user.

The five methods

Four of five leave the number in the file. The highlighter, which is probably the most common gesture of the lot, removes nothing and warns about nothing. In A, B, C and E, pypdf returned exactly the covered digits.

The one method that worked is the one built for the job. Preview also deserves credit for case C: it tells you that a filled annotation hides nothing. Word and Acrobat stayed silent.

What our detector said

Before the fixes described below, our detector flagged one of the four leaks. Three files with a copyable social security number under a black box got a clean report. A tool that exists to catch exactly this was telling people their document was safe.

The detector works in three steps, and each step had a bug.

1. Collect candidate zones. Walk the page's operator list (via pdf.js), track the transformation matrix and fill colour, and keep every dark filled path. Separately, keep every Square, Redact and Highlight annotation.
2. Confirm by rendering. Render the page to a canvas and keep a zone only if its pixels are uniformly dark. This guard is deliberate: a dark title banner with white text on it is a dark filled path too, and calling it a hiding place would be a false alarm.
3. Match text to zones. For each confirmed zone, collect the text items from getTextContent() that fall inside it.

Bug 1: half-lit pixels on the edge

The render check counted every pixel in the zone's bounding box and tolerated 2% light pixels. A zone edge rarely lands on a pixel boundary, so the outermost row and column get anti-aliased into grey. On a zone a few points tall (a highlighted line of text, say), that grey fringe alone pushed the light share to 7%. Word's highlighter failed the check and was discarded as "not really opaque".

The fix samples only pixels that are entirely inside the zone, rounding the left and top edges up and the right and bottom edges down, after stepping in by a small margin:

const retrait = Math.min(2, (z.x1 - z.x0) * 0.15, (z.y1 - z.y0) * 0.15)
const gauche = (z.x0 + retrait) * echelle
const droite = (z.x1 - retrait) * echelle
const haut = (hauteurPage - z.y1 + retrait) * echelle
const bas = (hauteurPage - z.y0 - retrait) * echelle
let px = Math.max(0, Math.ceil(gauche))
let py = Math.max(0, Math.ceil(haut))
let pw = Math.floor(droite) - px
let ph = Math.floor(bas) - py

If that leaves less than one whole pixel, the check falls back to the zone's centre pixel. The canvas origin is at the top while PDF's is at the bottom, hence hauteurPage - y.

Bug 2: colours read on the wrong scale

For annotations, the old code filtered by colour before the render check: it read the annotation's interior colour, fell back to its border colour, and assumed components between 0 and 1. Two things were wrong. The annotation data we receive from pdf.js gives color on a 0–255 scale, and the interior fill colour wasn't available to us at all, so the code was really judging the border.

Acrobat's rectangle comment has a red border and a black fill. Judged by its red border, it came out light and was dropped before anyone looked at the pixels.

The fix was to delete the colour filter for annotations. Every rectangle, redaction or highlight annotation becomes a candidate, and the render check alone decides. A light or empty annotation fails that check anyway, so the filter was buying nothing except a way to be wrong.

One more detail surfaced with Preview's filled rectangle: the annotation's frame leaves a thin transparent margin, about a point wide, around the fill. That is what the 2-point step-in in the snippet above absorbs. With both changes, C and E were caught.

Bug 3: one text item, one whole line

pdf.js frequently returns a whole line as a single text item: label, colon and number together. The old code computed what share of that item's bounding box fell under the zone and required 55%. Word's rectangle covered the number but not the label, roughly half the line, so the share stayed under the threshold and the text was never matched.

The fix judges character by character. The item's width is split evenly across its characters; a character counts as covered if at least three quarters of its slot falls inside the zone, and the line must still be covered over more than half its height:

const pas = (fx1 - fx0) / n
for (let k = 0; k < n; k++) {
  const c0 = fx0 + k * pas
  const c1 = c0 + pas
  const dedans = Math.min(c1, z.x1) - Math.max(c0, z.x0)
  if (dedans >= pas * 0.75) {
    if (debut < 0) debut = k
    fin = k
  }
}

Splitting the width evenly is an approximation, since proportional fonts don't give every glyph the same advance; hence 75% rather than 100%. The function returns only the covered slice of the string, so the report shows the hidden digits rather than the whole line, and it ignores a slice that contains no letter or digit: a lone colon peeking out from under the edge of a box is not a leak.

How we know it's fixed

The test script runs six fixture cases against the detector, each with the expected verdict. One of them is new: the highlighter case, which fails on the old code and passes on the new one. On the five-method test, the detector now recovers exactly the covered number in A, B, C and E, and nothing in D.

What we didn't test: the markup pen in the iPhone Photos app, for lack of a device. That pen draws on an image, so whether text survives depends on whether some app later added an OCR layer to the picture, which is a different check.

What the bugs had in common

Each one was a silent false negative. Nothing crashed, nothing logged a warning; the detector just returned an empty list. Our earlier fixtures were PDFs generated in memory with pdf-lib: black rectangles drawn into the page content with edges on whole points, and no annotations at all. That is why the suite was green. The files people actually produce come out of Word, Preview and Acrobat, each with its own rendering habits.

So the lesson we took is about test material, not about the algorithm: build fixtures with the tools your users use, and keep at least one case that the old code fails. A green suite made only of hand-drawn rectangles proved nothing about the highlighter.

If you need to hide something in a PDF

Use a tool that removes the content instead of covering it. Preview's Redact does. Acrobat Pro has a redaction tool as well; the free comment rectangle is not one.

The redact tool on VellumPDF takes the blunt route: each page that carries a redaction is rendered to an image at 300 DPI, the boxes are painted onto the pixels, and the page is replaced by that image. The original text of those pages no longer exists in the output. Pages without a box are copied as they are, so a long report keeps its searchable text, unless you tick the option to render every page, which is safer when a file may hide text elsewhere. The output is a fresh document: the original's metadata, bookmarks and attachments don't carry over.

If you receive a "redacted" file, or are about to send one, the black box check runs the detector described above on your file and shows what is still readable under the dark areas. The same check is part of the pre-send inspection, alongside metadata and earlier saved versions. Both run in the browser; the PDF isn't uploaded.