How Face Recognition Works

Developer note: Series 3, Episode 3 deep dive for Paragon AI MAG. How does face recognition on your phone work? The mechanism: structured infrared light — a flood illuminator, a dot projector casting 30,000+ points, an infrared camera reading their displacement — builds a depth map that a neural engine turns into an embedding, compared inside the Secure Enclave against the enrolled template, with a 1-in-1,000,000 false-match threshold. We also cover the history (Bledsoe 1964, Eigenfaces 1991, DeepFace 97.35% in 2014, FaceNet's 128-number embedding at 99.63%), the 1:1 verification vs 1:N identification split, the NISTIR 8280 demographic gaps, the Kneron spoof tests, and the practical side: five settings to change tonight and three self-tests. Canonical URL: https://paragonaimag.substack.com/p/how-face-recognition-works-series

Watch the video: https://youtu.be/PhatIybN_A4

Every time you unlock your phone, it fires more than 30,000 invisible dots at your face. It does not take a photo of you. It builds a 3D map from those dots and turns it into numbers. That is the secret: literal maths, not a picture.

And yet the same core technology once locked an innocent man in jail for 30 hours. This article explains both sides — how the phone version works so well, and why the same idea fails so badly when it is pointed at a crowd.

THE MAN THE COMPUTER PICKED

On 9 January 2020, police arrived at Robert Williams' home in Detroit and arrested him in front of his wife and his two little daughters, aged two and five. They held him for about 30 hours in a dirty, overcrowded cell. A police computer had matched his driver's licence photo to a blurry surveillance still of a man who had stolen watches from a Shinola shop in 2018. The computer was wrong. Williams was nowhere near the shop, and he barely looked like the suspect. It was the first publicly reported wrongful arrest caused by a face-recognition match in the United States. In June 2024, the city of Detroit settled his case for $300,000 and banned arrests based on a face match alone. So here is the question that carries this whole story: if this technology fails that badly out in the real world, why do we trust it to protect our bank accounts, our private emails, and our most sensitive data? The answer is that the phone and the police computer are doing two different jobs.

THE SHORT ANSWER

Your phone does not take a photo of you. It projects more than 30,000 invisible infrared dots across your face to build a three-dimensional depth map, then turns that map into a list of numbers — an embedding. The phone compares numbers against numbers, never pictures against pictures. Those numbers live in the Secure Enclave, a separate small computer inside the phone's chip with its own key. The operating system cannot read them, apps cannot read them, and they are never sent to Apple or backed up to iCloud. When you look at the screen, the phone asks the vault one question — does this new maths match the saved maths? — and the vault answers yes or no.

THE SEVEN STEPS

When an Apple device wakes up and checks your face, it runs seven steps: 1. Flood illuminator. An infrared light bathes your face so the phone can find it — even in total darkness. 2. Dot projector. One laser shines through a special plate and splits into more than 30,000 invisible dots across your face. 3. Infrared camera. The only sensor that can see those dots reads how they bend around the shape of your nose, cheekbones and eye sockets. 4. Depth map. The projector sits at one end of the notch and the camera at the other, so each dot's sideways shift gives its distance. Thirty thousand distances make a 3D face. 5. Neural Engine. A dedicated chip turns the depth map and the infrared photo into a list of numbers. 6. Secure Enclave. The fresh numbers are compared with the enrolled ones inside the vault chip. Only yes or no ever leaves. 7. Threshold. A yes needs the numbers close enough that a random stranger has less than a 1 in 1,000,000 chance of matching. The old fingerprint sensor, Touch ID, was rated at 1 in 50,000 — Face ID is twenty times harder to fool with a random face. There is also an attention check: by default your eyes must be open and looking at the phone, so it cannot unlock while you sleep. And Face ID learns — a beard, glasses, a hat, ageing — it updates the stored numbers over time.

ONE-TO-ONE VERSUS ONE-TO-MANY

This is the difference that explains everything. Your phone does verification (1:1): it asks, "are you this one specific person?" and checks one live face against one enrolled face. It is a closed loop, and it knows exactly who it is looking for. The police computer does identification (1:N): it asks, "who is this?" and scans one face — often a blurry one — against a database of millions. It is a dragnet, guessing at a needle in a haystack. The maths is similar. The error is not.

SIXTY YEARS IN SIXTY SECONDS

Face recognition did not appear overnight. In 1964–1966, mathematician Woodrow Bledsoe and his colleagues Helen Chan Wolf and Charles Bisson built the first system: a human marked the eyes, nose and ears by hand on a RAND tablet, and the computer measured the distances between the marks. In 1991, MIT researchers Matthew Turk and Alex Pentland introduced Eigenfaces, the first time faces were turned into maths for real-time recognition. In 2014, Facebook's DeepFace reached 97.35% accuracy on the standard LFW test — almost matching humans at 97.53%. A year later, Google's FaceNet mapped each face to 128 numbers and hit 99.63% on the same test. That 128-number embedding is the idea every system still uses. In 2017, the iPhone X launched Face ID, and suddenly 3D infrared scanning was mainstream. Today the best systems claim accuracy above 99.8%. But that number is a dangerous illusion, because the accuracy is not distributed equally.

THE NIST STUDY: THE 100-TIMES GAP

In December 2019, the US National Institute of Standards and Technology published the biggest accuracy test ever run: NISTIR 8280. It tested 189 algorithms from 99 developers on 18.27 million images of 8.49 million people — mugshots, visa photos, border crossings. The false-positive rates — when the system wrongly matched two different people — were severely skewed. Algorithms made false matches on West African, East African and East Asian faces at rates up to 100 times higher than on Eastern European faces. Error rates also spiked for women, children and older adults. In one-to-many police searches, African American women faced the highest false-positive rates. The bias was not in the concept. Algorithms developed in Asia did not show the same gap on East Asian faces — the bias follows the training data. And Detroit's police chief admitted in 2020 that the department's system misidentified suspects in 96% of cases. A false negative on your phone just means it asks for your passcode. A false positive in a police search means a stranger gets arrested. That is why Robert Williams spent 30 hours in a cell.

THE MASK TESTS

In December 2019, the AI company Kneron tested how easy it is to fool face recognition. Researchers commissioned high-quality 3D masks and, with permission, tried them on systems across three continents. They paid with someone else's account on AliPay and WeChat payment tablets. They tricked a self-boarding gate at Amsterdam's Schiphol Airport with just a photo on a phone screen. They passed face-pay rail gates in China. Then they tried the same masks against Apple's 3D technology — and the system stayed perfectly secure. Flat images and printed masks simply do not have the real 3D depth to trick the dot projector. Not all face recognition is the same technology: a 2D camera photo-match and a 3D infrared depth map are different products with the same name. Apple also had to solve the pandemic problem. On 14 March 2022, iOS 15.4 added Face ID with a mask: it reads only the area around your eyes. It works on iPhone 12 and newer, and it does not work with sunglasses.

YOUR PHOTO LIBRARY IS DOING IT TOO

Even Apple is not immune. The Photos app on your phone turns every face in your library into a list of numbers and groups them into People albums — including faces of people who never agreed to be scanned. In August 2026, a federal judge in Illinois certified a class action, Doe v. Apple, over that face sorting: up to 6.5 million Illinois residents, up to $5,000 each under the state's 2008 Biometric Information Privacy Act, a theoretical maximum near $32.5 billion. Apple says the numbers are end-to-end encrypted and cannot be read back into a face. A jury trial is set for April 2027. Nothing has been decided.

THE COMPANY WITH 50 BILLION FACES

Outside your phone, the landscape is wilder. Clearview AI scraped more than 50 billion images from public websites to build a face-search database for police and government agencies — without anyone's permission. It signed a $9.2 million contract with US Immigration and Customs Enforcement in 2025. The lack of consent led to heavy fines in the EU and the UK. Regulators have pushed back hard. Under the EU AI Act, Article 5(1)(h), real-time face recognition in public spaces by police is banned across the EU since 2 February 2025, with only three narrow exceptions, each needing a judge's authorisation. Fines reach €35 million or 7% of worldwide turnover. The same article also bans untargeted scraping of face images from the internet — the exact method Clearview used. The United States still has no single federal rule.

FIVE THINGS TO DO TONIGHT

1. Use a long alphanumeric passcode. Your face scan falls back to this code, so make it strong. A four-digit PIN has ten thousand combinations; a six-digit code has a million. 2. Turn on Require Attention in Settings > Face ID & Passcode. It forces the phone to check your eyes are actually open, so nobody can unlock it while you sleep or look away. 3. Audit your app privacy settings to see exactly which apps have access to your camera — and take it away from any that do not need it. 4. If you run iOS 15.4 or later, enable Face ID with a mask, which securely maps just the area around your eyes. 5. Subscribe to Paragon AI MAG — one video and one podcast episode every day, every fact checked against sources.

THREE SELF-TESTS

The photo test. Hold a printed photo of yourself, or a picture on another screen, up to your locked phone. If it is a secure 3D system, it will fail every time — a flat image has no depth for the dots to measure. The dark test. Take your phone into a pitch-black room and try to unlock it. It should work perfectly, because the TrueDepth camera brings its own invisible infrared light. The Android hardware check. If you are on Android, look up your phone model to see whether it relies on secure 3D dot projection or just a foolable 2D camera for face unlock. That is a massive difference in security. A quick check: try unlocking in total darkness — if it fails, or the screen has to light up blindingly bright just to see your face, your phone likely uses a much less secure 2D camera.

THE VERDICT

CLEARED for the phone. One in a million, on-device, photo-proof, attention-checked, and it learns your face over time. WATCHING for the technology at large. NIST's 10-to-100-times gaps, a 50-billion-image database built without consent, a man held 30 hours on a wrong match, and a $32.5 billion lawsuit that is still undecided. As the researchers put it: accuracy, not the concept, is the variable. When your device uses 30,000 invisible dots to map your specific face and simply asks, "are you this one person?", it is an incredible, highly secure tool. When systems use unbalanced algorithms and flat, flawed 2D images to search a crowd of millions, accuracy plummets — and innocent people get falsely accused. So here is the closing question: will you ever trust a simple 2D camera with your 3D identity again?

Paragon AI MAG — one video and one podcast episode every day. Subscribe to the Paragon AI MAG YouTube channel and follow the show on RedCircle so you never miss an episode.