Invisible Unicode characters are silently breaking your strings
Invisible Unicode characters are silently breaking your strings You copy text from a chat, a doc, or a webpage. It looks empty. But str.length says 3. Your === comparison fails. Your form validation rejects input the u
Invisible Unicode characters are silently breaking your strings
You copy text from a chat, a doc, or a webpage. It looks empty. But str.length says 3. Your === comparison fails. Your form validation rejects input the user swears they typed.
Nothing is wrong with your code. The string contains characters you cannot see.
The usual suspects
A small set of Unicode code points renders as completely blank in most fonts, yet counts as a real character:
- U+200B Zero Width Space
- U+200C Zero Width Non-Joiner
- U+200D Zero Width Joiner
- U+FEFF Zero Width No-Break Space (BOM)
- U+3164 Hangul Filler
- U+2800 Braille Pattern Blank
They sneak in through copy-paste from messaging apps, rich text editors, and web pages. U+3164 is also the character people use on purpose for blank WhatsApp messages and empty Instagram names. Same characters, two very different problems.
Finding them in JavaScript
function findInvisible(str) {
const targets = new Set([0x200B, 0x200C, 0x200D, 0xFEFF, 0x3164, 0x2800]);
const found = [];
for (const ch of str) {
const cp = ch.codePointAt(0);
if (targets.has(cp)) {
found.push('U+' + cp.toString(16).toUpperCase().padStart(4, '0'));
}
}
return found;
}
Finding them in Python
import unicodedata
def find_invisible(s):
targets = {0x200B, 0x200C, 0x200D, 0xFEFF, 0x3164, 0x2800}
return [
(c, f"U+{ord(c):04X}", unicodedata.name(c, "UNKNOWN"))
for c in s if ord(c) in targets
]
Stripping them
Once you know what you are dealing with, removal is one line. JavaScript:
const clean = str.replace(/[\u200B-\u200D\uFEFF\u3164\u2800]/g, '');
(That character class contains U+200B through U+200D, U+FEFF, U+3164, and U+2800.)
Python:
import re
clean = re.sub(r'[\u200B-\u200D\uFEFF\u3164\u2800]', '', s)
Be careful with U+200D: it is load-bearing inside emoji sequences (family emojis, skin-tone modifiers). Stripping it blindly from user content will break emoji. Only strip it when you know the field should be plain text.
When you do not want to run code
Not every case deserves a script. If you just need to check a suspicious string, or you want to copy a specific invisible character for a blank message or username, there are free web tools for it. I built one at invisiblechar.com: paste any text into the detector and it highlights each invisible code point by name, and the generator gives you one-click copy for U+3164, U+200B, and the rest, plus a full Unicode table of 107 blank characters.
Invisible characters are a feature of Unicode, not a bug. They break things only when you do not know they are there.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.