---
title: "363 of our published translations had a letter from the wrong alphabet hiding in a word" published: true tags: i18n, localization, python, unicode canonical_url: https://lightsoutfactory.ai/tools/mixed-alphabets/
title: "363 of our published translations had a letter from the wrong alphabet hiding in a word"
published: true
tags: i18n, localization, python, unicode
canonical_url: https://lightsoutfactory.ai/tools/mixed-alphabets/
cover_image: https://lightsoutfactory.ai/static/img/og-tool-mixed-alphabets.jpg
Look at this word: PΡΡΡΠΊΠΈΠΉ. It reads as Russian for "Russian", and it is not quite: the first letter is the Latin P, not the Cyrillic Π . The two look the same in every font and are different characters, so search engines, the search box of the site, Ctrl+F and spell checkers all treat it as a different word.
We went looking for words like this in our own sites, and found more than we expected.
What we found
Our sites are translated into 42 languages by a local model, and every line passes mechanical checks before it is published: placeholders, numbers, tags, the right script for the language. Those checks let Latin letters through anywhere, because brand names and code are written in Latin letters in every language.
So we ran one more check over everything published: a word whose letters come from two of the Latin, Cyrillic and Greek alphabets. Out of 613,570 published page translations across our 11 sites, 363 had such a word. Every one we looked at was a real slip:
| Language | Word | What happened |
|---|---|---|
| Russian | TΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ | Latin T |
| Bulgarian, Serbian | Πdds | Cyrillic Π in an English word |
| Ukrainian | WΠΈΠ³ΡΠ°Π»ΠΈ | Latin W |
| Greek | FΟΟΞΏΟ | Latin F |
| Bulgarian | ΠΌΠ΅Π΄ΠΈΠΉΠ½ΠΈplacements | an English word glued on |
| Macedonian | ΡΓ¨ | Latin Γ¨; Macedonian has its own Ρ |
By language: Macedonian 242 (190 of them the word for "everything" with a Latin Γ¨), Bulgarian 46, Serbian 44, Greek 13, Russian 9, Ukrainian 9. No reader would report them: on the screen they look right.
Finding them takes a dozen lines
import re
import unicodedata
ALPHABETS = ("LATIN", "CYRILLIC", "GREEK")
def alphabet(ch):
name = unicodedata.name(ch, "")
return next((a for a in ALPHABETS if name.startswith(a + " ")), None)
def mixed_words(text):
words = re.findall(r"[^\W\d_]+", unicodedata.normalize("NFC", text))
return [w for w in words if len({alphabet(c) for c in w} - {None}) > 1]
A word here is a run of letters, so USB-ΠΊΠ°Π±Π΅Π»Ρ is two words and neither is mixed. Two exceptions keep it free of false alarms on our data: a word the English source already has (a brand written that way on purpose), and units such as kΞ© and ΞΌm, where a Greek letter belongs next to Latin ones.
Fixing them without breaking pages
About three in four can be fixed without the model: when most letters of the word are in the language's own alphabet and each odd letter has a twin there, we spell the word in that alphabet (TΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ becomes Π’ΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ, ΡΓ¨ becomes ΡΡ). The rest, mostly English words glued to a translated one, go back to the model with the reason: "the word ... mixes letters of two alphabets".
Serbian needed its own rule. It is written in both Latin and Cyrillic letters, and its two alphabets map one to one, so a mixed Serbian word is transliterated into the alphabet most of its letters are in. We learned that the hard way: on a line that quoted Russian words, the model wrote proΔitΠ°Π½ three times in a row, the line never passed, and the page dropped out of the Serbian build until the transliteration rule went in. Since then, a line refused only for a mixed word keeps its old translation if the new one fails too, so the rule cannot take a published page down.
Try it on your own strings
- In the browser: lookalike letter checker. Paste text or a
.po, JSON or XLIFF file; it marks every mixed word, gives it back in one alphabet, and lists invisible characters such as zero-width spaces. Nothing leaves the page. - In CI: translation-checks 1.5.0, free under the MIT licence, now fails a translation with such a word:
from translation_checks import check
check("Wagering requirements", "TΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ ΠΏΠΎ ΠΎΠ±ΠΎΡΠΎΡΡ", "ru")
# ['letters of two alphabets in one word: TΡΠ΅Π±ΠΎΠ²Π°Π½ΠΈΡ']
- repo: https://github.com/lightsoutfactory/translation-checks
rev: v1.5.0
hooks:
- id: translation-checks
If you ship translations in Cyrillic or Greek, it is worth one run: these slips are invisible until someone searches for the word.
Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.