DarijaKit: an open-source toolkit for Moroccan Darija on local LLMs
Moroccan Darija is spoken by tens of millions of people, but local LLMs handle it unevenly. People write it in Arabic script and in Arabizi (ch7al mn sa3a khasni nkhdem?), mix it with French, and spell the same word seve
Moroccan Darija is spoken by tens of millions of people, but local LLMs handle it unevenly. People write it in Arabic script and in Arabizi (ch7al mn sa3a khasni nkhdem?), mix it with French, and spell the same word several ways.
We wanted a practical toolkit for developers working with Darija on their own hardware, so we built DarijaKit. It is a single stdlib-only Python module with a CLI, and it runs against Ollama, llama.cpp, LM Studio or vLLM.
What is inside
- Normalize: unify spelling variants, diacritics and alef/ya forms before text reaches your model.
- Transliterate: convert Arabizi to Arabic script.
- Detect: tell whether a message is in Arabic script, Arabizi or something else.
- Benchmark: measure how well a local model understands Darija.
- Fine-tune: a LoRA recipe with a 4-bit base model, sized for one 12 GB GPU.
Quickstart
pip install git+https://github.com/ManadgerTech/darijakit.git
darijakit arabizi "ch7al mn sa3a khasni nkhdem?"
# ุดุญุงู ู
ู ุณุงุนุฉ ุฎุงุตูู ูุฎุฏู
?
darijakit normalize "ุจุด ูู
ุดู ููุณูู ุฏุงุจ"
# ุจุงุด ูู
ุดู ููุณูู ุฏุงุจุง
darijakit detect "salam labas 3lik?"
# arabizi
As a library:
import darijakit as dk
dk.arabizi_to_arabic("bghit nakol tajine") # "ุจุบูุช ูุงูู ุทุงุฌูู"
dk.normalize("ุดูุฑุงู ุจุฒุงู") # "ุดูุฑุง ุจุฒุงู"
dk.detect_script("fin ghadi?") # "arabizi"
The benchmark
evalset.jsonl ships with 30 hand-written Darija items across six areas: comprehension, translation to Modern Standard Arabic, translation to English, sentiment, entities, and grammar and culture.
Scoring is deterministic (option letters and normalized containment), so results are comparable across models and over time. Thirty items is a small set. It is a starting point, not a leaderboard, and you can add your own lines to the JSONL file.
# Ollama
darijakit eval --model qwen2.5:7b
# llama.cpp server
darijakit eval --model local --endpoint http://localhost:8080/v1
# JSON for CI
darijakit eval --model qwen2.5:7b --json > results/$(date +%F).json
Fine-tuning
finetune_lora.py is a ready recipe, and sample_instruct.jsonl is a small seed set that shows the data format. Grow it with your own data, then use the eval set to check whether the fine-tune actually helped.
Roadmap and contributing
We plan a larger community-driven eval set, better Darija and French code-switch handling, an Ollama Modelfile template and a Hugging Face release of the eval set.
If you speak Darija, you can help. New eval items and spelling-variant mappings from native speakers are the most useful contributions.
The code is MIT licensed: https://github.com/ManadgerTech/darijakit
Made in Morocco by Manadger Tech.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.