Dev.to AI ๐Ÿค– Ai ๐Ÿ‘ 0 ๐Ÿ“– 2 min read

DarijaKit: an open-source toolkit for Moroccan Darija on local LLMs

Moroccan Darija is spoken by tens of millions of people, but local LLMs handle it unevenly. People write it in Arabic script and in Arabizi (ch7al mn sa3a khasni nkhdem?), mix it with French, and spell the same word seve

Moroccan Darija is spoken by tens of millions of people, but local LLMs handle it unevenly. People write it in Arabic script and in Arabizi (ch7al mn sa3a khasni nkhdem?), mix it with French, and spell the same word several ways.

We wanted a practical toolkit for developers working with Darija on their own hardware, so we built DarijaKit. It is a single stdlib-only Python module with a CLI, and it runs against Ollama, llama.cpp, LM Studio or vLLM.

What is inside

  • Normalize: unify spelling variants, diacritics and alef/ya forms before text reaches your model.
  • Transliterate: convert Arabizi to Arabic script.
  • Detect: tell whether a message is in Arabic script, Arabizi or something else.
  • Benchmark: measure how well a local model understands Darija.
  • Fine-tune: a LoRA recipe with a 4-bit base model, sized for one 12 GB GPU.

Quickstart

pip install git+https://github.com/ManadgerTech/darijakit.git

darijakit arabizi "ch7al mn sa3a khasni nkhdem?"
# ุดุญุงู„ ู…ู† ุณุงุนุฉ ุฎุงุตู†ูŠ ู†ุฎุฏู…?

darijakit normalize "ุจุด ู†ู…ุดูŠ ู„ู„ุณูˆู‚ ุฏุงุจ"
# ุจุงุด ู†ู…ุดูŠ ู„ู„ุณูˆู‚ ุฏุงุจุง

darijakit detect "salam labas 3lik?"
# arabizi

As a library:

import darijakit as dk

dk.arabizi_to_arabic("bghit nakol tajine")   # "ุจุบูŠุช ู†ุงูƒู„ ุทุงุฌูŠู†"
dk.normalize("ุดูƒุฑุงู‹ ุจุฒุงู")                      # "ุดูƒุฑุง ุจุฒุงู"
dk.detect_script("fin ghadi?")                # "arabizi"

The benchmark

evalset.jsonl ships with 30 hand-written Darija items across six areas: comprehension, translation to Modern Standard Arabic, translation to English, sentiment, entities, and grammar and culture.

Scoring is deterministic (option letters and normalized containment), so results are comparable across models and over time. Thirty items is a small set. It is a starting point, not a leaderboard, and you can add your own lines to the JSONL file.

# Ollama
darijakit eval --model qwen2.5:7b

# llama.cpp server
darijakit eval --model local --endpoint http://localhost:8080/v1

# JSON for CI
darijakit eval --model qwen2.5:7b --json > results/$(date +%F).json

Fine-tuning

finetune_lora.py is a ready recipe, and sample_instruct.jsonl is a small seed set that shows the data format. Grow it with your own data, then use the eval set to check whether the fine-tune actually helped.

Roadmap and contributing

We plan a larger community-driven eval set, better Darija and French code-switch handling, an Ollama Modelfile template and a Hugging Face release of the eval set.

If you speak Darija, you can help. New eval items and spelling-variant mappings from native speakers are the most useful contributions.

The code is MIT licensed: https://github.com/ManadgerTech/darijakit

Made in Morocco by Manadger Tech.

๐Ÿ“ฐ Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.