Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

Teaching a Discord moderation bot to tell a joke from an insult

Word filters are the default for Discord moderation, and they fail in both directions: friends roasting each other get punished, and a cruel message with no swear word goes through. I wanted a bot that judges what a mess

Word filters are the default for Discord moderation, and they fail in both directions: friends roasting each other get punished, and a cruel message with no swear word goes through. I wanted a bot that judges what a message means.

One short answer per message

Each enabled module gets a number in the prompt. The model answers in a few characters: 0 if nothing is wrong, otherwise category + severity (1 to 3) + tone, H for banter or S for serious. 32;H means "a clear insult, but a joke". Short answers keep it fast and cheap, and a strict regex rejects anything else.

Sensitivity is applied by the bot, not the model

The prompt is identical for every server, so it can be cached. Each server's sensitivity only changes which severities count locally, and banter counts one level less. Changing a setting never needs another API call.

Few-shot examples as real turns

Listing the examples inside the system prompt made the small model answer 0 to almost everything. Sending them as real user/assistant exchanges fixed it. Their order matters too: the last example pulls the answer towards it.

Context is for tone only

The model sees the author's previous messages and the message being replied to, but an innocent "what??" under an insult must stay innocent. That rule is written into the prompt and backed by examples.

A second opinion before any sanction

When the first model wants to act, a stronger model gets a yes/no question: does this message itself really contain it? If it says no, or doesn't answer, nothing happens. The rule: better to miss a culprit than punish an innocent.

Learning from mistakes

Staff can "pardon" an automatic sanction. The pardoned message, what the model saw and answered, and the server's settings are kept (anonymised) and reviewed before each update to fix the prompt and the examples.

The bot is AI Automod. The rule-based moderation is free; I'm happy to answer questions about the prompt design in the comments.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.