A Japanese toxicity detector whose headline metric is not flagging legitimate criticism (Macro F1 0.975)
TL;DR — We released ja-toxicity-guard-v1, a multi-label classifier that flags seven kinds of harmful Japanese text — insult, indirect attack, exposure intent, threat, discrimination, sexual harassment, self-harm signals.
TL;DR — We released ja-toxicity-guard-v1, a multi-label classifier that flags seven kinds of harmful Japanese text — insult, indirect attack, exposure intent, threat, discrimination, sexual harassment, self-harm signals. On a sealed test of 5,000 items it scores Macro F1 0.9747, and it flagged zero of 600 legitimate criticism texts. It also false-flags about 13–14% of argumentative and ordinary harmless text, which is why its stated job is sorting the human review queue — never automated removal. Apache 2.0, ONNX included, ~38–40 ms per item on CPU.
- The design target was the false positive, not the catch rate. A moderation model that flags legitimate criticism, reporting, and quotation as abuse does more damage than the abuse it catches. So the headline measurement is split by harmless category: legitimate criticism 0.0000 (0 of 600), reporting and quotation 0.0083 (5 of 600) — and then, honestly, argument/metaphor/impression text 0.1317 and general harmless text 0.1450. The first two numbers are the design working. The last two are why a human stays in the loop.
- Seven labels, seven thresholds. Multi-label classification with per-label thresholds fixed on development data and shipped in the config (0.5 for insult, exposure intent, discrimination, self-harm; 0.85 for indirect attack, threat, sexual harassment — the labels where a false flag costs most). Sealed-test per-label F1 runs from 1.0000 (sexual harassment) and 0.9972 (discrimination) down to 0.9384 for indirect attack, the weakest label: sarcasm and deniable aggression are genuinely hard, and the score says so.
- The self-harm label routes, it does not respond. It exists to move a post to a human quickly. What happens next is the operator's responsibility and policy — the model detects a signal and makes no decision about the person.
What it is for, and what it is not for
For: ordering a moderation queue so the worst text gets human eyes first; adding a warning signal to a posting flow; triage at volumes where reading everything in arrival order means reading the worst things last.
Not for: deleting posts, suspending accounts, or any automated action against a user. The false-positive profile below is exactly why. A tool that misreads one polite rebuttal in eight as an indirect attack is a useful sorter and an unacceptable judge.
Use it
from transformers import AutoModelForSequenceClassification, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("japan-opendata/ja-toxicity-guard-v1")
model = AutoModelForSequenceClassification.from_pretrained("japan-opendata/ja-toxicity-guard-v1")
The reference implementation in examples/infer_toxicity_guard.py applies the per-label thresholds from inference_config.json and returns labels with probabilities. The ONNX build (fp32) produced identical labels to PyTorch at the adopted thresholds on 200 development and 200 sealed items. Speed: ~38–40 ms per item on a Core i7-11700F.
Measured results (sealed test, 5,000 items, single final run)
- Macro F1 0.9747, Micro F1 0.9717, exact-match rate 0.9566
- Per-label F1: sexual harassment 1.0000 / discrimination 0.9972 / self-harm 0.9845 / threat 0.9709 / insult 0.9669 / exposure intent 0.9652 / indirect attack 0.9384
False-positive rates on harmless text, by category:
- Legitimate criticism: 0.0000 (0 / 600)
- Reporting and quotation: 0.0083 (5 / 600)
- Argument, metaphor, impressions: 0.1317 (79 / 600)
- General harmless text: 0.1450 (29 / 200)
- All harmless text: 0.0511
Training data
Entirely synthetic — no real posts. Attack texts target only fictional personas, generic positions, and fictional organizations; no real individuals, groups, or regions are targeted anywhere in the data. The full dataset was audited for real proper nouns, contact details, and concrete self-harm methods: zero violations found. Splits: 100,000 train / 5,000 development / 5,000 sealed, checked for cross-split overlap.
What didn't work / Limitations
- Argumentative and polite text false-flags at over 10%. Polite rebuttal structures and strong-negation idioms get read as indirect attack or insult. This is the model's known failure mode, it is printed on the card, and it is the reason the intended use is review prioritization only.
- Indirect attack is the weakest label (F1 0.9384). Sarcasm, exposure intent by implication, and deniable phrasing are where synthetic training data least resembles the real thing.
- Synthetic data, as with the whole series. Real posts will score lower than the sealed numbers — different slang, different targets, different cruelty. We publish the synthetic measurement and the caveat together.
- Detection is not a decision. This model must not be the mechanism that removes content or acts on an account. It sorts and signals; people decide.
This is the eighth in our line of small, single-purpose Japanese models. The pattern across the series is deliberate: narrow job, sealed test, and the embarrassing slice of the results printed in the same table as the good ones.
If you moderate a Japanese-language community: what does your review queue look like today — chronological, report-count order, or something smarter?
- Model on Hugging Face: https://huggingface.co/japan-opendata/ja-toxicity-guard-v1
- Code (the model series): https://github.com/japan-opendata-lab/ja-small-models
- Model page with usage notes: https://jpopendata.jp/models/ja-toxicity-guard-v1/
Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.