Dev.to AI 🤖 Ai 👁 0 📖 4 min read

A Japanese error detector that points at the suspicious span and stops there (F1 0.867)

TL;DR — We released ja-error-detect-v1, a small token-classification model that finds the spans in a Japanese sentence that look wrong — wrong particle, wrong homophone, bad okurigana, a look-alike kanji, a missing chara

TL;DR — We released ja-error-detect-v1, a small token-classification model that finds the spans in a Japanese sentence that look wrong — wrong particle, wrong homophone, bad okurigana, a look-alike kanji, a missing character, an extra character — and proposes no correction. On a sealed test of 5,000 items it scores span-level F1 0.8674 (precision 0.9094, recall 0.8291). When we hand-checked every span it flagged in clean text, 29 of 65 turned out to be real errors in the source itself; the true false-alarm rate is 0.2538 per 1,000 characters. Apache 2.0, ONNX included, ~20 ms per sentence on CPU.

  1. Detection only, by design. Correction is a different, harder, more opinionated problem. This model is a pre-filter: it tells a proofreading tool or an input method where to look. The final judgment — and any fix — belongs to a human or to downstream logic.
  2. The measurement: thresholds were fixed on development data before the sealed test was opened. Exact-span match F1 0.8674 across six error types: okurigana 0.9573, look-alike kanji 0.9285, particles 0.8880, missing characters 0.8596, homophones 0.8436, extra characters 0.7686 — the weakest, and named as such.
  3. The false-alarm number has a footnote we are proud of: the raw machine count said 0.4582 false detections per 1,000 characters. We then read all 65 flagged spans in supposedly clean sentences by hand, and 29 were genuine errors that existed in the source text — the "clean" data wasn't. The corrected rate is 0.2538 per 1,000 characters (1.20% of sentences). We publish the corrected number and the correction method, not the flattering silence.

What counts as an error here

Six types, all mechanical rather than stylistic:

  • Particle substitution (は/が/を/に and friends in the wrong slot)
  • Homophone mis-conversion (the classic IME failure: right sound, wrong word)
  • Okurigana errors (送り仮名 — the kana endings of inflected words)
  • Substitution with a visually similar kanji
  • Missing characters (脱字)
  • Extra characters (衍字)

Dialect, register, and colloquial phrasing are explicitly not errors. A detector that flags how people actually talk is worse than no detector.

Use it

The reference implementation (in the repo's examples/infer_error_detect.py) applies the shipped threshold — keep spans whose mean error probability reaches 0.80. The core is short:

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

repo = "japan-opendata/ja-error-detect-v1"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)
model.eval()

text = "来週の会議は月曜日に行なわれます。"
enc = tok(text, return_offsets_mapping=True, return_tensors="pt")
offsets = enc.pop("offset_mapping")[0].tolist()
with torch.no_grad():
    probs = model(**enc).logits.softmax(-1)[0]
err_prob = 1.0 - probs[:, 0]  # label 0 is "no error" (O)
spans = []
cur = None
for (s, e), p in zip(offsets, err_prob.tolist()):
    if s == e:
        continue
    if p > 0.5:
        if cur is None:
            cur = [s, e, [p]]
        else:
            cur[1] = e
            cur[2].append(p)
    elif cur is not None:
        spans.append(cur)
        cur = None
if cur is not None:
    spans.append(cur)
kept = [(s, e) for s, e, ps in spans if sum(ps) / len(ps) >= 0.80]
print([(text[s:e], s, e) for s, e in kept])

The ONNX build (fp32) is included; its predictions matched PyTorch on all 200 development and 200 sealed items checked. Speed: ~20 ms per average sentence on a Core i7-11700F.

Measured results (sealed test, 5,000 items)

  • Exact-span match: Precision 0.9094, Recall 0.8291, F1 0.8674
  • Per-type F1: okurigana 0.9573 / look-alike kanji 0.9285 / particles 0.8880 / missing characters 0.8596 / homophones 0.8436 / extra characters 0.7686
  • False alarms on clean text, after hand-checking every flagged span: 0.2538 per 1,000 characters, 1.20% of sentences (uncorrected machine count: 0.4582, 2.13%)

What didn't work / Limitations

  • Extra characters are the weak type. F1 0.7686, the lowest of the six. An inserted character often still reads plausibly, and the model hesitates where a human proofreader would not.
  • The training data is synthetic. Errors were injected programmatically into FineWeb-2 Japanese sentences. Real-world error corpora — actual typos by actual people, with their real distribution — will score lower. Treat 0.8674 as the synthetic-test value it is.
  • Homophones split the judges. Whether a homophone is wrong can depend on intent; in paraphrasable spots the model's call and a human's call diverge, which caps that type around 0.84.
  • Proper nouns attract false alarms. The hand-checked true false positives were mostly legitimate particle contexts misjudged, loanwords and katakana terms, and proper nouns. Text dense with names will over-flag; budget review time accordingly.
  • A flagged span is a candidate, not a verdict. The model can be wrong in both directions, and it never tells you what the text should say instead.

This is the sixth in our line of small, single-purpose Japanese models. The pattern holds: one narrow job, ONNX included, a sealed test, and a manual audit of the embarrassing number before publishing it.

If you build Japanese text tooling: would a point-only detector slot into your pipeline, or is a flag without a suggested fix just extra work?

Get the next release notes and our Kaggle experiment logs by email: https://jpopendata.jp/ (newsletter signup in the footer).

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.