Splitting "jumped" into jump/ed with three regexes, and the 3 words out of 36 it got wrong

Speech-language pathologists measure a child's MLU, mean length of utterance, by counting morphemes, not words. "Doggie jumped" is two words but three morphemes: doggie, jump, -ed. On a 50-utterance sample that counting is slow, so the MLU calculator I maintain has an "auto-mark" button: it rewrites jumped as jump/ed, and every slash adds one morpheme to the count. The clinician then fixes whatever it got wrong.

This post is about how little code that takes, and where it breaks.

The whole marker

Three suffix rules, three exception lists, and two guards for capitalised words:

if (lw.length >= 5 && /ing$/.test(lw) && KEEP_ING.indexOf(lw) < 0) return w.slice(0, -3) + '/' + w.slice(-3);
if (lw.length >= 4 && /ed$/.test(lw)  && KEEP_ED.indexOf(lw)  < 0) return w.slice(0, -2) + '/' + w.slice(-2);
if (lw.length >= 4 && /s$/.test(lw)   && KEEP_S.indexOf(lw)   < 0 && !/ss$|us$|is$|ous$/.test(lw))
  return w.slice(0, -1) + '/' + w.slice(-1);

The length checks keep bed, red and is away from the rules before the lists are even consulted. The exception lists hold the words that only look inflected: KEEP_ING has thing, ceiling, morning, nothing; KEEP_ED has need, bread, hundred, instead; KEEP_S has always, pants, scissors, news, lots. Words that start with a capital in the middle of a line are treated as names and left alone, and anything that already contains an apostrophe is skipped, because contractions and possessives are counted by a separate rule (Mommy's and can't are 2 each).

Irregular forms need nothing at all. went, feet and children have no suffix to strip, so they stay at 1, which is how most counting guides want them. Same for gonna, birthday and doggie.

Testing it on 36 words

I ran the real function from the page in Node on 36 words a 3-year-old might say, and compared each count with the convention the tool itself follows (its own exception lists):

jumped     -> jump/ed      2
sleeping   -> sleep/ing    2
hiking     -> hik/ing      2
dishes     -> dishe/s      2
ladies     -> ladie/s      2
pants      -> pants        1
scissors   -> scissors     1
pajamas    -> pajama/s     2   <- should be 1
jeans      -> jean/s       2   <- should be 1
does       -> doe/s        2   <- depends on your manual

33 of 36 came out as intended. The three misses are instructive.

pajamas and jeans. The tool already counts pants and scissors as one morpheme, because there is no singular "pant" a toddler is contrasting it with. pajamas and jeans are the same kind of word, they just weren't in the list. The rule is right; the list is incomplete. That's the general failure mode of this design: every miss is a missing entry, and you only find them by feeding it real transcripts.

does. Here the code isn't wrong so much as opinionated. goes comes out as goe/s and does as doe/s, two morphemes each. Some manuals count does as a single irregular form, the way went is. If you follow one of those, you fix it by hand. A tool can't settle that for you, and pretending it can would be worse than leaving the slash visible.

The cosmetic bug that doesn't matter

dishes becomes dishe/s, ladies becomes ladie/s. Linguistically the cut is dish/es, and ladies is lady plus a plural ending, but MLU only cares how many pieces there are, not where the cut falls, and both versions give 2. I left it, because the clinician sees the marked text and dishe/s is obviously "dish plus plural" to a human reader. Getting the boundary right would mean a small stemmer for -es, -ies and doubled consonants (running, stopped), and none of that changes a single count.

Why not a real stemmer or an NLP library

A Porter stemmer would give hike from hiking, which looks nicer, but it also strips derivational endings like -ness and -ful, which usually don't count as separate morphemes in this kind of sample. A part-of-speech tagger would tell bored (adjective) from jumped (verb), but child speech is full of fragments that taggers handle badly, and the whole page runs offline in the browser with no model to download. Three regexes plus lists that grow from real misses turned out to be the more honest tool: when it is wrong, you can see exactly why.

If you want to try it on your own sample, it's the MLU calculator on SLP Age Calc. Paste one utterance per line, press auto-mark, and check the slashes before you trust the number.