fatal error: concurrent map writes -- the Go crash recover() can't catch

Day 25/90: Mutexes and shared state

90 Day Go Challenge: Zero to Production

This morning a service of mine started returning 502s under load, and the pod logs ended with a line that is not a panic:

fatal error: concurrent map writes

The word fatal is the whole story. This is the runtime aborting the process, not a panic you can intercept. recover does nothing for it. Here is the full path from crash to fix.

The bug

A read-through cache in front of a slow upstream. On a miss it fetches and writes the result back. The cache was a plain map:

type Registry struct {
    items map[string]int
}

func (r *Registry) Bump(name string) {
    r.items[name]++ // read-modify-write on a shared map
}

Single-threaded, this is correct. But Go's HTTP server runs each request in its own goroutine, so two requests that miss at the same time call Bump at the same time, and now two goroutines write one map.

Why the runtime kills the process

A Go map is buckets, a count, and a load factor. Crossing the load factor makes the next write trigger a rehash that relocates entries between buckets. A second concurrent write in the middle of that rehash leaves the structure half-rebuilt -- a corrupted map that can return wrong values or read freed memory for the rest of the program's life.

The runtime would rather crash than corrupt. It carries a cheap concurrency check on maps, and when it sees two goroutines writing at once, it aborts immediately with fatal error: concurrent map writes. A loud crash now beats a silent wrong answer later -- and that design choice is exactly why recover cannot save you.

Prove it with -race

You don't have to reproduce the crash to find the bug. The race detector instruments memory accesses:

func main() {
    r := &Registry{items: map[string]int{}}
    done := make(chan struct{})
    for i := 0; i < 8; i++ {
        go func() { r.Bump("hits"); done <- struct{}{} }()
    }
    for i := 0; i < 8; i++ {
        <-done
    }
}

Run go run -race main.go:

==================
WARNING: DATA RACE
Read at 0x00c0000b4010 by goroutine 9:
  main.(*Registry).Bump()
Previous write at 0x00c0000b4010 by goroutine 8:
  main.(*Registry).Bump()
==================

It names the racing goroutines and the line, even on a run that survived. Put go test -race ./... in CI and you catch this class of bug before it ships.

The fix: sync.Mutex

type Registry struct {
    guard sync.Mutex
    items map[string]int
}

func (r *Registry) Bump(name string) {
    r.guard.Lock()
    defer r.guard.Unlock()
    r.items[name]++
}

sync.Mutex gives mutual exclusion: one holder at a time. The shape is Lock, then defer Unlock, then the critical section. The defer releases the lock on every exit path -- early return or panic -- so you never strand a lock and deadlock every later caller. And readers must lock too: an unguarded read still races a guarded write.

RWMutex, Once, and the channel rule

When reads vastly outnumber writes (and you have measured it), sync.RWMutex lets readers share an RLock while a writer keeps an exclusive Lock. When initialization must happen exactly once, sync.Once's Do runs the function once no matter how many goroutines race in -- lazy init without the hand-rolled double-checked-locking bugs.

And when you are not protecting shared state but handing a value from one goroutine to another, reach for a channel instead: don't communicate by sharing memory; share memory by communicating. The question that picks the tool: am I protecting state many goroutines touch, or passing ownership from one goroutine to the next? The first is a mutex; the second is a channel.

Tomorrow: sync.WaitGroup, for waiting on the goroutines I keep launching.