I accidentally built a 4-connection HTTP server

Fixing my server's HTTP/1.1 keep-alive violation introduced a much worse bug.

In my previous post, I wrote about how Ariadne, my custom C++ HTTP framework, was claiming to speak HTTP/1.1 but acting like HTTP/1.0 by immediately closing sockets after serving a single request.

To fix the keep-alive protocol violation, I restructured the server so that each worker thread owned a connection until the client explicitly requested Connection: close or the socket timed out.

Adding a CPU-Bound Endpoint

With persistent connections working, I added a CPU-intensive endpoint (GET /api/compute recursively calculating fib(30)) to the benchmark workload to test how the server handled mixed I/O and CPU-bound traffic.

The results were dramatically worse than expected. Throughput collapsed to roughly 40 requests per second, and k6 reported widespread timeouts, even for the lightweight routes like / and /api/user.

The recursive computation wasn't the bottleneck. The root cause was how the thread pool was initialized.

Before: One Thread, One Request

Before the keep-alive fix, a worker thread processed exactly one request and immediately closed the socket, returning to the thread pool:

  ThreadPool pool(std::thread::hardware_concurrency());

  while (serverRunning) {
    int clientFd = accept(serverFd, nullptr, nullptr);
    if (clientFd == -1) {
      if (errno == EINTR) {
        continue;
      }
      perror("accept");
      break;
    }
    pool.enqueue([this, clientFd] {
      Request req = parseRequest(clientFd);
      Response res(clientFd);
      executeMiddlewares(0, req, res);
      close(clientFd);
    });
  }

After: One Thread, One Connection

After implementing proper HTTP/1.1 persistent connections, each worker thread instead entered a continuous loop pinned to an entire TCP connection:

    pool.enqueue([this, clientFd] {
      while (true) {
        try {
          auto req = parseRequest(clientFd);
          if (!req)
            break;
          Response res(clientFd);
          bool keepAlive = true;
          auto it = req->headers.find("Connection");
          if (it != req->headers.end() && it->second == "close") {
            keepAlive = false;
          }
          if (keepAlive)
            res.set("Connection", "keep-alive");
          else
            res.set("Connection", "close");
          executeMiddlewares(0, *req, res);
          if (!keepAlive)
            break;
        } catch (const std::exception &e) {
          Response res(clientFd);
          res.status(400).send("Bad Request");
          break;
        }
      }
      close(clientFd);
    });

The Root Cause

The issue was std::thread::hardware_concurrency(). For the 4-core benchmark container, this resulted in a fixed pool of 4 worker threads.

Once each worker owned an entire persistent connection, the pool size became the hard limit on simultaneous active client connections. The server could only process 4 connections at a time, regardless of how lightweight the requests were.

During the benchmark, k6 generated up to 200 concurrent virtual users (VUs). Four client connections occupied the worker threads, and the remaining 196 waited in the accept queue. No additional requests could begin processing until an existing connection closed.

The expensive fib(30) endpoint merely exposed this architectural limitation by keeping each connection busy for longer. The thread pool had been sized for CPU parallelism rather than expected connection concurrency.

The fix was straightforward:

  ThreadPool pool(250);

At first, allocating 250 threads seemed expensive. At 8MB per thread stack, that implies 2GB of RAM. However, Linux reserves Virtual Memory for thread stacks, and Docker's memory limits only restrict Resident Set Size (Physical RAM). A thread only uses physical RAM for the stack space it actually touches. During the benchmark, total RAM usage by the container never crossed 15MB.

With the thread pool correctly sized to accommodate the 200 concurrent VUs, performance recovered completely. Throughput hit 756 req/s, and the p95 latency during the ramping load dropped to 3.77ms.

Where This Still Falls Short

A thread-per-connection model like this works, but it doesn't scale indefinitely. Every idle connection still holds a thread hostage even when it's doing nothing but waiting on the network. Get to a few thousand concurrent connections and you're paying real scheduling and context-switch overhead just to keep sockets open, long before any of them have actual work to do.

The real fix for that isn't a bigger thread pool, it's not blocking a thread per connection at all. An event-driven model using epoll lets a single thread monitor thousands of file descriptors and only wake up when a socket actually has data ready, instead of one thread sitting parked on each connection. That's the architecture Go's netpoller and Node's event loop are both built on, in different ways. I'll be implementing epoll-based I/O in Ariadne next, and writing that up when it's done.

I'll be back with more next week. Till then, stay consistent!