vLLM for one person: serving a small model

vLLM is the serving engine I use for the local models in my stack, and the honest summary is this: for one person running a small Japanese-optimized model for embeddings and light inference, vLLM gives you high-throughput, continuously batched serving that you can point at and forget — once you treat the model as a service instead of a script. The hard part is not starting vLLM. It is remembering that a served model now needs monitoring, restarts, and version pinning, just like anything else you run.

A lot of one-person setups never make that leap. They run a model inline in a Jupyter process, or start a server by hand when they need it. That works until it does not. Here is the workflow I landed on.

Why vLLM for a small model

The pitch of vLLM is high-throughput inference with continuous batching. What that means in practice: rather than processing one request at a time or blocking while a batch fills, the engine keeps a queue of pending requests and packs new work into the gaps as each generated token completes. For my workload — repetitive inference, embeddings, and light extraction that runs constantly in the background — that throughput matters more than raw latency on any single request.

It is suited to exactly the kind of jobs I run: many small requests that arrive steadily rather than one large burst. Continuous batching turns that stream of work into near-constant utilization of the GPU.

Picking the right model

I serve a small Japanese-optimized model. The choice of Japanese optimization is not cosmetic: for Japanese text, generic English models degrade noticeably on embeddings and generation alike. Matching the model to the language of your documents is the difference between retrieval that works and retrieval that quietly loses precision.

For a one-person setup the model should be small enough that you are never fighting your hardware. A small model on modest hardware beats a large model that you cannot keep running. The volume of embeddings in my memory stack is high, so throughput per token is the metric that matters, not benchmark scores.

Serving makes it a service

The mental shift is the part nobody warns you about. Before serving, a model is a thing you invoke. After serving, it is a service with an address, and services fail in new ways. Three disciplines made the difference for me:

• Monitoring. A silent serving process is a liability. Watch request latency, error rate, and queue depth, because those tell you when the model has degraded before a user notices.
• Restarts. The process will eventually die or hang. Plan for restarting it cleanly, the same way you would any other long-running service.
• Version pinning. Pin the exact model and engine versions you serve. A silent upgrade can change output behavior and corrupt embeddings without you realizing it.

This is the same discipline I apply to the rest of the self-hosted stack — assume things break and make recovery boring. A served model is production infrastructure, so it gets treated like it.

Where it fits in the stack

The embeddings this model produces feed the memory layer, which is the highest-volume inference in my stack, so keeping it self-hosted and fast is a deliberate cost decision. Running it locally keeps that repetitive work off the paid path entirely.

Serving a model properly is the difference between a working demo and infrastructure you can rely on. The engine is the easy part; the operational habits around it are what make it dependable.

What does your smallest, most repetitive inference job look like — and is it still running as an unmanaged script?

Originally published at agentjournal.dev.