Restart=always Is a Recovery Policy, Not a Health Check

By Bo

When an AI gateway stops answering, changing its restart policy is tempting. I have operating notes covering several failures that looked similar from the chat window: an out-of-memory kill, an invalid messaging credential, and a startup migration failure. They needed different responses. A supervisor can bring a process back, but it cannot decide whether that process can complete useful work.

During a recent Linux gateway maintenance session, I checked the effective user service and found that Restart=always was already configured. I added an explicit override with a longer restart delay and reloaded the unit definitions. The gateway kept the same process ID. That was a configuration change, not evidence that I had recovered a failed service.

Here is the distinction I now use when reviewing a gateway service.

Start with the effective configuration

The following commands are illustrative diagnostics for an existing service named agent-gateway.service. Substitute your actual unit name. I have not run this anonymized example against a separate demonstration service.

systemctl --user cat agent-gateway.service
systemctl --user show agent-gateway.service \
  -p ActiveState -p SubState -p MainPID \
  -p Restart -p RestartUSec -p NRestarts -p DropInPaths
journalctl --user -u agent-gateway.service --since '30 min ago'

I look for overrides before editing the main unit. A remembered configuration file is weaker evidence than the settings the manager has loaded. I also record the process ID before a change, so I can distinguish a configuration reload from a process replacement afterward.

Journal output may contain private messages or credentials. Inspect it locally and redact it before attaching an excerpt to a public issue.

Choose the policy for the workload

For a continuously running gateway, restarting after an unexpected clean exit can be useful. According to the systemd service documentation, Restart=always covers clean and abnormal exits, while Restart=on-failure covers failure conditions. An explicit stop through the service manager does not trigger that automatic restart.

For an existing long-running unit, this is an illustrative drop-in:

# ~/.config/systemd/user/agent-gateway.service.d/restart.conf
[Service]
Restart=always
RestartSec=10s

The delay here is an example policy value. It is not a measured optimum for gateways. A persistent startup error needs diagnosis, and an aggressive retry loop can obscure the useful part of the log.

I would not copy this policy into a batch job that should finish and stay finished. The desired lifecycle comes first; the directive follows it.

Keep reload, restart, and readiness separate

systemctl --user daemon-reload makes the manager reload unit definitions; it does not replace the running gateway. In the maintenance record, a separate runtime override was loaded while the live process still used the previous runtime executable. Checking a version in a shell would have missed that distinction.

After an authorized restart, I want evidence about the running executable, the loaded settings, and application behavior. A package installed on disk proves installation. It does not prove that the service has started using it.

This matters especially with user services, where an interactive shell can have a different environment from the service manager. I check the unit's configured executable and environment rather than relying on whichever binary my terminal finds first.

Diagnose the failure that survives a restart

My earlier incident notes record an out-of-memory kill. Automatic recovery brought the gateway back, but the memory pressure still required attention. Restarting was recovery from the immediate interruption, not removal of its cause.

Another incident involved an unauthorized messaging credential. A process supervisor cannot make that credential valid. Repeatedly restarting the gateway would distract from the channel error.

A startup migration problem produced a different pattern: repeated failed starts followed by a start limit. The systemd unit documentation describes start-rate limiting separately from the restart policy. Resetting the failure state before resolving the startup error only gives the same broken startup another opportunity to fail.

These are historical incident observations, not a claim that every silent bot has one of these causes. I use them to decide what evidence to collect next.

Verify a useful path

My maintenance check confirmed that the service was active and that the gateway listener remained bound locally. Existing journal entries also showed messaging traffic. I did not send a new test message during that session, so I would not describe it as a fresh end-to-end messaging test.

For a planned acceptance check, I would separate process state, listener availability, channel authentication, and one harmless request completing through the application. A listener can exist while an upstream model is unavailable. A model can answer while a messaging channel cannot deliver its reply.

Finally, automatic process recovery says nothing about whether a user manager starts at boot and remains available after logout. That lifecycle needs its own host-specific check. I do not infer unattended operation from a successful interactive session.

The useful lesson from Restart=always is modest: define which process exits should recover automatically, then verify the application separately. My strongest maintenance notes say exactly what changed, what stayed running, and which user-visible behavior was actually observed.