How to Monitor systemd Failures and Send Automatic Alerts

When a critical background service crashes on a Linux server, systemd logs the event and, depending on its unit definition, either attempts a restart or enters a failed state. Unless you are actively checking systemctl status or monitoring server dashboards, that failure often goes unnoticed until downstream systems break or users report an outage.

While systemd provides native hooks like the OnFailure= directive, configuring it manually across dozens of individual unit files is tedious and prone to configuration drift. Developers often need an automated, centralized way to catch unit failures across all services on a host, capture recent log lines, determine whether a service is trapped in a crash-restart loop, and forward that diagnostic context to an endpoint like a webhook, an ntfy topic, or an email mailbox.

UnitFail Notifier is a standalone Python tool that solves this by managing global systemd drop-in configurations, introspecting failed services, and dispatching diagnostic payloads whenever an issue occurs.

Install & Usage

The script relies entirely on standard Python modules (subprocess, urllib, smtplib, email, json, argparse). You do not need to run pip install because there are no third-party package dependencies.

Installation

To use the tool, download the single Python script to your server, save it as unitfail_notifier_systemd_failure_alerts.py, and ensure Python 3 is installed.

# Verify the script is runnable and view available subcommands
python unitfail_notifier_systemd_failure_alerts.py --help

Previewing the Installation (Dry-Run)

By default, file modifications run in dry-run mode so you can verify paths and generated unit files before applying changes to your system.

python unitfail_notifier_systemd_failure_alerts.py install

This command outputs the planned drop-in configuration and the notification template service to standard output without altering /etc/systemd/system.

Applying the Drop-In Configuration

To install the global drop-in and the notification template service, execute the install command with elevated privileges and the --apply flag:

sudo python unitfail_notifier_systemd_failure_alerts.py install --apply

This writes:

1. /etc/systemd/system/service.d/10-unitfail-onfailure.conf – A systemd drop-in that assigns OnFailure=unitfail-notify@%n.service to all service units.
2. /etc/systemd/system/[email protected] – An instantiated systemd template unit that invokes unitfail_notifier_systemd_failure_alerts.py whenever any service enters a failed state.

After writing these files, the script automatically triggers systemctl daemon-reload.

Testing and Manual Dispatch

You can invoke the notifier manually against any existing unit to verify alerts without waiting for a production failure:

python unitfail_notifier_systemd_failure_alerts.py notify --unit nginx.service --config /etc/unitfail/config.json

Removing the Configuration

If you want to remove all systemd modifications, run the uninstall command:

sudo python unitfail_notifier_systemd_failure_alerts.py uninstall --apply

This removes the drop-in file and the notification template service, then runs systemctl daemon-reload to return the system to its initial state.

How It Works

UnitFail Notifier functions in two main phases: systemd configuration management and failure event handling.

1. Global Drop-In and Template Setup

Rather than editing existing service units individually, the tool generates a drop-in file that applies to all services under /etc/systemd/system/service.d/:

def generate_dropin_content() -> str:
    """Generate systemd service global drop-in configuration."""
    return (
        "# Generated by UnitFail Notifier\n"
        "[Unit]\n"
        "OnFailure=unitfail-notify@%n.service\n"
    )

The matching template unit triggers the Python script using systemd's %I specifier (which passes the name of the failed unit):

def generate_template_content(exec_path: str, config_path: str) -> str:
    """Generate systemd notify template with loop prevention and unprivileged fallbacks."""
    safe_exec = shlex.quote(exec_path)
    safe_cfg = shlex.quote(config_path)

    return (
        "# Generated by UnitFail Notifier\n"
        "[Unit]\n"
        "Description=UnitFail Alert Notification for %I\n"
        "After=network-online.target\n"
        "Wants=network-online.target\n"
        "OnFailure=\n"
        "\n"
        "[Service]\n"
        "Type=oneshot\n"
        f"ExecStart={safe_exec} notify --unit %I --config {safe_cfg}\n"
        "StandardOutput=journal\n"
        "StandardError=journal\n"
    )

Notice the line OnFailure=. By explicitly setting OnFailure= to an empty value inside the notification template unit, the script ensures that if the notification process itself encounters an error (such as a transient network timeout), it will never trigger a recursive cascade of failure alerts.

2. State Extraction and Loop Detection

When triggered, the script gathers operational metadata using systemctl show:

def inspect_unit_properties(unit_name: str) -> Dict[str, str]:
    """Extract key execution state and restart properties using systemctl show."""
    properties = {
        "Id": unit_name,
        "ActiveState": "unknown",
        "SubState": "unknown",
        "Result": "unknown",
        "NRestarts": "0",
        "ExecMainStatus": "0",
        "Restart": "no",
        "MainPID": "0",
    }
    if not is_systemd_available():
        return properties

    props_to_fetch = [
        "Id", "ActiveState", "SubState", "Result",
        "NRestarts", "ExecMainStatus", "Restart", "MainPID"
    ]
    cmd = ["systemctl", "show", unit_name, "--no-pager"]
    for p in props_to_fetch:
        cmd.extend(["-p", p])

    rc, out, _ = run_cmd(cmd)
    if rc == 0:
        for line in out.splitlines():
            if "=" in line:
                k, v = line.split("=", 1)
                properties[k.strip()] = v.strip()

    return properties

It queries the last 20 log lines from journalctl -u <unit> and evaluates whether the unit is flapping or caught in a boot-loop:

is_restart_loop = n_restarts >= 3 and props.get("ActiveState") != "active"

If the restart count is 3 or more while the service remains non-active, the payload marks restart_loop as True.

3. Multi-Channel Alert Dispatch

The tool packages these findings into an alert payload and dispatches it via one of three built-in dispatch channels:

• Webhook: Sends a standard JSON POST request with the complete error dictionary.
• ntfy: Formats the log snippet and host metadata, setting priority to urgent if a restart loop is detected.
• SMTP: Uses smtplib and email.message.EmailMessage to send a plain-text email with TLS encryption.

Limitations

Before deploying this script across your infrastructure, keep the following constraints in mind:

• Requires systemd as Init: The introspection methods and drop-in paths rely directly on systemctl and journalctl. The tool cannot monitor systems using alternative init systems.
• Root Access for Drop-Ins: Installing or removing drop-in units in /etc/systemd/system/ requires administrative permissions (sudo).
• Synchronous Execution: The alert dispatch functions (send_webhook, send_ntfy, send_email) make blocking network calls with a default 10-second timeout. If the host experiences total network loss, the one-shot notification unit may time out.
• No Non-Service Monitoring via Drop-Ins: The drop-in file is placed in /etc/systemd/system/service.d/, which applies exclusively to .service units. Other unit types (such as .socket or .mount) are not automatically hooked by this drop-in.
• Basic Authentication Only: The built-in SMTP client supports standard TLS and basic username/password authentication; modern token-based OAuth2 flows are not implemented.

FAQ

Can I use this script without root access?

You can use the introspection and alerting features without root privileges if your user can run systemctl show and journalctl -u. However, creating the global drop-in configuration in /etc/systemd/system/service.d/ requires root privileges.

What happens if the notification script itself fails?

Infinite loops are explicitly blocked. The template configuration [email protected] generated by the script contains an empty OnFailure= directive, which clears any inherited failure handlers for the alert task itself.

Do I need to restart my services after installing the drop-in?

No. Once systemctl daemon-reload is executed, systemd automatically picks up the service.d drop-in configuration for all existing and newly created service units. Existing running services will trigger the alert if they transition into a failed state.

Full version and the rest of the toolkit: UnitFail Notifier systemd failure alerts on PyForge. Product page: https://py-forge.com/products/unitfail-notifier-systemd-failure-alerts