Ops errors

systemd watchdog for a Python trading bot

systemctl status can say active while the scan loop is wedged on a hung HTTP call. A watchdog kills that process so systemd can start a clean one.

This is operations, not a strategy. Auto-restart on a live key can hammer the exchange. Keep paper-trade on until the unit stays quiet for a full scan interval. See systemd and journalctl first.

What Restart= does not do

Restart=on-failure only fires if the process exits. A bot blocked on requests.get with no timeout is still PID-alive. systemd thinks it is fine. You are not scanning.

Watchdog unit

[Unit]
Description=OpenClaw trading bot
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=300
StartLimitBurst=5

[Service]
User=YOURUSER
WorkingDirectory=/home/YOURUSER/bots/openclaw
ExecStart=/home/YOURUSER/bots/openclaw/.venv/bin/python main.py
Restart=on-failure
RestartSec=15
EnvironmentFile=/home/YOURUSER/bots/openclaw/.env
WatchdogSec=180
NotifyAccess=main

[Install]
WantedBy=multi-user.target

• WatchdogSec=180 — if the process does not pet the watchdog in 3 minutes, systemd SIGABRT + restart. Pick something longer than one scan interval (often 300s). If your interval is 300s, use 400–450, not 30.
• StartLimitBurst=5 — a crash loop stops after five restarts in five minutes. That is the feature. Do not raise it to hide a bad import.
• NotifyAccess=main — only the main PID may send watchdog pings.

Pet it from Python

import os, time
from systemd import daemon

def maybe_watchdog():
    usec = os.environ.get("WATCHDOG_USEC")
    if not usec:
        return
    daemon.notify("WATCHDOG=1")

# inside the scan loop, after a successful poll:
maybe_watchdog()

Install systemd-python in the venv. Ping after the poll succeeds, not at the top of a while True that never reaches the exchange. A ping on a hung request is how you fake health.

If you refuse the extra package, write to the notify socket yourself:

import os, socket
s = os.environ.get("NOTIFY_SOCKET")
if s:
    sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
    sock.connect(s)
    sock.sendall(b"WATCHDOG=1")
    sock.close()

Prove it

sudo systemctl daemon-reload
sudo systemctl restart openclaw
systemctl show openclaw -p WatchdogUSec -p WatchdogTimestamp
journalctl -u openclaw -f

To test the kill: comment out the ping, wait WatchdogSec, watch journalctl for Watchdog timeout then a restart. Do this on paper-trade only.

Common mistakes

A watchdog restart is not “the bot recovered.” It is “the last process was declared dead.” Check journalctl before you turn paper-trade off. Trading can lose money. Not financial advice.

Keep reading