Why I Banned Manual kill — Bot Process Discipline

The Day I Killed the Wrong Python

My MEV bot has a habit of revealing how sloppy my Linux hygiene really is. The latest reminder came on a Tuesday morning when I tried to restart the bot, ran the wrong pgrep, and SIGKILL'd a different long-running Python script — one that had been quietly rebuilding a local price cache for the last forty minutes. I lost the cache. I lost the morning. And I lost my excuse for keeping the bot under nohup.

I am writing down this new rule so I cannot weasel out of it later: no manual kill against the bot. Ever. If I need to stop a process and the only tool I have is kill -9 $(pgrep -f arb), the operations layer is broken and that is what I should be fixing, not the bot.

How the Mess Got Built

The path here is embarrassingly typical. Early in the project, when nothing worked yet, I just wanted the bot to keep running while I closed my laptop. The internet's first answer is always the same: nohup ./bot &. So I ran it under nohup, redirected logs to a file in the project folder, and went to bed.

That first night, it worked. The next morning the bot was still alive, the log file had grown, and I was very pleased with myself. So pleased that I didn't notice I had set up exactly the trap that every Linux veteran warns about.

A write-up on nohup calls out the failure modes plainly: no built-in auto-restart, default log location with permission issues, and disk space exhaustion from unbounded log files. I hit all three within two weeks. The bot crashed silently after an RPC outage and stayed dead until I checked manually. The log file ballooned to several gigabytes because I never set up rotation. And when I tried to restart, I found two copies of the bot running because I'd forgotten that the previous session's process had never actually died — it had just stopped logging anything useful.

A survey of background-service techniques [https://www.sobyte.net/post/2023-03/linux-services-in-the-background/] is even blunter about where nohup belongs in the toolbox: fine for short-lived tasks, wrong for anything that needs "output log cutting, running users, booting." A trading bot needs all three.

nohup Is Not Operations — It Is Abandonment

Let me be honest about what nohup ./bot & actually does. It detaches a process from the controlling terminal and points its output at a file. That is the entire feature set. There is no supervisor. There is no health check. There is no restart. There is no clean shutdown path. There is, crucially, no record of which process is yours when something goes wrong.

A comparison piece puts it more diplomatically: nohup is appropriate for "ad-hoc background tasks that are temporary and non-critical" — bulk file syncs, one-off database migrations, debugging sessions. Systemd, by contrast, is the "preferable way to run apps and web servers as daemon processes in Linux, in production." An MEV bot that touches real money on every successful run is the second category, not the first.

A forum post I came across while researching this captured the long-tail consequence in a single line: "Forgotten nohup processes can become zombie processes consuming resources months later." That is not hypothetical. I have my own collection of python3 PIDs whose origin I genuinely cannot identify because I started them under nohup six weeks ago and the shell history is long gone.

The Manual kill Anti-Pattern

The twin sin of nohup is pgrep | kill. Once a process is orphaned and has no PID file, the only way to find it is by name — and as I learned on Tuesday morning, names overlap. My MEV bot is a Python process. My data backfill job is a Python process. The Jupyter kernel I left open from last weekend is a Python process. pgrep python returns all of them. pkill -f arb matches the bot, the backtest script, and any text editor that happens to have "arb" in an open buffer's title.

Every shortcut here is a foot-gun. kill $(lsof -ti :8080) assumes only one thing is bound to that port. pkill -u myuser python will happily nuke anything I forgot about. kill -9 skips the SIGTERM handler entirely, which means whatever cleanup the bot was supposed to do — flushing logs, closing RPC connections, returning rent-reclaimed SOL to the main wallet — simply does not happen.

The Linux best-practice guidance that keeps showing up across the references I read is consistent: install a SIGTERM handler that "set[s] an atomic flag, avoid heavy work in the handler by just flipping a flag and returning, and have the main loop poll that flag and perform orderly cleanup: close files, remove PID file, flush logs, and exit with the correct code." kill -9 defeats this entire design. The process gets a SIGKILL it cannot trap, and whatever state it was in mid-flight is the state it stays in.

For a bot that holds open WebSocket subscriptions, in-flight transaction submissions, and a working memory of pool reserves, ungraceful death is not free. At best, the next startup spends extra time re-syncing state. At worst, it leaves the on-chain side in a partially-set-up configuration that costs SOL to clean up.

Why PID Files Are Trickier Than They Look

The obvious fix to "I can't find my process" is "write a PID file." Start the bot, capture $!, write it to bot.pid, and kill $(cat bot.pid) whenever I want to stop it.

This works until it doesn't. The two things I had not internalized before reading Guido Flohr's "Never Delete Your PID File!" were:

1. Stale PID files are the dominant failure mode. If the bot crashes without cleaning up, the next start sees a PID file, assumes the bot is already running, and refuses to start. Or worse: it doesn't check at all, and now two bots are racing each other to land the same arbitrage.
2. Deleting the PID file is itself a race condition. Flohr walks through the unlink-then-unlock and unlock-then-unlink orderings and shows that both can result in two processes acquiring locks on different file descriptors that happen to share a name. His one-line summary is the whole post: "The best way to delete a pid file (or rather a lock file) is not to delete it."

The technical reason is subtle. On Linux, unlink() only removes the directory entry; the kernel keeps the inode alive as long as any file descriptor still references it. So one process can hold a lock on an inode that no longer has a name, while a second process creates a brand-new inode under the same name and locks that. Two locks. Same path. Different files. Both processes think they have exclusive access.

Research summaries from the same problem space estimate that stale PID files from unclean shutdowns account for the overwhelming majority of "already running" false positives in custom singleton scripts. That matches my experience exactly — every time I have rolled my own PID-file singleton check, I have eventually had to add a kill -0 liveness probe, then a /proc/$PID/cmdline comparison, then a force-override flag, and at that point I have written a worse version of something the kernel already provides.

What the Kernel Already Gives You

The "something the kernel already provides" is flock. A write-up on flock [https://www.commandinline.com/shell-script-lock-file-flock/] puts the value proposition crisply: "flock from the util-linux package is the modern approach. It operates on a file descriptor and the kernel releases the lock automatically when the process dies — no stale locks."

The core pattern is small enough to memorize:

#!/usr/bin/env bash
set -euo pipefail

LOCKFILE="/run/user/$(id -u)/mybot.lock"
exec 200>"$LOCKFILE"

if ! flock -n 200; then
  echo "another instance is running, exiting" >&2
  exit 0
fi

echo $$ >&200
exec ./bot

Three things are happening. First, file descriptor 200 is opened against the lock file. Second, flock -n tries to acquire the lock without blocking; if anyone else holds it, the script exits cleanly instead of stacking duplicate runs. Third, the script execs into the bot, replacing itself so that the bot inherits the open file descriptor and the lock. When the bot dies — gracefully, ungracefully, OOM-killed, or kill -9'd — the kernel closes the descriptor and the lock evaporates. No stale state. No file to delete. No race condition.

The comparison table in the same article puts the contrast with hand-rolled PID files in stark terms:

• flock: no stale-lock risk, automatic cleanup, no race condition.
• PID file only: stale-lock risk present, cleanup requires trap, race condition vulnerability.

For a bot whose worst failure mode is two copies racing the same opportunity, this is not a stylistic preference. It is a correctness requirement.

Where the Lock File Should Live

A guide on storing PID files for user-run daemons lays out the candidate directories with their trade-offs. The summary:

• /run/user/$UID/ is the right answer for session-scoped daemons. It's user-owned, mode 700, managed by systemd, and tmpfs-backed.
• ~/.local/run/ is the right answer for daemons that need to survive logout — disk-backed, follows the XDG base directory spec.
• /tmp is the wrong answer because it's world-writable and gets cleaned on reboot in a way that can interact poorly with locking.
• /var/run/ is the wrong answer for non-root daemons because it's reserved for system services.

A detail worth lifting straight from the same article: PID and lock files should be chmod 600. The lock content itself is not sensitive, but the file's existence reveals operational details, and a permissive mode invites another local process to clobber it.

Why I Am Not Going to Write a Supervisor From Scratch

There is a tempting next step here: write a small shell wrapper that handles start, stop, status, restart, log rotation, and PID tracking. I am not going to do that, and the reason is that this is exactly the kind of "five lines of bash that grew into a thousand" project that ends with a brittle, undocumented, single-server-only operations layer. Every production-grade reference I looked at points to the same answer for long-running services: hand the lifecycle off to a real process supervisor.

The field has converged on a small set of options. A 2026 process-manager comparison [https://oxmgr.empellio.com/blog/process-manager-comparison] benchmarks the major ones on memory footprint and recovery speed. The standout figure for my purposes is the systemd column: zero additional resident memory, because it is built into the OS, and a median cold startup measured in tens of milliseconds. Other supervisors each have their use cases, but for a single-language single-host bot, systemd has the smallest blast radius and the deepest integration.

The pattern from the same survey that matters more than the absolute numbers is the qualitative one. Real supervisors give you four things that no hand-rolled wrapper reliably gives you:

1. Restart policy. Systemd's Restart=on-failure with backoff is one line in a unit file. The equivalent in shell is a retry loop with timeout, jitter, and crash-loop detection — and it's the kind of code I will absolutely get wrong.
2. Standard logging. journalctl -u mybot is a single command that gives me filtered, time-aware, rotated logs. No bespoke tail -F mybot.log | grep ERROR | less choreography.
3. A defined shutdown contract. systemctl stop mybot sends SIGTERM, waits for TimeoutStopSec, then escalates to SIGKILL. That's the entire shutdown lifecycle — defined, documented, and predictable.
4. An audit trail. Every start, stop, crash, and restart is logged with timestamps. The next time something weird happens at 3 a.m., I do not have to reconstruct the timeline from last, dmesg, and my own memory.

The one trade-off worth being honest about: systemd is Linux-only and its unit-file syntax is verbose. For my use case — one Linux box running one bot — neither of those is a deal-breaker. For someone running the same bot across mixed environments, a cross-platform supervisor might be the right choice. The principle is the same either way: use something that was designed for this, not something I built in a panic on a Tuesday night.

The SIGTERM Handler Is the Real Test

Getting the bot under a supervisor only does half the job. The other half is making sure the bot itself responds correctly when it receives SIGTERM.

The references I read agree on the pattern: signal handlers should be small. They should flip an atomic flag and return immediately. The main loop checks the flag at the top of every iteration and, when it's set, runs the shutdown sequence: stop accepting new work, finish in-flight work or roll it back, flush logs, release locks, exit with a meaningful status code.

For my bot, "in-flight work" has a specific meaning. If a transaction has been signed and sent but not yet confirmed, I have to decide whether shutdown waits for confirmation or accepts an unknown outcome. If a WebSocket subscription is in the middle of a backfill, I have to decide whether to finish or restart from scratch on the next boot. None of these decisions can be made in the half-second between SIGTERM and SIGKILL if I haven't designed for them up front.

The asymmetry that finally convinced me to take this seriously: a graceful shutdown costs me a few extra seconds of systemctl stop time. An ungraceful one costs me whatever the worst-case in-flight state happens to be. On a Solana MEV bot, the worst case includes orphaned on-chain accounts that have to be cleaned up by hand, which is the most aggravating kind of work because it is both tedious and traceable to my own carelessness.

The Rule, Restated

So here is the rule I am writing down, with the reasoning attached so future-me cannot pretend he forgot:

1. No nohup. Long-running services do not belong under a tool whose entire job is "ignore SIGHUP."
2. No manual kill against the bot. If I need to stop it, the command is systemctl --user stop mybot. If that doesn't work, the bug is in the unit file or the SIGTERM handler, and that is what I fix.
3. No PID-file deletion in cleanup code. Either use flock and let the kernel handle it, or accept that the PID file will be stale on next start and design the startup script around that.
4. No multiple ways to start the bot. One entry point, one unit file. If I'm tempted to add a second "just for debugging," the debugging story belongs inside the same entry point, gated by an environment variable.
5. No kill -9 unless I have already tried kill -TERM and waited. And if kill -9 is the only thing that works, the bot has a bug in its shutdown path, full stop.

This is the kind of rule that feels like overkill until the day it isn't. Tuesday morning's misfired pgrep was that day for me. The cache I lost was recoverable in a couple of hours. The next mistake might not be.

What This Means for the Rest of the Stack

The operational discipline question turns out to be entangled with several other things I've been putting off. Logging configuration has to be redesigned so that the supervisor handles rotation instead of the bot writing into ever-growing files. The startup script has to do environment validation before invoking the bot, so a misconfigured run fails fast instead of crash-looping under the supervisor's restart policy. Crash-loop detection has to be wired up to whatever alerting I'm using, because a bot that keeps crashing and being restarted every few seconds is a worse problem than a bot that crashed once and stayed down.

Each of these is a small piece of work. None of them are interesting. All of them are the kind of thing that, taken together, separate "I have a script that runs on my server" from "I have a service." And until the bot is a service, every story about it is going to end with me losing data because I typed the wrong pgrep at 9 a.m.

The path from nohup ./bot & to a real operational posture is short and well-documented. The reason I haven't walked it is the same reason most solo developers don't: it feels like infrastructure work that isn't moving the bot forward. The cost of not walking it is the same cost every veteran sysadmin has been warning about for thirty years — it just takes a while to feel it when you're the only one running things.

Key Takeaways

• nohup is for ad-hoc work, not production services. Use it for one-off syncs and migrations; do not use it for any process whose downtime costs you money or correctness.
• Hand-rolled PID-file singletons are race-prone. The unlink/lock ordering problem on Linux means delete-based cleanup has corner cases that are easy to hit and hard to debug. flock against an open file descriptor sidesteps the entire class of bugs.
• /run/user/$UID/ is the right home for user-scoped lock files. Avoid /tmp (world-writable) and /var/run/ (system-reserved).
• A real supervisor — systemd in most cases — replaces a thousand lines of fragile shell. You get restart policy, structured logging, defined shutdown lifecycle, and an audit trail without writing any of them yourself.
• kill -9 is a failure of the shutdown path, not a tool. If SIGTERM doesn't bring the bot down cleanly, the bug is in the bot, not in the kill command. Fix the SIGTERM handler so SIGKILL never has to be the answer.