One box is enough

Everyone we meet assumes we run a fleet of cloud instances. We run one dedicated box in a Falkenstein data center, and it hosts every production site, database and bot we operate.

This is not nostalgia. It is a forcing function. When everything lives on one machine, you cannot hide sloppy engineering behind autoscaling. A memory leak is a neighbor problem within the hour. A runaway cron shows up in everyone's latency. So the discipline has to live in three places, and all three are enforced by config, not by promises.

1. Every service gets a ceiling

# systemd override, applied to every unit, no exceptions
[Service]
MemoryMax=512M
CPUQuota=80%
Restart=on-failure
RestartSec=5

The ceiling is not there to be hit. It is there so that when something does go wrong, the kernel kills one service instead of the OOM killer choosing a victim at random among forty.

2. Every scheduled job gets a lock, a timeout, and a heartbeat

*/5 * * * * flock -n /run/lock/sync.lock timeout 240 /opt/jobs/sync.sh \
            && date > /opt/health/sync.ok

The lock stops a slow run from stacking behind itself. The timeout stops a hung run from holding the lock forever. The heartbeat file is the interesting one: a separate watchdog checks the mtime of every .ok file and treats staleness itself as the alarm. Jobs do not fail loudly in production. They fail by not running.

3. Routing is one file you can read

example.com {
  encode zstd gzip
  root * /srv/sites/example
  file_server
  handle /api/* { reverse_proxy 127.0.0.1:3160 }
}

Caddy terminates TLS for every domain and routes by hostname. Behind it, apps are containers or static directories. Databases are plain Postgres and MySQL, one instance per engine, many schemas. Nothing exotic, everything inspectable with standard tools.

What one box buys you: latency you can reason about (every internal hop is a loopback), a single filesystem of truth (debugging is grep, not a distributed tracing seminar), and honest capacity planning (htop is the dashboard). The catch is that the box becomes precious, and precious things need paranoia. That paranoia, backups we actually restore, silence detection, blast-radius caps, is most of what the rest of these notes are about.