Writing this up because the symptoms looked like four unrelated outages and the actual cause was one number.
We run a service in a pod that shells out a lot (az, kubectl, some bash pipelines). One morning it started throwing errors from everywhere at once:
- DB client:
connection reset by peer - shell runner:
fork/exec /usr/bin/bash: resource temporarily unavailable - a fetcher: “config not initialized”
- a cloud integration marked INVALID with
az login failed and 500+ retries behind it
Different errors, different components, same minute. And the liveness probe was green the whole time, so nothing restarted it.
resource temporarily unavailable is EAGAIN. On fork it means the kernel wouldn’t give the container another PID. The pod had hit its pids cgroup limit. Everything that needed to spawn a process was failing, each in its own words. The health endpoint is a plain HTTP handler that never forks, so it answered 200 right through the outage.
What was eating the PIDs: subprocesses that never exited. Some execs had no timeout at all. The one that did have a timeout killed the bash -c wrapper on expiry, but not the az and kubectl it had started, which got reparented and kept running. kubectl in particular was hanging forever on auth for clusters where our service principal was missing a role. A few leaked per scan, a scan every hour, for weeks.
How we confirmed it from inside the pod:
cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max # cgroup v2
cat /sys/fs/cgroup/pids/pids.current /sys/fs/cgroup/pids/pids.max # cgroup v1
ps -eo pid,etime,cmd | sort -k2 -r | head # oldest processes first
current was at max, and the oldest entries were weeks-old kubectl processes.
What we changed:
- Every exec gets a context with a deadline, and cancellation kills the whole process group, not just the direct child (in Go:
Setpgid on the child, signal the negative pid). This was the actual bug. - Explicit
podPidsLimit on the kubelet (--pod-max-pids), low enough that a leaking pod dies before the node does. We were on the default, which is high enough to take the node down first. - Alert on
container_processes from cAdvisor trending up against the limit. It had been climbing in a straight line for weeks and nobody was looking at it. - Considering a liveness probe that actually forks something, since an HTTP 200 told us nothing here.
The lesson I keep coming back to: when several unrelated things break in the same minute, stop reading the individual errors and look for what they share. Here it was the kernel saying no to fork.
Curious what people use for the liveness side. Is “probe that execs something” a thing, or do you just alert on the pids metric and leave liveness alone?