A 1-Second Health Check Took Down a DB

How a 1-Second Health Check Took Down a Database

AIPub Engineering Blog · Incident Report

It started with a report that image pulls were failing with 401 Unauthorized, and it ended somewhere that had nothing to do with authentication. Along the way we misidentified the cause three times, and all three times measurement corrected us.

Summary. A 1-second health check judged a briefly slow database to be dead and restarted it. After the restart, it judged the recovering database to be dead again, and fell into a loop.

We never measured why the database slowed down in the first place, so that part remains inference. What we did confirm is that the system was structured so that the moment anything slowed down, the database was the first to be starved.

01 — The thing that looked like an auth error

We got a report that pulls from the image registry were failing. The error read:

401 unauthorized - Server message: unauthorized: unexpected EOF

It looks like an auth error. But something was odd. It sometimes succeeded.

That one line is half the diagnosis. Wrong credentials fail 100% deterministically. A wrong password does not log you in one time out of three. If success and failure alternate, it cannot be a credentials problem.

So why the 401? An HTTP status code is not a cause, it is a bin. Whatever goes wrong during the authentication step, it gets dropped into the 401 bin on the way out. The unexpected EOF appended to it (Go’s io.ErrUnexpectedEOF, “the connection was cut before the response was fully read”) was the real hint.

At the same moment, the registry core logs showed this:

unauthorized: failed to connect to `host=<database> user=postgres database=registry`:
  server error (FATAL: the database system is in recovery mode (SQLSTATE 57P03))

Issuing a token requires the database, and that database was in recovery mode. That single line is enough to connect the 401 to the database.

A 401 is a bin, not a cause.

All three come out as the same 401 unauthorized. What tells them apart is not the status code but ① the original message appended behind it, and ② whether the failure is deterministic or intermittent. Intermittent success, on its own, erases the first box.

A status code only tells you where a symptom surfaced, not what went wrong. Every incident on the auth path converges to 401. So checking credentials first when you see a 401 almost always goes nowhere.

02 — Three wrong diagnoses

This is where the real reason for writing this post begins. We identified the cause wrongly three times. All three times, the mistake was promoting a plausible number to a cause.

① “Five minutes, that’s the ingress timeout.”

We asked how long it took to fail. “About five minutes.” Checking the ingress controller config, there was streamIdleTimeout: 300s. 300 seconds = 5 minutes. An exact match.

It wasn’t right. The actual interval was 5.7 minutes, a number that came from somewhere else entirely. We had found a config value that matched the number we were given, and that became confirmation bias. The most dangerous kind of error, because it looks well-grounded. This diagnosis actually led to one ingress config change. It was not the fix.

② “NFS is 93% full, so it must be slow.”

We found this in the logs:

LOG: syncing data directory (pre-fsync), elapsed time: 17.19 s

And the data directory was on NFS, 93% used. The explanation seemed to hold.

Again, no. Measuring the same metrics side by side against a healthy cluster, it didn’t hold up.

Everyday I/O performance was effectively identical. The 93% was correlation, not cause. And that 17-second pre-fsync was a value logged on the shutdown/startup path, with nothing to do with the health check. The health check is SELECT 1. It never touches disk. It answers from memory.

But even this exclusion was only half right.

Every fsync value we compared was from a quiet period. We never once measured the value while a 16GB pull was running. The interval where we confirmed “not slow” was wrong to begin with. In the middle of correcting a wrong judgment, we measured in the wrong window again. We come back to this story later.

③ “Our cluster never had restarts.”

We asserted “the pod has never restarted,” because we had seen RESTARTS 0 on the internal cluster. The problem environment’s describe pod showed:

Restart Count:   42

We used a value measured in one environment as a fact about another. When you cite a value as evidence, you have to state which environment it was measured in, every time.

03 — The interval named the cause

One line of describe pod was all it took.

Liveness: exec [/docker-healthcheck.sh] delay=300s timeout=1s period=10s #failure=3

First, what this line means.

There are four numbers. Here is what each one sets, because all the arithmetic that follows comes from here.

⚠️ One caveat: describe pod prints abbreviated names, and the actual spec field names are different. If you don’t line these up, you’ll get lost looking things up in the docs.

You can search the describe pod output all you like for initialDelaySeconds and never find it. There it’s written as delay. To see the actual field names, you have to look at the raw spec, like kubectl get statefulset <name> -o yaml. The rest of this post, and the YAML, use the spec field names in the right-hand column.

The handler is exec. The kubelet runs /docker-healthcheck.sh inside the container, and that script connects over loopback TCP with psql and runs SELECT 1. Exit code 0 means success.

What happens on failure.

When a single check exceeds timeoutSeconds, that round is merely recorded as a failure, with no action yet. Only after failureThreshold (3) consecutive failures does the kubelet kill the container. That is the exact path that increments the RESTARTS count.

When the container is killed, the entire database instance inside it goes down with it. This instance ran databases for multiple services together, so one container kill brought down all of those databases at once. And when it restarts without having shut down cleanly, PostgreSQL begins with crash recovery. The recovery mode (SQLSTATE 57P03) from section 1 is exactly that state.

The traces left in the problem environment’s describe pod:

141 is the number we never managed to explain in this incident. We revisit it in section 5.

Counting the intervals.

Pulling the outage timestamps from the logs, there were 16 over the 1 hour 26 minutes from 07:24 to 08:50. Laying out the 15 intervals between them as a distribution:

Not a single value fell outside 5–6 minutes. Average 5.7 minutes. An outage that fluctuated with load or request volume could not look like this. This regularity itself means the interval comes from a config value.

So we compared against the probe settings.

5.5 minutes and 5.7 minutes. It means the container never once survived past the probe grace period after starting. Killed at the same point every time, so the interval comes out exactly as the config value.

When something is killed periodically, you can find the cause with arithmetic like this. If the crash interval matches initialDelay + failureThreshold × period, the cause is the probe.

04 — The cycle that blocks recovery

The most serious part was elsewhere.

During crash recovery, PostgreSQL refuses connections. FATAL: the database system is in recovery mode (SQLSTATE 57P03) is normal behavior, because it must not accept queries before recovery finishes.

But the liveness probe sends a query by running psql via exec. So during recovery, the probe fails by design. Walking one lap around the cycle in order:

Once it fails, it cannot recover on its own.

No amount of added resources breaks this cycle, because ④ is not caused by resource shortage, it is PostgreSQL’s normal behavior. More CPU only reduces the frequency of ①. Once you enter ③, there is no way out.

Switching liveness to tcpSocket makes ④ disappear. The port is open even during recovery, so the probe passes and the DB can recover to the end. That one change breaks the cycle.

If you judge “is it alive” using a signal that obstructs recovery, recovery never finishes. The heart of this cycle is ④: the probe cannot distinguish “the DB is recovering normally” from “the DB is stopped.”

The log database system was interrupted while in recovery in section 8 is direct evidence of this cycle. It means it was shut down again in the middle of recovery.

05 — What the exit codes told us

We used exit codes to rule hypotheses out. And one we never managed to explain.

If it had been OOM, it would have to be 137 (with Reason: OOMKilled logged alongside). It was not a memory problem. That single check let us exclude one hypothesis with evidence. But mind the direction: OOM implies 137, but 137 does not imply OOM. A forced kill after exceeding the termination grace period is also 137.

141 went unexplained.

At first we read it this way: “the probe exited and the socket was cut, and the PostgreSQL backend writing the response died from SIGPIPE.” Plausible, but it doesn’t hold. PostgreSQL configures both the postmaster and backends to ignore SIGPIPE (pqsignal(SIGPIPE, SIG_IGN) in postmaster.c and postgres.c). When a client leaves, the backend simply terminates quietly and normally. And 141 was the container’s exit code to begin with, not a trace of one backend dying.

If the postmaster were PID 1, it could not end with 141. That makes it likely PID 1 was the entrypoint shell. But we couldn’t re-examine the failed environment, so we couldn’t confirm it.

The first half of this post is a record of promoting numbers to causes and getting it wrong. Building a mechanism straight from the arithmetic “141 = SIGPIPE” was the same mistake. So here we write only as far as “the container exited with 141.”

Why only with the large image.

The image that failed to pull was 16GB. The hypothesis “too big, so it’s a timeout” arises naturally. It was not a timeout. But the size was genuinely relevant, and the reason why is the final twist of this post.

06 — Why the contention happened: everything on one node

So far we’ve stopped at “the probe exceeded 1 second.” Why it exceeded is not yet answered. When we got the topology, half the answer was in the structure.

The customer environment is 1 worker node + 1 master node. There is no HA. And this product’s Helm chart pins every registry component, including the database, to the control plane with requiredDuringScheduling. With a single master, there is no choice.

Same platform, different topology.

Internal cluster, measured, shared across 3 control-plane nodes (12 cores each):

• 105 pods · 30.2 cores

• of which the monitoring stack alone is 20.2 cores, the largest consumer

• distributed across 3 nodes

Customer environment, single-master configuration:

• control plane + the entire registry + monitoring + backends

• the same components on one node (we could not measure this environment’s load)

• all on one node

• of which only the DB is BestEffort

• when contention arises, the DB with cpu.weight 1 is starved first

The difference is whether the same platform is distributed across 3 nodes or concentrated on 1. The numbers above are measured on the internal cluster, not values from the customer environment. We write them separately so as not to repeat the mistake of 2-③. On that node, the only workload with no resources declared was the database alone. Cause and victim connect like this.

And the pull creates its own contention.

Here we have to revisit the “NFS is not the cause” exclusion from before. The exclusion wasn’t wrong; the problem was excluding it without distinguishing the paths. Ruling out, one by one, the paths by which a health check could slow down due to storage:

It’s true that SELECT 1 uses no disk. But when WAL writes are delayed, the whole instance is delayed, and even a query that uses no disk doesn’t get its turn to execute. What we excluded at ② was the path “the health check directly waits on disk,” and that exclusion still holds. Only one path remains.

And the registry’s image store and the database’s volume use the same filer.

Worker requests a 16GB pull
  → registry (master node) reads 16GB from NFS and streams it
  → DB's WAL fsync competes over the same filer
  → CPU/I/O load on the master node rises
  → the BestEffort DB is pushed to the back → probe timeout → outage → pull gets 401
  → kubelet auto-retries → back to the start

If this is right, then “only happens with large images” is not probability but causation. The pull creates, by itself, the contention that stops it.

And this cycle repeats without human intervention: when an image pull fails during workload deployment, the kubelet retries automatically. That’s why it repeated 16 times like clockwork over an hour and a half.

What was delayed: two candidates, arithmetic points to one.

“The probe exceeded 1 second” is confirmed. What made it exceed remains two-way, and fortunately the two leave different traces.

A late topology report opened the second possibility. The master is not connected to the storage-dedicated network. It reaches the filer via NAT through the worker as a gateway, and that master’s NIC is 1G.

So during one 16GB pull, the master’s 1G link is occupied throughout, reading from the filer and sending back out to the worker. On the same link, the DB’s WAL writes also have to go to the filer.

16GB ÷ 1Gbps ≈ 2.1 min   (theoretical minimum, per direction)
Measured pull time        6–7 min
Crash interval            5.7 min

⚠️ At first we computed read-plus-send as 32GB ÷ 1Gbps ≈ 4.3 min, which looked close to the measured time and the crash interval. It was wrong arithmetic. Ethernet is full-duplex, so inbound and outbound do not add up. We had picked the calculation that made the numbers line up, the exact mistake this post warns about.

With the corrected number, the pull ran at about a third of line rate. Whether the link was saturated cannot be stated from this arithmetic.

But this section is still unmeasured. “The structure is like this” is confirmed; “so contention actually occurred” is inference. And the first half of this post is a record of promoting lined-up numbers to cause and being wrong three times. We can fall into the same trap here. Measuring node CPU, probe duration, and checkpoint sync= together while the pull is running would settle the table above in one shot.

Until then, the description should read not “there was contention” but “it was structured so that when contention arises, the DB is starved first.” What’s confirmed is enough on its own.

Adding workers doesn’t fix it.

The natural response is “add nodes.” In this structure, it has no effect.

Since every registry component is pinned to the control plane, the database stays on the master no matter how many workers you add. And as workers increase, all the pull traffic they take on passes through the master’s 1G, so the more you add, the worse the bottleneck.

One more risk worth noting: if the master’s NFS access goes through a specific worker, the moment that worker stops, the database loses its storage. Adding workers does not make this path redundant either.

07 — What we fixed

First, what we applied and where. In the failed environment, we raised the probe timeout to 20 seconds to stop the restarts. The corrections below (probe separation, resource requests) were reflected as defaults for new installs, and environments already in operation need a one-time database restart, so we apply those during scheduled maintenance windows.

First: why was it 1 second? The answer was in the chart.

Before fixing, there was something to check. Why did upstream set this value to 1 second? Did they assume an optimal environment? Checking the chart’s values.yaml, the answer was in the comment:

yaml

# The timeout used in livenessProbe; 1 to 5 seconds
livenessProbe:
  timeoutSeconds: 1
# The timeout used in readinessProbe; 1 to 5 seconds
readinessProbe:
  timeoutSeconds: 1

They spelled out a range of “1 to 5 seconds” and set the lower bound as the default. And 1 second is also Kubernetes’ default for timeoutSeconds.

In other words, 1 second was not chosen as an optimum, it is the value you get when you don’t change it.

There’s something more decisive. In the entire chart, the only component that exposes the probe timeout is the database. No other component has this setting at all.

Nobody leaves a setting adjustable that no one has trouble with. Upstream also knew this value falls short depending on the environment, and left it adjustable. We simply didn’t use that setting.

One more thing turned up in the same file:

yaml

# resources:
#  requests:
#    memory: 256Mi
#    cpu: 100m
priorityClassName:

A resource-request example, commented out. Earlier we said “we used the upstream default as-is,” but more precisely, we failed to enable what the chart recommended.

How to choose the value: look at the asymmetry.

The probe timeout is a trade-off between two risks. But the loss leans heavily to one side.

Detecting late only fails requests; killing wrongly corrupts state. The asymmetry is clear, so setting it long is the right call. So for a query-based probe we chose 5 seconds, the upper bound of the documented range.

And there’s a better approach still: not making the trade-off at all.

We separated the roles of liveness and readiness.

yaml

# "Is the process running?" — kubelet checks directly, no container CPU needed
livenessProbe:
  tcpSocket: { port: 5432 }
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 3        # omit and it defaults to 1s, the exact value this post flags
  failureThreshold: 6      # the port must be closed for 60 consecutive seconds to kill

# "Is it answering queries?" — on failure, only traffic is excluded
readinessProbe:
  exec: { command: ["/docker-healthcheck.sh"] }
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

⚠️ timeoutSeconds defaults to 1 if omitted. tcpSocket is usually fine even at 1s because the kernel accepts the connection, but we set it explicitly rather than relying on the default.

Liveness should only check “is the process running?” Slow is not stopped.

The port is open even during recovery, so tcpSocket passes and the DB can recover to the end. Blocking traffic is readiness’ job. This separation breaks the earlier cycle.

Incidentally, an exec probe spawns a new process inside the container (fork + exec + TCP connect + auth). tcpSocket is checked by the kubelet in its own process, so it’s independent of the container’s CPU situation.

This is not a trap we alone fell into. Kubernetes itself hit the same thing with etcd: using etcdctl get foo as liveness, the kubelet restarted a healthy etcd, which meant the whole control plane became unstable. kubeadm eventually removed exec and switched to HTTP /health (kubeadm#1708). The cost of the exec probe itself also remains an upstream issue: a frequently run probe creates the very timeout it was meant to detect.

But only one item was adjustable.

The YAML above is “how it should be,” not values you can apply as-is. The only probe setting the chart exposes as values is timeoutSeconds. The type (exec/tcpSocket), initialDelaySeconds, periodSeconds, and failureThreshold are hardcoded in the template. In the previous section we said “upstream left it adjustable,” but that doesn’t apply to the very items that most needed changing.

So we chose to replace the probe with kubectl patch after install. There’s a trap here.

json

{ "livenessProbe": { "exec": null, "tcpSocket": { "port": 5432 } } }

Omit "exec": null and the two handlers coexist, which the API server rejects. A strategic merge patch deletes only what you explicitly mark for deletion. And adding --dry-run=server lets you run just this validation without a rollout, worth running once before any downtime work.

We gave it resource requests.

This DB had resources: {}, so its QoS was BestEffort. In the failed environment we couldn’t read the weights directly, so on the internal cluster running the same chart and the same settings, we read the cgroup weights alongside the other workloads on the same node:

cgroup cpu.weight, measured (higher is better in CPU contention):

100 = cgroup v2 default. The single point of failure for the entire system had the lowest CPU allocation priority, one-hundredth of a 1-core request.

What you declare becomes, directly, your share in CPU contention. Every other component declared requests; only the DB, left at the upstream chart default, was missing it.

Note the 100 here. In cgroup v2, the default weight of a process with nothing set is 100. The node’s systemd, kubelet, and kernel tasks all have that value. A pod requesting 1 core maps to exactly that value, and the database with empty requests is one-hundredth of it.

“Low priority” doesn’t cover it. It’s starved by a two-digit factor.

⚠️ This number depends on the container runtime version. Kubernetes converts cgroup v1’s cpu.shares to v2’s cpu.weight, and that conversion formula changed in January 2026. Under the old linear formula, the same workloads got 250m→10 and 1 core→39, meaning even a 1-core request got less than 40% of the default 100, which was why they changed the formula (kubernetes#131216). The values above are read under the new formula (runc 1.3.2+).

Whichever formula, the one thing that doesn’t change is that no requests means 1. Only the size of the gap changes.

Two traps, noted here:

• We did not set a CPU limit. CFS quota throttling recreates the very delay we’re trying to prevent. Guarantee the floor, set no ceiling.

• We did not set the memory limit equal to the request. We left one notch of headroom: request 2Gi, limit 3Gi. Setting them equal reproduces the same problem by a different path, via OOMKill. The basis is actual usage: internal cluster measured 678Mi, 167 active connections. For reference, max_connections 1024 × work_mem 4MB ≈ 4GB is not the worst case, because work_mem is per sort/hash operation, not per connection, so the theoretical ceiling is higher. In an environment where connection count approaches the configured ceiling, the limit should be recomputed.

08 — It was not “not enough hardware”

When we raised the probe timeout to 20 seconds in the failed environment, the restarts stopped. And a natural summary came out: “so in the end it was caused by not enough compute power.”

What disproved that conclusion was the mitigation result itself.

The resources and the node configuration were unchanged. If compute power had truly been short, the DB would still be too slow to use even after raising the timeout. But it works normally. That is, the DB was doing its job from the start. What couldn’t cope was not the DB, but the 1-second probe.

SELECT 1 taking 1.2 seconds does not mean the DB stopped, yet it was judged as stopped and killed. This distinction is not a matter of wording.

The normal response to a momentary 1-second delay is “briefly slow.” The actual response was forced container kill → unclean shutdown → crash recovery → simultaneous database outage, and at the end it even logged this:

LOG:  database system was interrupted while in recovery
HINT: ... some data is corrupted and you will have to use the last backup

When shut down again in the middle of recovery, PostgreSQL advises restoring from backup. That is far too severe a state for a single 1-second timeout to reach. The response is grossly out of proportion to the cause. It is not a capacity problem, it is a design vulnerability.

Attribute the cause to “the environment,” and the same default gets deployed to the next environment unchanged.

In fact, our internal cluster’s config was not one character different. The reason it had no trouble was not that it was safe, but that its load was low: health check measured at 55ms, 18x headroom against 1 second. One high-load day and the same thing happens.

09 — What remains

On diagnosis

• Intermittent success is evidence in itself. It excludes any hypothesis that should fail deterministically.

• An HTTP status code is a bin. A 401 means “something went wrong at the auth step,” not “the credentials are wrong.”

• Periodic failure yields its cause to arithmetic. If it matches initialDelay + failureThreshold × period, it’s the probe.

• Exclude hypotheses by exit code. If it’s not 137, it’s not OOM.

• Lined-up numbers are not causation. Measure directly at that layer, and compare side by side against a healthy baseline.

• The arithmetic of an exit code is not an explanation. 141 = 128 + 13 is just a calculation. Whether that signal actually kills that process has to be confirmed in the source. PostgreSQL ignores SIGPIPE.

• Check that the window you measured is the right one. Measure during a quiet period and declare “normal,” and you miss the entire state of the moment under load.

On design

• Liveness should only check “is it running?” Check “is it fast?” and you kill the slow.

• Do not put a 1-second timeout probe on a DB. The probe stops detecting outages and starts creating them.

• Put an exec probe on a component that refuses during recovery and you create a loop. Choose a signal that does not obstruct recovery.

• Do not leave resources empty on a stateful workload. BestEffort is the lowest cgroup weight.

• Check whether a default is “a chosen value” or “an unchanged value.” If the upstream comment states a range and you’re using the lower bound, that’s not a choice, it’s unset.

• Choose thresholds by asymmetry. When the failure costs in the two directions differ, lean toward the lower-cost side.

• If one instance is shared by many, you have to confirm every way that single instance can go down.

• What the chart leaves adjustable and what you need to adjust are different things. If there’s no adjustable item, a post-install patch is the fix, and that patch has to be treated as part of configuration management.

• Adding nodes may not be the answer. If scheduling is pinned, the added nodes only send more load while the bottleneck stays. Measure where the bottleneck is before scaling.

Further reading

• Liveness Probes are Dangerous — Henning Jacobs (Zalando). The standard reference on this topic. “The combination of a Liveness Probe and an external DB health check is the worst, one transient DB failure restarts all containers.” Our incident is one layer deeper: the DB is killed by its own probe, and killed again while trying to recover.

• kubeadm#1708 — stop using etcdctl as a liveness probe / kubernetes#82440 — the system load of exec probes

• kubernetes#131216 — the new cgroup v1→v2 CPU conversion formula, the source of the weight numbers above

• Pod Quality of Service Classes — the definition of BestEffort and the eviction order

• Kubernetes Patterns (Ibryam & Huß), Predictable Demands — “In an environment where many processes of differing priority share resources, the only way to coexist is to know each process’s demand in advance.”

Customer, hostnames, and ticket numbers are anonymized. Logs and figures are real values.

Confirmed during diagnosis but not included in the body: token response size (825–848 bytes, independent of image name length), 47-second transfer via ingress completing normally, no robot-account expiry, all used to erase hypotheses.