How to Set Up Alerting That Doesn't Wake You for Nothing

Uptime monitoring alert fatigue is what happens when your monitoring tells you about so many non-problems that you stop reading the messages, and the fix is to make every alert pass through a confirmation step before it reaches a human. The goal is simple: an alert should only fire when something is genuinely broken, the breakage is actionable by whoever is on call, and nobody has already acknowledged it.

Most noisy setups fail at the first gate. A single failed HTTP check is not an outage. It is a dropped packet, a slow TLS handshake, a load balancer briefly rotating a backend, or a check running from a probe that lost its own network for two seconds. The monitoring tool cannot tell the difference between "your site is down" and "my probe is having a bad time" unless you make it prove otherwise.

Confirm before you notify

The single most effective change is requiring consecutive failures from more than one vantage point. Every serious monitoring service exposes this as a retry or confirmation setting, though the names differ. Look for settings called retries, confirmation attempts, or failure threshold, and set them so that a check must fail several times in a row before the incident opens.

If your tool lets you run checks from multiple regions, require failures from at least two of them. A probe in one datacenter losing connectivity to your origin is a probe problem until a second region agrees. This one rule removes the majority of false positives from most accounts, because most false positives are network blips on the probe side.

You can reproduce the reasoning yourself with a trivial loop. Run this from two machines and compare:

for i in 1 2 3 4 5; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}\n" \
    --max-time 10 https://example.com/health
  sleep 5
done

The output is one line per attempt, an HTTP status and the total time in seconds. If you see a single 000 or a lone 503 surrounded by 200 lines, that is exactly the kind of transient your monitoring should swallow rather than page on. If every line is 000, you have a real outage and the alert is earned.

Alert on symptoms, not on every metric

A common source of uptime monitoring alert fatigue is alerting on causes instead of symptoms. CPU above some level, memory climbing, a disk filling, a queue depth rising: these are interesting on a dashboard and mostly useless as pages. What matters to a user is whether the service answers requests correctly and quickly. Alert on the error rate and the latency of the actual user-facing endpoint, and let the resource metrics inform your investigation after the fact.

Pick one or two health endpoints per service and make them meaningful. A health check that returns 200 as long as the process is running tells you almost nothing. A health check that touches the database, the cache, and any critical dependency, and returns a non-200 when any of them is unavailable, tells you what your users are experiencing. Keep it cheap enough to run often, and make sure it does not itself become a load problem.

Watch the response headers when you build this. A useful health endpoint returns something you can assert on, not just a status code:

HTTP/1.1 200 OK
Content-Type: application/json
Cache-Control: no-store
X-Health-Db: ok
X-Health-Cache: ok

{"status":"ok","db_ms":3,"cache_ms":1}

If your monitoring tool can assert on a header or a JSON field, use that instead of matching on the status code alone. A 200 with X-Health-Db: fail is a degraded service, and you want to know before users tell you.

Route, deduplicate and escalate deliberately

Once an alert is real, the next question is who hears it and how often. Configure your tool so that a single incident produces a single notification, with reminders on a schedule you choose, rather than a new message on every failed check. Most tools call this grouping, deduplication, or incident mode. Turn it on. An incident that pages every minute for an hour trains people to mute the channel, which is the exact behaviour you are trying to avoid.

Separate severity into at least two tiers and route them differently. A hard down on a user-facing endpoint goes to the on-call rotation through a channel that actually rings. A degraded dependency, a certificate nearing expiry, or a single failed check from one region goes to a chat channel or a ticket queue that someone reads during working hours. If everything is urgent, nothing is.

Escalation should be time-based and explicit. Define that if the primary on-call has not acknowledged within a set window, the alert moves to the secondary, and then to a manager or a wider channel. The window is a judgement call that depends on your team size and response expectations, so pick it deliberately and write it down rather than leaving the default in place.

Finally, review the alerts themselves. Keep a short log of every page and what happened, and once a month look at which alerts fired and whether each one needed a human. Any alert that fired and required no action is a candidate for deletion, a higher threshold, or a move to a lower severity. This maintenance is the part people skip, and it is the part that keeps the system honest.

What to do next

Start by turning on confirmation and multi-region checks for your most important endpoints, since that single change removes most noise. Then audit your existing alerts one at a time and ask whether each would have woken someone for a real, actionable problem. Delete or downgrade the ones that would not, and set a recurring reminder to repeat that audit, because alerting drifts out of shape the moment nobody is watching it.