# Monitoring Outgoing Webhooks: Failure Rates, Backlogs, and Alerts

> What to measure when you send webhooks - time failing per endpoint, retry backlog, latency - when to alert, whom to tell, and how to warn before auto-disabling.

Published: 2026-11-03
Source: https://webhookadmin.com/en/blog/webhook-monitoring-alerts/

## Key points

- Two numbers matter most when you send webhooks: how long each endpoint has been failing, and how many deliveries are waiting to be retried.
- A global failure rate hides a single broken endpoint, so measure failures per endpoint, and look at latency as a distribution rather than an average.
- Alert both your own team and the endpoint's owner, and group repeated alerts for the same endpoint.

When you send webhooks, two numbers tell you most of what you need: how long each endpoint has been failing, and how many deliveries are waiting to be retried. The first tells you a particular customer's receiver is broken. The second tells you your own sending pipeline is backing up. This post covers what to measure, when to alert and whom, and the order in which to warn people before you disable an endpoint.

## What to measure

| Metric | Scope | What it tells you |
|---|---|---|
| Time failing | Per endpoint | Whether the receiver is broken; counted from the first failure |
| Retry backlog | Whole environment | Whether your pipeline is stuck or many endpoints are failing at once |
| Failure rate | Per endpoint and overall | Trends; the overall number is for day-over-day comparison |
| Latency distribution | Per endpoint | Slow endpoints tying up workers; endpoints near the timeout fail next |
| Disabled endpoints | Whole environment | Whether auto-disabling is increasing |
| Deliveries sent | Whole environment | Whether events stopped flowing (a sudden drop to zero is an incident too) |

### Why not the average failure rate

Say you send 100,000 messages a day to 500 endpoints. One endpoint can be down all day, and if it receives 0.2% of the traffic, your overall failure rate moves by 0.2%. Meanwhile an endpoint that receives one message a day shows a 100% failure rate after a single blip. The overall rate is useful for trends; to spot a broken receiver, track how long each endpoint has been failing.

### Tracking how long an endpoint has been failing

Add one column to your endpoints table: when the current run of failures started. Clear it on success, set it on failure if it's empty.

```sql
-- update after every delivery attempt
UPDATE endpoints
SET failing_since = CASE
  WHEN :success THEN NULL
  ELSE COALESCE(failing_since, :now)
END
WHERE id = :endpoint_id;

-- endpoints failing for over an hour
SELECT id, url, failing_since
FROM endpoints
WHERE failing_since IS NOT NULL
  AND failing_since < :now - INTERVAL '1 hour'
  AND status = 'active';
```

The same column drives both "alert after an hour" and "disable after five days."

### Retry backlog

The backlog is the number of deliveries that failed and are waiting for their next attempt. It jumps when many endpoints fail together, and also when your own sending breaks (DNS, egress network), so put a threshold on the environment-wide number separately from per-endpoint alerts.

## When to alert

Alert on stages, not attempts. Alerting on every failed attempt buries the alerts that matter.

1. **Failing for a sustained period** (say, one hour). Filters out blips; what's left needs fixing.
2. **Retries exhausted.** That event won't arrive on its own anymore; someone has to decide whether to replay it.
3. **Endpoint disabled.** New events stop or are held; someone has to re-enable it.
4. **Backlog over a threshold** (say, 1,000). A candidate for an incident in your own pipeline.

Group alerts per endpoint, for example at most one an hour, so a long outage doesn't produce an alert per retry.

## Whom to alert

| Recipient | What | Channel |
|---|---|---|
| Your operations team | Every stage, backlog growth | Slack, Teams, email, your monitoring system |
| The endpoint's owner (your customer) | Failures and disabling of their own endpoints | Email, the customer portal |

Most failing endpoints are broken on the receiving side: an outage, an expired certificate, a URL that changed. The customer is the one who can fix it. If only your team hears about it, someone on your team has to contact the customer every time.

## How other providers do it

Checked against official documentation on 2026-09-27:

| Provider | Notification | Stopping |
|---|---|---|
| Stripe | Emails you when webhooks are failing (live mode) | Retries each event for up to 3 days |
| Shopify | Sends warning emails to the app's emergency developer email | Deletes Admin API subscriptions after 8 consecutive failures |
| Slack | Not stated | Disables event subscriptions when over 95% of attempts in 60 minutes fail |
| Contentstack | One email to the owner, not one per failure | Disables the webhook after 10 consecutive failures with no success in between |

Sources: [Stripe](https://docs.stripe.com/billing/subscriptions/webhooks), [Shopify](https://shopify.dev/docs/apps/build/webhooks/subscribe/https), [Slack](https://docs.slack.dev/apis/events-api/), [Contentstack](https://www.contentstack.com/docs/developers/set-up-webhooks/webhook-circuit-breaker)

Contentstack's single email on disable is one answer to alert noise. The trade-off is that without an earlier warning, the customer first hears about it after deliveries have already stopped.

## Warn before you disable

Auto-disabling protects your workers from endpoints that will never answer, but from that moment the customer receives nothing. Give them a chance to fix it first:

1. After an hour of failures, notify your team and the customer.
2. When an event exhausts its retries, notify again and mention that it can be replayed.
3. When you disable the endpoint, notify and explain how to re-enable it (dashboard or API).
4. Decide in advance what happens to events from the disabled period: send them on re-enable, or replay by hand.

Make the time to disable longer than your retry window. Disabling mid-window throws away events that could still have been delivered.

## What to put in the alert

Enough for someone to decide without opening a dashboard:

- The environment or project and the endpoint URL
- When the failures started
- The latest status code and the start of the response (for example `503 Service Unavailable`)
- A link to the delivery log

Response bodies can contain the receiver's error messages; [what to log for outgoing webhooks](https://webhookadmin.com/en/blog/webhook-delivery-logs/) covers what to keep.

## What Webhook Admin shows and sends

The project overview in Webhook Admin shows deliveries, success rate, failures and average duration over the last 24 hours, the retry backlog, the number of disabled endpoints, and daily successes and failures for the past 7 days. Alerts follow the stages above: an endpoint failing for an hour or a message out of retries (at most once an hour per endpoint), an endpoint auto-disabled after five days, more than 1,000 retries waiting in an environment, and usage thresholds. They go to Slack, Microsoft Teams, email, a signed webhook or Chatwork, with the endpoint URL and a dashboard link.

Channel setup, defaults and test alerts are in [the alerts setup guide](https://webhookadmin.com/en/blog/webhook-failure-alerts-setup/), and the retry schedule in [designing webhook retries](https://webhookadmin.com/en/blog/webhook-retry-design/). Log retention by plan is on the [pricing page](https://webhookadmin.com/en/#pricing).

If you'd like delivery monitoring and alerts without building them, you can try Webhook Admin for free (50,000 messages a month): [https://app.webhookadmin.com/signup](https://app.webhookadmin.com/signup)

## FAQ

### At what failure rate should I alert?

Rather than a threshold on the overall failure rate, alert per endpoint on duration, for example when an endpoint has been failing for an hour. One dead endpoint barely moves the global rate, while a single failure on a low-volume endpoint looks like 100%.

### Should I notify my customers too?

Yes. Most failing endpoints are broken on the receiver's side - an outage, an expired certificate, a changed URL - and only the customer can fix them. Stripe and Shopify both email the receiving developer when deliveries keep failing.

### We get too many alerts. What should we change?

Group alerts for the same endpoint (at most one an hour, say) and alert on stages - failing for a while, retries exhausted, endpoint disabled - instead of on every failed attempt. Transient failures then stop drowning out real ones.