Slack webhook failed at 2am? Nothing retried it

You did the sensible thing. Somewhere in your app there is a function called notifySlack, and inside it is a fetch to an incoming-webhook URL. A payment lands, a message appears in #payments, everyone feels good. It took ten minutes to build and it has worked for months.

This post is not about that being wrong. It is about the night it stops working and nobody finds out.

The message that was never sent

Tuesday, 2:07 AM. A customer's card is declined. Your webhook handler catches the error, and in the catch block it calls notifySlack("Payment failed for …"). That call makes an HTTP request to Slack.

Slack answers 429. Or your host has a thirty-second DNS hiccup. Or the request just times out, because it is 2am and something upstream is being restarted. The fetch rejects. Your catch block was not written to catch its own notification failing, so either the exception escapes and takes the rest of the handler with it, or it is swallowed by a second catch you added after the first time this happened.

Either way, the message is gone. There is no queue, no retry, no record that anything was ever attempted. In the morning, #payments looks like a quiet night.

What a pasted URL cannot do

An incoming webhook is a fire-and-forget HTTP call. That is the whole feature, and it is why it is so easy to set up. It is also why it has none of the properties you would want from anything that carries the words payment failed:

• No retries. One request, one chance. A 429, a 5xx or a dropped connection is the end of the story.
• No history. There is nowhere to look to answer "did that alert go out?" The only record is the Slack channel, and a missing message looks exactly like a quiet evening.
• No fan-out. Want the same failure in Discord for the on-call contractor, and on your phone if it is critical? That is three URLs, three requests, three ways to fail.
• No filtering. Everything the code sends, the channel gets. The only way to stop a noisy event is another deploy.
• No protection from floods. When the payment provider has an outage, you do not get one alert. You get one per attempt, until someone mutes the channel, which is how the real alert gets missed an hour later.
• No quiet hours. The signup at 3am buzzes exactly as loudly as the outage at 3am.

None of that is a criticism of Slack. A webhook URL is a mailbox, not a mail carrier.

Put a router between your app and the channel

EventSend is the mail carrier. Your app makes one HTTP POST, and everything above happens on the other side of it.

curl -X POST https://eventsend.io/api/events \
  -H "Authorization: Bearer $EVENTSEND_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "event": "payment-failed",
    "message": "Payment failed: card declined for [email protected]",
    "level": "error",
    "payload": {"customer": "cus_123", "reason": "card_declined"},
    "unique_key": "evt_1Q2w3e"
  }'

A 202 means the event is queued. From there:

• Retries, per destination. Each destination gets up to five attempts on a fixed schedule: 30 seconds, then 2 minutes, then 4.5, then 8. That is about fifteen minutes of trying. A 429 waits exactly what the Retry-After header says. A destination that is being rate limited or is mid-outage gets the message late rather than never.
• A delivery log, per attempt. Every attempt to every destination is recorded with its outcome, and the exact error when it failed. "Did that alert go out?" has an answer.
• Fan-out from one request. A topic connects the route your app sends to with any number of destinations: Slack, Discord, Telegram, your own signed webhook, or a personal push destination on your phone. The app sends once.
• Filtering in the dashboard, not in a deploy. A topic's rules combine a minimum level with an event-name pattern. "Everything to #payments, only error and above to Telegram, only payment-* to the finance webhook" is three topics, no code.
• Rollups. A destination can deliver the first few repeats of an event and collapse the rest into one summary when the burst ends. Two hundred payment-failed events during a provider outage become three messages and a line that says ×200.
• Threshold alerts. Or turn the burst itself into the signal: "at least 20 payment-* events at error or higher within 2 minutes" fires one critical alert, to any destination you choose.
• Quiet hours. A destination can hold everything except critical overnight, in your organization's time zone, and summarise what it held in the morning.

And because a notification is a claim about now, a delivery that is more than 24 hours old is skipped as stale rather than sent, and the log says so. A Slack critical expires after fifteen minutes. An alert about a problem that is no longer happening is a second problem.

The 2am replay

Same Tuesday, same declined card, same 429 from Slack. This time the catch block posted one event to EventSend and got its 202 in a few milliseconds. The Slack attempt failed, the log says 429, and the next attempt thirty seconds later succeeded. The message is in #payments, timestamped 2:08 AM. Because the event was error, quiet hours held it off your phone until seven. Because the topic also feeds the finance webhook, a signed copy is sitting in your own system, verified with the signature EventSend put on it.

You still wrote a notifySlack function. It just stopped being the thing that decides whether anyone hears about the payment.

It is still one HTTP call

If you already have an incoming webhook wired in, moving is not a project. The request above is the whole API. Swap the URL and the body, keep the token in an environment variable, and delete the retry loop you were going to write one day. The free plan includes 2,500 events a month, three team destinations and one source, with no card and no trial clock.

The one thing to change in your head is smaller than the code: the channel is where the message ends up. It should never have been the thing that decides whether it arrives.