Webhooks, Retries, Duplication, Signatures, and Event Order

The problem

Many teams treat a webhook as if it were just a normal endpoint with reversed initiative.

Something like:

“The provider will POST here and we process it.”

That explanation is too short to be safe.

Because real webhooks usually bring four pains together:

• automatic retry
• duplication
• uncertain order
• the need to validate that the message really came from who it claims to come from

If you ignore those four things, the integration looks simple right up until the first incident.

Mental model

A webhook is an external, asynchronous notification from a system you do not control.

That changes the design.

Instead of thinking “one request to my backend,” think:

“I am receiving an event from outside, possibly repeated, possibly delayed, and I need to decide whether I trust it and how to turn it into safe internal state.”

That mental model removes a lot of naive assumptions early.

Breaking the problem down

Retries are normal

The provider retries when:

• it got a timeout
• your application returned 5xx
• acknowledgement was not clear

That means the same webhook may arrive more than once without any bug on the provider side.

Duplication needs deduplication

If the provider sends the same event_id twice, your system cannot:

• create two charges
• grant access twice
• send the same email ten times

The most common fix is storing an event identifier that was already processed.

Signature is not a detail

Webhook traffic comes from outside.

So you need to validate that the message really came from the expected provider and was not tampered with.

That is exactly what the signature helps with.

It does not replace:

• internal authorization
• idempotency
• ordering validation

But without signature validation, you risk accepting an invented event from any origin.

Event order is not always guaranteed

This is where many people get caught.

The provider may emit:

1. payment_succeeded
2. refund_created

But delivery may arrive reversed in your system.

If your logic depends on “I always receive things in order,” you built on a fragile premise.

Simple example

Imagine a payment provider sending webhooks to your system.

A more robust flow looks like this:

1. receive the request
2. validate the signature
3. extract event_id
4. record that the event arrived
5. respond quickly
6. process the business effect asynchronously and idempotently

Now, if the same event comes again, the system recognizes it has already been seen.

If processing fails, you can reprocess it without depending on the provider to insist again.

If order arrives in a strange way, your internal model needs to reconcile state without assuming perfect delivery linearity.

Common mistakes

• Doing heavy processing before replying to the webhook.
• Trusting the request body without validating the signature.
• Assuming the event only arrives once.
• Assuming implicit order without reading the provider’s real guarantee.
• Using retry without idempotency and multiplying side effects.

How a senior thinks about it

Someone with more experience treats a webhook as distributed integration, not as a form submitted by another system.

The reasoning usually sounds like this:

“I need to authenticate the source, accept repetition, survive delay, and decouple receipt from processing.”

That posture looks more expensive at first, but it prevents weak integrations and annoying incidents later.

What the interviewer wants to see

In interviews, webhooks reveal whether you understand real-world integration.

The evaluator wants to see whether you:

• talk about signatures
• talk about duplication
• do not assume order without a guarantee
• suggest fast acknowledgement and later processing

A strong answer often sounds like this:

“I would treat the webhook as an external asynchronous event. I would validate the signature, deduplicate by event_id, reply quickly, and process idempotently without assuming perfect order.”

A robust webhook does not depend on network goodwill or on the fantasy of perfectly ordered events.