Build a Webhook Receiver That Verifies and Recovers

With a normal API call, your app asks another app for something. A webhook flips that around. The other app calls you when something happens. A new support ticket gets created, the help desk sends your service a message, and your triage workflow starts without anyone checking every few minutes.

Here's the catch. A webhook tells you something happened. It doesn't promise your work ran exactly once.

The sender might send the same event twice because it never saw your reply. Your receiver might say "got it" and then crash before doing anything. This lesson builds those cases in from the start, before any model is involved.

Download the Python receiver and delivery fixtures. It starts a temporary web server on your machine, sends it eight deliveries, and then checks a small SQLite database after a deliberately broken worker run. You don't need a model, an account, a public URL, or a tunnel.

1. Run the complete example

You need Python 3.10 or newer. Save the file in a new folder, then run:

python3 webhook_lesson.py

If python is your Python 3 command, use that instead. There's nothing to install. The script only listens on 127.0.0.1, which means only your own machine can reach it. It picks a free port, creates a throwaway database, and deletes the database and stops the server when the checks finish.

The recorded September 17, 2026 run passed all eight HTTP cases and left one finished work record. It also confirmed two recovery behaviors. When the script raised an error partway through a database write, the half-finished write was undone. And a fresh database connection could then pick up the waiting event and finish it.

That test raises an error and reopens the database. It doesn't kill the process or pull the power, so don't read more into it than that.

2. Look at how the local sender signs each request

This lesson uses its own made-up format, so you can see every piece. The two headers are X-Lab-Timestamp and X-Lab-Signature. It isn't GitHub's, Stripe's, or any other provider's webhook format.

The sender posts these exact bytes to /webhook:

{"event_id":"evt-demo-1","type":"ticket.created","ticket_id":"DEMO-42"}

The timestamp is Unix time in seconds. The signature is a fingerprint made from three things joined together: the timestamp as text, a period, and the exact body bytes. Only someone holding the shared secret can produce it.

def signature(secret: bytes, timestamp: str, body: bytes) -> str:
    return "sha256=" + hmac.new(
        secret,
        timestamp.encode("ascii") + b"." + body,
        sha256,
    ).hexdigest()

Each test run makes a new random secret that only the local sender and receiver know. When a request arrives, the receiver does two checks before anything else.

First, it rejects any timestamp more than 300 seconds away from its own clock. That stops someone from replaying an old request they captured.

Second, it recomputes the signature and compares it with hmac.compare_digest. Python's HMAC reference documents that function for checking signatures.

Only after both checks pass does the receiver read the JSON. It accepts exactly the three fields shown above, the event type ticket.created, an event ID that isn't empty and is at most 80 characters, and the made-up ticket DEMO-42. It also refuses any body over 16 KiB. These rules are strict so the example is easy to inspect. They aren't a real support system's format.

One detail trips people up. The signature covers the exact bytes, so even a change in spacing breaks it. One fixture adds a single space to a valid body and keeps the old signature. The receiver returns 401 invalid_signature, even though the JSON means the same thing.

3. Save the event before saying "got it"

When a valid event arrives, the receiver writes it down before it replies.

It opens a SQLite transaction with BEGIN IMMEDIATE, which locks the database so two deliveries can't write at the same moment. It looks up the event_id in an inbox table. That ID is the table's primary key, so the database itself refuses a second row with the same ID. The receiver saves a hash of the original body and a pending state, commits, and only then returns:

{
  "event_id": "evt-demo-1",
  "state": "pending",
  "duplicate": false
}

The HTTP status is 202, which means "stored, will process later." It doesn't mean a model has looked at the ticket or that anything has changed yet.

If the exact same delivery arrives again, the receiver returns 200 with the existing state and duplicate: true. Nothing new is written.

If the same event ID arrives with different bytes, the receiver returns 409 event_id_conflict. That one deserves a look by a person. Quietly accepting a different payload under an ID you've already stored would make it impossible to tell which version was real.

In this format, "the same" means identical byte for byte. A real provider might resend the same data with different spacing or field order. When you adapt this receiver, decide on purpose what counts as a repeat and what counts as a conflict.

4. Follow a worker that crashes halfway

Now the worker. It picks one pending event and writes a ready_for_triage result into a second table. Then it marks the inbox row completed. Both of those changes happen inside one SQLite transaction, which means they either both happen or neither does:

db.execute("INSERT INTO work VALUES (?, 'ready_for_triage')", row)
if fail_before_commit:
    raise RuntimeError("injected failure before commit")
db.execute("UPDATE inbox SET state='completed' WHERE event_id=?", row)

The fixture raises an error right between those two lines. Here's what happens next:

1. The transaction is undone, so the first write disappears too.
2. A newly opened connection sees zero work rows and a receipt that's still pending.
3. Running the worker again, this time without the error, writes one result and marks the receipt completed.
4. A third run finds nothing pending and does nothing.

This only works because both changes live in the same local database. It doesn't protect a call to an outside API.

Picture your worker telling a CRM to create a contact. The CRM creates it, then your connection drops before you hear back. You can't tell whether it worked. Use that system's own duplicate protection, often called an idempotency key, or its "check the status of this request" feature. If it has neither, set the result aside for a person to reconcile. A retry is not proof the first attempt failed.

5. Read the results table

These are the eight recorded HTTP outcomes. The worker crash-and-recover check happens between the seventh and eighth deliveries.

The output ends with:

{
  "http_passed": 8,
  "reopened_after_rollback": true,
  "work_rows": 1
}

Nothing here measures AI accuracy. The one work row is just a local marker that says "this event is ready for triage." It isn't a prediction, and nothing was sent to a customer.

6. Change two fixtures, then adapt it to a real provider

Open the fixtures() function and find the send("stale-delivery", ...) call. Change its timestamp from 301 seconds old to 30 seconds old, and its expected status from 401 to 200.

The send() helper re-signs the request for you using the new timestamp and body. If you edit a saved HTTP request by hand instead, you have to re-sign it yourself.

Run the script again. The request now passes the freshness check, but it carries the same event ID and body as the first accepted event. So the right result is "duplicate, still pending," not a second work record. That's the point of the exercise: the freshness check and the duplicate check solve different problems, and you need both.

Next, find send("signed-invalid-json", ...) and replace raw=b"not json" with valid JSON that has one extra field. The helper signs the new body automatically. Keep its expected status at 400. A valid signature only proves the request came from someone with the secret and wasn't changed on the way. Your app still decides which events it accepts.

For a real sender, swap out this lesson's headers and signature code for that provider's documented check. For example, GitHub signs the payload body and sends X-Hub-Signature-256. Don't bolt this lesson's timestamp format onto a GitHub check and expect the signatures to match. Look up separately how that provider identifies a delivery, how often it retries, and how long it waits for your reply.

Python's built-in development server is fine for this exercise on your own machine. A deployed receiver needs a proper web server, HTTPS, request limits, storage someone monitors, and a plan for restarting the worker. Keep the receiving step quick, and do slow work like classification after the event is safely stored.

When you're done, you should have the eight-case output, your two changed fixtures, and a short trace that shows an event being accepted, the worker crashing and rolling back, and the retry succeeding. Then go to the prompt-contract lesson to define what the model should do once an event has made it this far.

Related Guides

• Use GitHub Copilot to Fix a Stale-Delete Bug You Can Verify
• Evaluate a Ticket Agent: Dataset, Scoring Code and Failure Report
• Plan and Execute: Build a Bounded Ticket Workflow