navree
Documentation

Webhooks

Send HMAC-signed webhooks from your events, with queue-backed retries, a delivery log and endpoint management, and verify incoming webhooks from Stripe, GitHub, Paddle, AWS SNS, Google Pub/Sub, Microsoft Graph, Twilio, Slack, Mailgun, Postmark and any Standard Webhooks sender.

On this page

Introduction

Featurewebhooks Webhooks are how your app tells other systems that something happened, and how other systems tell you. navree-webhooks handles both directions.

Outgoing. Your customers register endpoint URLs and pick the events they want. When a deal is won, every subscribed endpoint receives a signed POST within seconds. Delivery runs on the queue, with retries for about a day, and every attempt lands in a delivery log your customers can read.

Incoming. The VerifiedWebhook extractor checks that a request comes from who it claims before your handler sees it. It knows Stripe, GitHub, Paddle, AWS SNS (SES events among them), Google Pub/Sub push (Gmail notifications among them), Microsoft Graph change notifications, Twilio, Slack, Mailgun, Postmark and any sender that follows the Standard Webhooks specification, and it answers their handshakes (Graph's validationToken, Slack's challenge, SNS subscription confirmations) for you.

Outgoing messages follow Standard Webhooks too, so your customers can verify them with any Standard Webhooks library, in any language.

Installation

The webhooks feature of the navree facade is part of full, which is on by default. With default-features = false, turn it on:

Cargo.toml

Add WebhooksModule to your app root. Outgoing delivery needs the database, the queue, the events bus and the HTTP client. Verifying incoming webhooks needs none of them; AWS SNS and Google Pub/Sub fetch their senders' public keys with the app's HTTP client, or a default one when the app has no HttpClientModule:

src/app.rs

The module contributes the migration set webhooks, which creates webhook_endpoints, webhook_deliveries, webhook_attempts and webhook_receipts. The delivery log is time series: on TimescaleDB, webhook_deliveries and webhook_attempts are hypertables on created_at, a day per chunk. A hypertable's key includes its time column, so attempts reference their endpoint rather than their delivery: deleting an endpoint deletes its deliveries and attempts, and webhooks:prune deletes a delivery's attempts with it.

Endpoint secrets are encrypted at rest with your app key when the app includes CryptoModule and APP_KEY is set. Without a key they are stored as they are, and the module logs a warning the first time it stores one.

Configuration

config/app.toml

Deliveries run on the webhooks queue here, so give it a worker: list it in queue.queues, or run navree queue:work --queue webhooks. A dedicated worker keeps slow endpoints from delaying your other jobs.

Incoming secrets go in .env, one per source:

.env

Every key also comes from the environment with the APP_ prefix: APP_WEBHOOKS__REQUIRE_HTTPS=false for local development. WebhooksModule::for_root(WebhooksOptions { .. }) configures the module in code.

Sending webhooks

Event-driven, with an outboxA request moves a deal to won. DealService emits DealWon. A sync listener runs at once, inside the same transaction. Everything else is written into the request's unit of work: the deal row, a jobs row for each queued listener and a micro_outbox row for other services, all committed in one transaction. After the commit, a worker runs the queued listener with retries, the deal.updated broadcast reaches browsers over the WebSocket server, and deal.won is published to other services through the broker. A rollback sends nothing.Requestdeal → wonDealServiceemits DealWonSync listenersame transactionUnit of workone transactiondealschangejobslistenersmicro_outboxeventsQueued listenerqueue:work · retriesBroadcastbrowsers, liveOther servicesdeal.won · brokeremit
Event-driven, with an outbox
  • in the request
  • after the commit

Choosing the events

Any event can be a webhook. Implement WebhookEvent for it to say what the payload looks like and whose endpoints receive it:

src/modules/deals/webhooks.rs

The event's NAME (deal.won) is the webhook's type. Without a payload method, data is the event itself as JSON. An event with a workspace_id goes to that workspace's endpoints only; without one (the default), it goes to app-wide endpoints only. Register each webhook event with for_feature in the module that owns it:

src/modules/deals/mod.rs

From then on, emitting the event is all it takes. The deals service already emits DealWon; nothing in it changes:

Rust

When the event is emitted inside a unit of work, as in a request handler, the fan-out job joins it: a request that rolls back sends no webhook. The job builds the payload once, finds the enabled endpoints subscribed to deal.won in the event's workspace, stores one delivery per endpoint and queues each delivery.

To send a webhook that isn't an event, dispatch it by name. Register the name first, with m.webhook_type(name, description) in a module's register function:

src/modules/exports/mod.rs

Then send it with the Webhooks service. send returns the message id; inside a unit of work the job joins it, like an event's:

Rust

An unregistered name, or a payload larger than webhooks.max_payload_bytes, fails the dispatch.

What endpoints receive

Every delivery is a POST with a JSON body and three signature headers:

HTTP
  • webhook-id is the message id: the same for every endpoint of one event and across retries, so receivers can drop duplicates.
  • webhook-timestamp is when the attempt was signed, in Unix seconds; the body's timestamp is when the event happened.
  • webhook-signature is v1, followed by the base64 HMAC-SHA256 of {webhook-id}.{webhook-timestamp}.{body}, keyed with the endpoint's secret (the bytes after whsec_, base64-decoded).
  • While a secret is being rotated, the header carries two signatures separated by a space, one per secret, for 24 hours.
  • Endpoints can add their own headers (an Authorization token, a team name). Host, Content-Type, Content-Length, User-Agent, Connection, Transfer-Encoding, TE and every webhook-* header are reserved.

A receiver in any language verifies it with a Standard Webhooks library. In a Navree app on the other end, that's VerifiedWebhook<S> with the StandardWebhooks verifier.

Retries and the delivery log

An attempt succeeds when the endpoint answers 2xx within webhooks.timeout_secs. Anything else (a non-2xx status, a timeout, a refused connection) is a failed attempt, and the delivery is tried again on this schedule (RETRY_SCHEDULE):

AttemptAfter the previous one
1at once
25 seconds
35 minutes
430 minutes
52 hours
65 hours
710 hours
810 hours

After the eighth attempt, about 27 hours later, the delivery is marked failed and WebhookDeliveryFailed is emitted. An endpoint whose attempts have done nothing but fail for webhooks.disable_after_days (5) is disabled, and WebhookEndpointDisabled is emitted so you can tell its owner. An endpoint that answers 410 Gone is disabled at once. A success clears the failure clock. Deliveries still pending for a disabled endpoint fail when their next attempt is due.

Each attempt is recorded with its status code, the first 4 KB of the response, the error and the duration. Read them with Webhooks:

Rust

A delivery's status is Pending, Succeeded or Failed. retry gives a failed delivery one more attempt, as soon as a worker is free, with the same id and webhook-id; retrying a delivery that has not failed is a NotRetryable error (409 over HTTP).

Deliveries go through the HTTP client and its SSRF guard: an endpoint URL that points to, or resolves to, a private, loopback or cloud metadata address is refused when it is created or changed, and again on every attempt. To deliver to a service on your own network, list it in http_client.ssrf.allow_hosts.

Managing endpoints

The Webhooks service

Rust
MethodWhat it does
create_endpoint(&uow, new)Creates an endpoint and returns it with its secret
update_endpoint(&uow, id, changes)Changes the URL, events, description or headers (EndpointChanges::new().url(..).events(..))
delete_endpoint(&uow, id)Deletes it with its deliveries; pending deliveries are dropped
rotate_secret(&uow, id)A new secret; the old one keeps signing for 24 hours
enable(&uow, id), disable(&uow, id)Turns delivery on or off (enabling forgets past failures)
endpoint(id), endpoints(workspace_id)One endpoint; a workspace's endpoints (None for app-wide ones)
send_test(id)Sends a webhook.test message at once and returns the attempt
deliveries(id, page), delivery(id), attempts(delivery_id)The log
retry(delivery_id), retry_failed()Queues a failed delivery, or every one, again
prune(days)Deletes old deliveries and incoming receipts
event_types()Every registered event type, with its description

The URL must use https (unless webhooks.require_https is off) and carry no credentials; put tokens in a header instead. The events must be registered types; an endpoint subscribed to * receives every event type. A test message is recorded in the log but never counts towards disabling its endpoint. Deleting a workspace deletes its endpoints.

Built-in routes

With webhooks.routes = true (the default) and DatabaseModule in the app, the module mounts JSON routes for a webhooks settings page. They require the webhooks.manage permission, so the app also needs AccessModule and an authenticator. In an app with workspaces they act on the request's workspace (from the X-Workspace header or the other resolvers), and the permission is checked in it; a workspace role gets it through workspace.roles.<role>.permissions, and the owner has it already.

RouteDoes
GET /api/webhooks/event-typesEvery registered type, with its description
GET /api/webhooks/endpoints, POSTLists or creates endpoints (the secret is in the create response only)
GET /api/webhooks/endpoints/{id}, PATCH, DELETEOne endpoint; PATCH takes {url?, events?, description?, headers?, enabled?}
POST /api/webhooks/endpoints/{id}/rotate-secret{secret}
POST /api/webhooks/endpoints/{id}/testSends webhook.test and returns the attempt
GET /api/webhooks/endpoints/{id}/deliveriesThe delivery log, newest first, paginated
POST /api/webhooks/deliveries/{id}/retryQueues a failed delivery again

The routes appear in the OpenAPI document under the Webhooks tag.

Verifying incoming webhooks

A webhook endpoint is a public route that anyone can call, so check that the request comes from who it claims. Take VerifiedWebhook<S> instead of Json<T>, as the last argument: it reads the raw body, verifies the signature with the secret of source S, rejects a signed timestamp more than tolerance_secs away from now, and only then runs your handler.

src/modules/integrations/github.rs

A request that fails verification never reaches the handler:

JSON
RequestAnswer
VerifiedYour handler runs
A handshake (see Handshakes)200, with the text/plain answer the sender expects; your handler doesn't run
Not signed, signed by someone else, or too old400 Invalid webhook signature; the reason is logged at debug level
A body over webhooks.incoming.max_body_bytes413
A sender's key can't be fetched503, so the sender retries; the reason is logged as a warning
The source lacks its secret or a required setting500; the error log names the key
VerifiedWebhook<S>
json::<T>()The body as T; a body that doesn't match is a 400
form::<T>()A form-encoded body (Twilio, Slack commands, Mailgun routes) as T; a body that doesn't match is a 400
body(), headers()The raw bytes that were verified, and the request headers
id()The sender's delivery id (see the table of built-in sources), or the SHA-256 of the body when it sends none
event_type()The sender's event type, when it sends one
timestamp()The signed timestamp (None for senders that sign none)
mark_processed(&uow)Records the id in webhook_receipts in your unit of work; false if it was already there

Senders retry when they don't get a 2xx in time, so the same message can arrive twice. mark_processed writes the receipt in the same transaction as your changes: a message whose handling rolled back is handled again on the next delivery, and one that committed isn't. webhooks:prune deletes receipts older than webhooks.incoming.receipt_days (30).

Built-in sources

Each built-in source is a type you name in VerifiedWebhook<S>, configured under webhooks.incoming.<name>:

SourceNameConfigureChecks
StripestripesecretStripe-Signature (t=, v1=)
GitHubgithubsecretX-Hub-Signature-256 (sha256=)
PaddlepaddlesecretPaddle-Signature (ts=, h1=)
StandardWebhooksstandardsecretwebhook-id, webhook-timestamp, webhook-signature
AwsSnsaws_snstopicsThe message's RSA signature (SHA-1 for signature version 1, SHA-256 for version 2) with the certificate at SigningCertURL, which must be on an SNS host; TopicArn in topics
GooglePubSubgoogle_pubsubservice_account, and audience or urlThe Authorization: Bearer OIDC token: RS256 with Google's keys, issuer, audience, service account and expiry
MicrosoftGraphmicrosoft_graphsecret: the subscription's clientStateThe clientState of every notification in the batch
Twiliotwiliosecret: the auth token; urlX-Twilio-Signature: HMAC-SHA1 of the URL and the sorted form parameters, or of the URL alone with bodySHA256 for a JSON body
Slackslacksecret: the signing secretX-Slack-Signature (v0=) over X-Slack-Request-Timestamp and the body
Mailgunmailgunsecret: the webhook signing keyThe body's timestamp, token and signature, as JSON (event webhooks) or form fields (routes)
Postmarkpostmarksecret: username:passwordThe Authorization: Basic credentials of the webhook URL
Vonagevonagesecret: the signature secret (signed webhooks on)The Authorization: Bearer JWT: HS256 with the secret, its payload_hash equal to the body's SHA-256, its iat within the tolerance
Metametasecret: the app secret; verify_token for the handshakeX-Hub-Signature-256 (sha256=) over the body

What each one says about a request:

Sourceid()event_type()timestamp()
StripeThe event's idIts typet
GitHubX-GitHub-DeliveryX-GitHub-EventNone
Paddleevent_idevent_typets
StandardWebhookswebhook-idThe payload's typewebhook-timestamp
AwsSnsMessageIdType (Notification)Timestamp
GooglePubSubmessage.messageIdNoneThe token's iat
MicrosoftGraphThe body's SHA-256The first notification's lifecycleEvent, else its changeTypeNone
TwilioI-Twilio-Idempotency-TokenNoneNone
Slackevent_id (Events API)event.type, else typeX-Slack-Request-Timestamp
Mailgunevent-data.id, else the tokenevent-data.eventtimestamp
PostmarkMessageIDRecordType (Delivery, Bounce...)None
VonageThe token's jtiThe body's status, else its message_typeThe token's iat
MetaThe body's SHA-256The body's object (whatsapp_business_account)None

GitHub, Twilio, Postmark, Graph and Meta sign no timestamp, so there is no age check for them; use mark_processed to drop replays. SNS retries a failed delivery for a while, so its Timestamp is checked only against the source's own tolerance_secs, when you set one.

The verifiers are also plain functions that take any secret, such as Stripe::verify(secret, headers, body, tolerance), GitHub::verify(secret, headers, body), Twilio::verify(auth_token, url, headers, body), Slack::verify(secret, headers, body, tolerance), Vonage::verify(secret, headers, body, tolerance) or Meta::verify(app_secret, headers, body), so a module that keeps its own secrets can check requests with them; Billing uses them for its Stripe and Paddle webhooks.

A source table ([webhooks.incoming.<source>]) with an empty secret fails the boot, naming the key, unless it sets topics or service_account. A request for a source that lacks its secret or required setting, including one with no table at all, is answered with 500, and the error log names the missing key.

Senders that sign with their own keys

AWS SNS and Google Pub/Sub share no secret with you. They sign with their own private keys, and the app checks the signature with the public half, which it fetches from the sender and caches: an SNS signing certificate for a day, Google's key set for as long as Google's Cache-Control says (an hour when it says nothing), fetched again at most once a minute when a token names a key the cached set lacks.

A valid signature only says the request comes from SNS or Google, not that it is meant for you: anyone can publish to an SNS topic of their own pointed at your URL, and any Google service account can mint a token for any audience. So each source needs a setting that ties the request to your subscription, and refuses to run without it:

config/app.toml

An SES event arrives as an SNS notification whose Message is the event's JSON. Read the body as SnsMessage:

src/modules/mail/ses_events.rs

Leave raw message delivery off on the subscription: a raw message carries no signature. The certificate URL must be HTTPS on an SNS host (sns.<region>.amazonaws.com, or .amazonaws.com.cn), with a path ending in .pem and no query; any other is refused before anything is fetched.

A Pub/Sub push body is a PubSubPush, whose message carries its payload in base64. data_json::<T>() decodes it, here a Gmail notification:

src/modules/mail/gmail_push.rs

Handshakes

Some senders check that your URL answers before they send it anything. The extractor answers these requests itself, with a 200 and the text/plain body the sender expects, so your handler only sees deliveries:

SourceHandshakeAnswer
MicrosoftGraphA request with a validationToken query parameter, when a subscription is createdThe token, decoded
SlackA signed Events API request with "type": "url_verification"Its challenge
AwsSnsA signed SubscriptionConfirmationConfirms it by fetching its SubscribeURL (on an SNS host), then an empty 200
AwsSnsA signed UnsubscribeConfirmationAn empty 200
MetaGET ?hub.mode=subscribe&hub.verify_token=..&hub.challenge=.., when the webhook is set upThe challenge, when the token equals the source's verify_token; otherwise a 400

A Meta route answers the handshake only if it is mounted for GET as well as POST.

With auto_confirm = false, SNS confirmations reach your handler instead (event_type() is SubscriptionConfirmation or UnsubscribeConfirmation), and you confirm with SnsMessage::subscribe_url yourself.

Graph sends a batch of notifications, {"value": [...]}; read it as GraphNotifications. Lifecycle notifications (reauthorizationRequired, subscriptionRemoved, missed) come the same way and carry the same clientState. Graph expects a 202 Accepted within three seconds, so queue the work and answer at once. Notifications with resource data also carry validationTokens, JWTs signed by Microsoft; they are in GraphNotifications::validation_tokens, and the extractor doesn't check them: the clientState is what it verifies.

Senders that sign the URL

Twilio signs the full URL it called, query string included. Behind a proxy or load balancer the app sees another URL, so set the public one:

config/app.toml

Without url, the URL is rebuilt from X-Forwarded-Proto, X-Forwarded-Host and Host, which is right only when your proxy sets them as the caller saw them. The URL is also tried with its default port added or removed, because Twilio sometimes signs it with :443. Read Twilio's form parameters with form::<T>():

Rust

GooglePubSub uses the same public URL as its default audience, which is what Pub/Sub puts in the token when the subscription names none.

Your own sources

Many senders use Standard Webhooks, each with its own secret. Declare a source per sender and pick its verifier:

Rust

Then take VerifiedWebhook<MailEvents>. A source can also set its own tolerance_secs. For a sender with its own scheme, implement WebhookVerifier, or use the generic Hmac verifier, configured with the header, algorithm and encoding:

Rust
TOML

format is the shape of the header value: {signature} once and {timestamp} at most once, with text between them. signed is what is signed, from {timestamp}, {id} and {body}. When the timestamp or the delivery id come in headers of their own, name them with timestamp_header and id_header; event_header names the header that carries the event type.

A WebhookVerifier that signs the headers and body implements verify_request(source, headers, body, tolerance). One that needs the URL, keys fetched from the sender or a handshake implements verify_incoming instead, which takes the whole IncomingRequest (the source's options, headers, body, public URL, arrival time, tolerance and SenderKeys) and returns a Verdict. A partner that checks your URL with a ?challenge= request before signing its deliveries the way GitHub does:

Rust

A verifier that needs a setting other than secret says so in missing_setting, so a request for a source without it is a 500 that names the key.

Notification callbacks

With notifications in the app too (the notify feature; webhooks and notify are both in full), WebhooksModule receives the status callbacks and replies of the SMS and WhatsApp providers. It mounts a public route for each of these sources you configure:

RouteSource
POST /api/webhooks/notify/twilioTwilio: message statuses, and incoming SMS and WhatsApp messages
POST /api/webhooks/notify/vonageVonage: Messages API statuses and inbound messages
GET, POST /api/webhooks/notify/metaMeta: the WhatsApp Cloud API's handshake, statuses and messages

Each request is verified like any VerifiedWebhook<S> of its source, then applied to the notification delivery log: statuses move deliveries forward, and replies of STOP or START update the opt-outs. Nothing is mounted without NotifyModule. Provider callbacks says which URL to give each provider.

Testing

Outgoing

Deliveries go through the app's HttpClient, so a fake client receives them. Run the queue, then assert on what was sent and check the signature with the verifier your customers would use:

tests/webhooks.rs

The last argument of verify is the tolerance: None checks the signature only, Some(Tolerance::secs(300)) also checks the timestamp against the system clock, and Tolerance::at(now, 300) against a time you pick.

Answer with FakeResponse::new(500) to test retries: the delivery stays Pending with one attempt. Override the app's clock with a FixedClock (.override_value::<dyn Clock>(clock.clone())), and moving it past the next wait (clock.advance(RETRY_SCHEDULE[1])) makes the next run_pending try again. To check that an event fans out without sending anything, install a QueueFake and assert on navree::webhooks::FanOut (job webhooks.fan_out):

Rust

Incoming

Each built-in source can sign a test request. Build the body, sign it with the secret you configured for the test, and post it:

Rust

Stripe::sign(secret, timestamp, body), Paddle::sign(secret, timestamp, body) and StandardWebhooks::sign(secret, id, timestamp, body) return the header values for theirs (timestamps in Unix seconds), and Hmac::sign(&source_options, Some(timestamp), None, body) for a configured source. The others:

SourceSign a test request with
TwilioTwilio::sign(auth_token, url, &[("MessageStatus", "delivered")]): the X-Twilio-Signature for the public URL and the form parameters
SlackSlack::sign(secret, timestamp, body): the X-Slack-Signature; send the timestamp in X-Slack-Request-Timestamp
MailgunMailgun::sign(signing_key, timestamp, token): the signature field of the body
PostmarkPostmark::authorization("user:password"): the Authorization header
MicrosoftGraphNothing to sign: put the configured clientState in each notification

AWS SNS and Google Pub/Sub verify with keys the app fetches over its HttpClient, so a fake client serves yours. For Pub/Sub, serve an RSA test key as Google's key set and sign tokens with it:

Rust

For SNS, serve a test certificate at the message's SigningCertURL and sign message.string_to_sign() with its key: key.sign(...) of an RSA SigningKey gives a signature version 2 (SHA-256) signature, base64 it into Signature.

Reference

KeyDefaultMeaning
webhooks.queuedefaultThe queue of fan-out and delivery jobs
webhooks.timeout_secs15Time allowed per attempt
webhooks.require_httpstrueRefuse http:// endpoint URLs
webhooks.disable_after_days5Disable an endpoint failing for this long; 0 never disables
webhooks.max_payload_bytes262144Largest payload (256 KB); a bigger one fails the dispatch
webhooks.retain_days30Deliveries older than this are pruned by webhooks:prune
webhooks.routestrueMounts the /api/webhooks routes (permission webhooks.manage)
webhooks.incoming.tolerance_secs300Accepted age of incoming signatures
webhooks.incoming.max_body_bytes1048576Largest incoming body
webhooks.incoming.receipt_days30How long mark_processed receipts are kept
webhooks.incoming.<source>.secretnoneThe secret of a source; from .env
webhooks.incoming.<source>.tolerance_secsnoneThe source's own tolerance
webhooks.incoming.<source>.header, algorithm, encoding, format, signed, timestamp_header, id_header, event_headersee aboveThe scheme of an Hmac source
webhooks.incoming.<source>.urlrebuilt from the requestThe public URL of the route (Twilio, and the default audience of Google Pub/Sub); the request's query string is appended
webhooks.incoming.aws_sns.topicsnoneThe topic ARNs accepted; required
webhooks.incoming.aws_sns.auto_confirmtrueConfirm subscriptions without the handler
webhooks.incoming.google_pubsub.service_accountnoneThe service account email of the push subscription; required
webhooks.incoming.google_pubsub.audiencethe public URLThe token's expected aud
CommandWhat it does
webhooks:endpoints [--workspace <slug>]Lists endpoints with their status and failure counts
webhooks:test <endpoint-id>Sends webhook.test and prints the response; fails when the endpoint does
webhooks:retry <delivery-id>|failedQueues one failed delivery, or all of them, again
webhooks:prune [--days N]Deletes deliveries older than retain_days and receipts older than receipt_days, or both older than N days
EventNAMEWhen
WebhookDeliveryFailedwebhook.delivery_failedA delivery used its last attempt (or its endpoint was disabled)
WebhookEndpointDisabledwebhook.endpoint_disabledAn endpoint was disabled after failing (reason failing or gone)