AI Product Manager that remembers everything
← Tickets · AET-828
Webhook deliveries dropping during 5k req/min bursts
In progress
High priority
Opened by Dev Kim
Under load, a slice of outbound webhooks never reach subscribers. Repro at ~5k req/min. Retry logic lives in webhooks/dispatch.py; the current MAX_RETRIES is low — bumping it looks like the obvious fix.
Tasks · 1 of 2
Reproduce under load harness
Decide retry strategy
Coding agent runs on your machine
Open in Terminal ▾
Implements the change on fix/webhook-drops and opens a pull request.
← AET-828 · Link a pull request
Add backpressure to webhook dispatch #482
3 files · +64 / −12
Summary
- New
Backpressuregate inbackpressure.py— returns 429 before enqueue when saturated. - Retry path unchanged —
MAX_RETRIES = 3, backoff intact.
Impact radius
lib/ratelimit.TokenBucket already does load-shedding — reuse it, don’t rebuild.Org norm: cross-service middleware lives in
lib/, not per-service.Shared webhook path — 3 other services dispatch through it.
Change log · 3 files
webhooks/backpressure.pynew+41
webhooks/dispatch.py+14−12
tests/test_backpressure.pynew+9
Tickets
Current sprint 3
AET-828Checkout 500s post-SSOHighIn progressS
AET-819SSO callback multi-domain mappingHighIn progressJ
AET-811Webhook backpressure at 5k rpsMedOpenY
Next sprint 2
AET-814CRM auto-link by Account IDMedOpen—
AET-782Audit-log retention configLowOpenA
It remembers everything.
And decides what’s next.
And decides what’s next.
AETHRIONXIt caught the November incident — before a line of code was written.
