Scaling a Public API and Webhooks Under Bursty Traffic

Problem / Why it mattered

This API was a real public contract that partner systems depended on, not just an internal endpoint layer. The hardest moments were bursty traffic patterns: large sync jobs, retry storms after transient failures, and webhook consumers going offline at the wrong time. On normal days the platform was fine, but spikes exposed expensive query paths, inconsistent endpoint guardrails, and webhook delivery that still behaved like best effort.

Constraints

• Backward compatibility had to remain the default, with additive changes preferred over breaking changes.
• Clients had mixed receiver quality and variable timeout behavior during peak load.
• Webhook semantics required at-least-once delivery, so duplicates had to be safe for consumers.
• Support needed clear observability to diagnose partner-specific failures quickly.

What I changed (design + architecture)

• Applied explicit API versioning rules with a clear migration path for breaking changes.
• Standardized pagination, filtering, and error shapes; removed one-off logic to reduce edge-case behavior.
• Added HTTP caching controls with Cache-Control and conditional requests using ETags where safe.
• Moved outbound webhooks to queue-first delivery with retries, backoff, jitter, and tenant isolation.
• Added replay tooling, signature verification, and structured lifecycle events for webhook operations.

What I measured

• Queue depth and time-to-drain during burst windows.
• Webhook delivery success by retry attempt, endpoint type, and tenant.
• Error-class distribution and duplicate-effect incidents after idempotency rollout.
• Mean time to diagnose and recover integration incidents.

Result

Spikes became a known and measurable shape instead of a recurring emergency. Versioning protected the contract, caching protected origin performance, and queue-based webhooks made delivery behavior predictable enough for both engineering and support teams to trust.

What I’d do next

Add partner-facing diagnostics and self-serve replay controls, then define explicit reliability SLOs per tenant class.