6 problems your API gateway is already suffering from

An API gateway looks like a thin proxy until the night it becomes the outage. Centralization is the point: one place for TLS, auth, rate limits, routing, and policy. Centralization is also the risk: every request shares the same finite event-loop budget, connection pools, and blast radius.

Microsoft’s Gateway Offloading guidance says it plainly: keep the gateway highly available, size it so it is not a bottleneck, and never push business logic into it. Canva’s public incident report for 12 November 2024 shows what happens when that shared front door saturates — roughly 1.5 million requests per second, about 3× typical peak, combined with a telemetry lock that starved the Netty event loop until Linux OOM-killed the fleet.

This article is a diagnostic checklist of six problems gateways already suffer from in production — and how to fix them before the next thundering herd.

What an API gateway is (and is not)

An API gateway is the edge facade in front of one or more backends. Typical duties: terminate TLS, authenticate callers, enforce quotas, route by path or header, emit metrics and traces, optionally transform headers or paths.

It is not a substitute for:

• Per-service authorization with domain context
• Application BFF composition that understands product screens
• Unlimited retries that hide systemic overload
• A place to dump “just one more plugin” until the hot path blocks

If you treat it as a passive pipe, you will miss its failure modes. If you treat it as an application server, you will invent new ones.

Problem 1 — Shared blast radius and saturation

Symptom: The site is “down” even though most backends are healthy. Autoscaling adds capacity that dies as soon as it becomes healthy.

Why it happens: One gateway cluster fronts everything. A spike, a blocking call on the event loop, or memory pressure takes the whole public API with it. Microsoft warns explicitly to avoid single points of failure and to keep the gateway from becoming the bottleneck.

Canva’s outage is the reference case. A CDN path delay queued 270,000+ clients on one asset. When the asset finally arrived, clients resumed and hit the API Gateway with a synchronized herd. Load balancers opened more connections into already overloaded tasks. Off-heap memory grew; the OOM killer cleared containers faster than autoscaling could refill them. Mitigation only worked after traffic was blocked at the CDN so new tasks could start cold.

What to do:

• Raise baseline capacity and memory headroom; load-test the gateway like a tier-0 service
• Add load shedding (reject early with 503) before queues eat the fleet
• Isolate failure domains (separate gateways or listener pools per product surface) so one herd cannot claim every task
• Keep a rehearsed runbook to shed or restore traffic at the CDN / edge, not only inside the cluster

Saturation is not “the gateway is slow.” It is “the gateway is the choke point for the product.”

Problem 2 — Broken timeout budgets

Symptom: Clients see 504 / abandoned requests while the gateway still holds upstream work. Connection pools fill. Latency climbs, then everything fails together.

Why it happens: Timeouts are set once and forgotten. If the gateway waits as long as (or longer than) the client, you keep spending gateway concurrency on requests the user already abandoned. Slow upstreams then fill pools; pending queues grow; new work is rejected while backends stay overloaded — a self-reinforcing loop.

AWS documents the hard edge of this class of failure for Amazon API Gateway: when an integration exceeds the configured maximum wait, callers get HTTP 504 (“Endpoint request timed out”). The default integration timeout has long been 29 seconds for many REST setups; as of June 2024 AWS allows raising it for Regional and private REST APIs (often with throttle-quota trade-offs). Raising the ceiling without fixing the hierarchy only keeps zombies alive longer.

What to do:

client timeout  >  gateway→upstream timeout  >  per-try timeout

• Make gateway→backend timeouts strictly smaller than client→gateway
• Cap pending-request queues; prefer early 503 over a 30-second wait that fails anyway (Envoy tracks max_pending_requests for this reason)
• Propagate a deadline (absolute time) so nested services can fail fast when the budget is already spent
• For long work, do not stretch the synchronous gateway path — use async patterns (queue + status) instead of “just increase the timeout”

See also our reference on timeouts, retries, jitter, and backoff.

Problem 3 — Retry storms without a budget

Symptom: Upstream error rates are elevated, then suddenly multiplied. Gateway CPU and upstream QPS spike together. Recovery takes longer than the original fault.

Why it happens: Retries are selfish: they spend more of the server’s time to improve your success rate. AWS Well-Architected REL05-BP03 calls out retries without backoff, jitter, and caps — and especially retries at multiple layers that compound into a storm. Envoy’s docs say the same in gateway terms: limit outstanding retries so sporadic failures can still be retried, but volume cannot explode into cascading failure. Prefer a retry budget (default guidance often centers on roughly 20% of active + pending traffic as concurrent retries) over a static max_retries alone.

What to do:

• Retry only idempotent, transient failures (connect errors, timeouts, selected 5xx / 429) — not validation or auth 4xx
• Cap attempts and total deadline; add exponential backoff with jitter
• Configure a cluster-level retry budget / max concurrent retries (Envoy retry_budget overrides static retry circuit breakers when set)
• Retry in one place in the stack (client or gateway or mesh) — not all three
• Pair with circuit breakers and outlier ejection so you stop sending full traffic to a dead host — while capping max_ejection_percent so a correlated bad deploy cannot eject the entire cluster (Envoy defaults this ceiling to 10%)

Problem 4 — The god gateway (business logic at the edge)

Symptom: Every product change needs a gateway deploy. Plugins parse JSON bodies, reshape fields, and encode domain rules. Latency and cognitive load climb together.

Why it happens: Offloading cross-cutting concerns is correct. Offloading business logic is not. Microsoft’s Gateway Offloading pattern states it without hedging: “Business logic should never be offloaded to the gateway.” Header injection, path rewrites for migrations, and correlation IDs belong at the edge. Filtering response fields by product semantics, aggregating five services into one “screen DTO,” or encoding entitlement rules in Lua/JS plugins duplicate the domain in the worst place: the shared hot path.

What to do:

Keep the gateway structurally aware and semantically blind. If a change needs a product owner’s approval more than an SRE’s, it probably should not live in gateway config.

Problem 5 — Blind observability and blocking work on the hot path

Symptom: Dashboards show “API red” with no split between edge overhead and upstream time. Or the gateway looks fine until load rises — then a “harmless” metric library locks threads and throughput collapses.

Why it happens: Gateways often emit one success/error rate and stop. You cannot tell whether clients wait on TLS, plugins, auth round-trips, or the backend. Worse: synchronous logging, token introspection on every request, or contended locks inside telemetry run on the event loop. Canva’s postmortem is explicit: Netty event-loop threads must not block; a telemetry re-registration under a lock reduced per-task throughput right when the herd arrived.

What to do:

• Separate gateway overhead (client-visible latency minus upstream time) from upstream latency; alert on both
• Track upstream pool health: active connections, pending queue depth, retry overflows, circuit-breaker opens (Envoy exposes counters such as upstream_rq_retry_overflow and pending overflows)
• Prefer local JWT verification with cached JWKS over synchronous introspection on every call when latency budgets are tight
• Treat telemetry and plugins as load-bearing code: no blocking I/O on the request path; load-test plugin chains at 50–80% capacity, not only at idle
• Propagate one correlation / trace ID end to end — generate at the edge if missing

If you cannot answer “is the edge sick or is payments sick?” in under a minute, this problem is already yours.

Problem 6 — Weak edge and control-plane hardening

Symptom: Rate limits that look correct on paper are trivially bypassed. Backends are reachable without the gateway. The Admin / control API is exposed wider than anyone intended.

Why it happens: Three classic gaps:

1. Per-node rate limits — in-memory counters on N replicas behave like N independent limits. Shared stores (for example Redis) fix global caps but add latency and a new dependency; choose the algorithm deliberately (token bucket is usually what public APIs want). Always return 429 with Retry-After (or equivalent) so clients do not retry immediately and amplify load.
2. Backend bypass — Microsoft’s gateway guidance reminds you the gateway is the public endpoint: limit public access to backends so policy cannot be skipped.
3. Control plane exposure — Kong documents that the Admin API grants full control of services, routes, plugins, and credentials. Default listening on localhost is intentional; binding 0.0.0.0:8001 can “seriously compromise the security of your whole Kong cluster.” Trend Micro’s case study on Kong misconfigurations makes the same point: Admin API and datastore access must stay tightly scoped.

What to do:

• Enforce authn and coarse throttling at the edge; put backends on private networks
• Use a shared, atomic counter for global rate limits when you need a true cluster-wide cap
• Lock down Admin / control APIs: localhost or private CIDR, firewall ACLs, authn on any remote path, RBAC where available — never a casual public bind
• Continuously scan for shadow routes that skip the gateway

The data plane is only as trustworthy as the control plane that configures it.

How the six problems reinforce each other

spike / deploy / CDN blip
        │
        ▼
   saturation (1)  ←── blocking telemetry / plugins (5)
        │
        ├── broken timeouts (2) → pool exhaustion
        ├── unbounded retries (3) → load ×N
        └── god-gateway work (4) → less headroom
                │
                ▼
        control gaps (6) → bypass or unsheddable traffic

Fixing only retries while the event loop blocks — or only HA while timeouts are inverted — leaves the amplifier intact.

FAQ

Is an API gateway always a single point of failure?

It is a shared failure domain unless you invest in redundancy, isolation, and shedding. Multiple instances behind a load balancer remove a single process as SPOF, but one shared config, one telemetry bug, or one global herd can still take the product offline — as Canva showed. Treat availability of the gateway as a product requirement, not a checkbox.

Should the gateway retry for me?

Sometimes, for idempotent GETs and safe transient errors — with budgets. Prefer one retry layer. Gateway retries that stack on client retries and mesh retries are a common path to storms (AWS REL05-BP03).

Gateway vs BFF vs service mesh — who does what?

• Gateway: north-south edge policy (TLS, authn, coarse limits, routing)
• BFF: product-shaped aggregation and UI-oriented contracts
• Mesh: east-west resilience (timeouts, retries, mTLS between services)

Overlap is fine; duplicate retry + authz logic in all three is not.

How do I know my timeout hierarchy is wrong?

If you see upstream work completing after the client has disconnected, growing pending queues under slow backends, or frequent 504 while integrations still run past the client’s patience, the budget is inverted or missing. Measure client timeout, gateway timeout, and upstream duration on the same trace.

Conclusion

Your API gateway is already under pressure from six directions: shared saturation, inverted timeouts, unbudgeted retries, domain logic at the edge, opaque or blocking observability, and soft hardening of data and control planes. None of these are theoretical — they show up in vendor docs, architecture patterns, and public postmortems.

Pick one production route this week. Verify the timeout hierarchy, confirm retries are budgeted in a single layer, measure gateway overhead separately from upstream time, and confirm Admin and backend networks cannot bypass policy. The goal is not a smarter god-proxy. It is a boring, shedding, observable edge that fails in small pieces instead of taking the product with it.

References

1. Canva incident report: API Gateway outage — Canva Engineering (Dec 2024)
2. Gateway Offloading pattern — Microsoft Azure Architecture Center
3. API gateways in microservices — Microsoft Azure Architecture Center
4. Gateway Routing pattern — Microsoft Azure Architecture Center
5. Circuit breaking — Envoy Proxy documentation
6. Outlier detection — Envoy Proxy documentation
7. Circuit breakers (proto) — retry_budget — Envoy Proxy API
8. REL05-BP03 Control and limit retry calls — AWS Well-Architected
9. REL05-BP05 Set client timeouts — AWS Well-Architected
10. Troubleshoot API Gateway HTTP 504 timeout errors — AWS re:Post
11. Amazon API Gateway integration timeout limit increase beyond 29 seconds — AWS News (Jun 2024)
12. Secure the Admin API — Kong Gateway documentation
13. Kong API Gateway Misconfigurations: An API Gateway Security Case Study — Trend Micro
14. Timeouts, retries, and backoff with jitter — Amazon Builders’ Library
15. Exponential Backoff And Jitter — AWS Architecture Blog