Centralized or Embedded: Where to Enforce Rate Limiting for High-Concurrency APIs

A backend engineer designing a system that can absorb a flash crowd without collapsing the database faces a decision early in the architecture: place the rate limiting logic inside the application or delegate it to a dedicated API gateway. Neither choice is inherently correct. The right answer depends on the shape of your traffic, your tolerance for latency overhead, and the maturity of your deployment pipeline. This article dissects the two approaches through the lens of API design best practices, focusing on the operational realities of high concurrency rather than abstract feature lists.

Deciding Where the Bucket Drops: Gateway vs Application Throttling

The first trade-off is coupling. An API gateway introduces a network hop and a single point of configuration. It decouples rate limiting from your business logic, meaning a change to the throttling policy does not require a redeployment of your service. If your team manages infrastructure separately from application code, the gateway reduces cross-team coordination. Conversely, embedding rate limiting inside your Node.js service ties the throttle policy to the application lifecycle. Every change to the rate limit requires a build and deploy, which may be acceptable if your release cadence is fast and your team owns both the infrastructure and the application.

Infrastructure ownership matters. A small team running a handful of microservices on a single Kubernetes cluster may find a managed API gateway (e.g., Kong, AWS API Gateway) to be overhead they do not need. They can implement rate limiting with a few lines of middleware in Express or Fastify and achieve identical results under most loads. A larger organization with multiple teams, each deploying independently, benefits from the gateway because it centralizes policy enforcement and prevents a single service from silently adopting a different algorithm.

Team capability also plays a role. Node.js performance tuning is easier when you control every line of the request path. If your team lacks experience with gateway configuration, the embedded approach reduces the risk of misconfiguring a reverse proxy that could accidentally drop legitimate requests.

When Your Database Can't Take Another Wave: Matching Throttle Placement to System Constraints

Choose an API gateway when your system has multiple backend services that share a single database or a common downstream resource. The gateway sees the aggregate request rate across all services and can enforce a global limit that no single service can circumvent. This is critical for high concurrency scenarios where the database connection pool is the scarcest resource. A service-level rate limiter, even if perfectly tuned, cannot prevent a different service from exhausting the pool.

Choose the embedded approach when your system is a single monolith or when each service owns its own isolated database. In that case, the gateway adds unnecessary network latency and a point of failure. Additionally, if your latencies are extremely tight (sub-millisecond p99 responses), the extra hop of a gateway can double your overhead. Some high-frequency trading or real-time analytics systems cannot afford that.

The non-obvious insight: In a microservices architecture, an API gateway's rate limiting can create a false sense of protection. The gateway enforces a request count per client, but the real bottleneck is often the database connection pool, which has a non-linear response curve under load. A gateway that limits to 100 requests per second per client might still allow 100 clients to each send 100 requests per second, resulting in 10,000 concurrent database connections. The gateway sees no violation; the database sees catastrophic contention. Effective rate limiting must be calibrated using observed database latency and connection pool metrics, not just request counts. This calibration is easier to achieve when the rate limiting logic lives inside the application, because it can directly observe response latency from the database driver.

Two Architectures, One Production Outage: Case Studies in Rate Limiting Placement

Consider a real-world scenario: a SaaS platform with a single PostgreSQL database and eight microservices. The team placed rate limiting in each service individually, using a token bucket algorithm with Redis. Under normal load, this worked. During a marketing campaign, one service received a surge that stayed below its own rate limit, but it triggered cascading retries from other services that also stayed below their limits. The database connection pool saturated. The gateway, had it been present, could have enforced a global cap at the entry point, preventing the aggregate from exceeding the pool size.

Now consider a second scenario: a low-latency chat application where every user interacts with a single Node.js process handling WebSocket connections and REST endpoints. The team placed rate limiting in a Kong gateway running on the same host. The gateway introduced a 2ms overhead per request, acceptable. However, a bug in the upstream proxy caused periodic connection drops, leading to 5xx errors. The team spent two days debugging because the error logs were scattered between gateway and application. If rate limiting had been embedded, the failure domain would have been the application itself, simplifying root cause analysis.

Operational Friction: What the Documentation Doesn't Tell You About Each Approach

Gateway advantages: centralized logging, unified policy, separation of concerns, and the ability to add rate limiting without touching application code. Gateway disadvantages: increased latency, additional operational complexity (monitor another service), and the risk of becoming a bottleneck if not scaled horizontally. Under high concurrency, the gateway itself must be rate-limited or autoscaled, or it becomes the single point of failure.

Embedded advantages: lower latency, simpler debugging, no extra network call, and the ability to make rate limiting decisions based on real-time application context (e.g., current database response time). Disadvantages: policy changes require redeployment, rate limiting configuration is scattered across services, and it is harder to enforce global aggregates without a shared state store like Redis. If Redis goes down, each service must fall back to local limits, potentially losing global fairness.

TL;DR

• Centralize rate limiting in an API gateway when multiple services share a downstream resource and you need a global per-client cap.
• Embed rate limiting in your Node.js application when latency is critical, the team owns the entire stack, or the downstream resource is per-service.
• Never calibrate rate limits solely on request counts; use observed database latency and connection pool metrics.
• Gateways add an extra hop and a failure domain; embedded logic adds deployment coupling and scattered policy.
• The choice is not permanent. Start with the simplest correct approach (usually embedded for a monolith, gateway for a polyglot microservice architecture) and migrate as complexity grows.

For backend engineering services that include high-concurrency API design, Node.js performance tuning, and production-grade rate limiting architecture, contact BaseStation Private Limited at [email protected].