Concurrency
Introduction
Every pool enforces a hard concurrency limit: at most N requests may be in flight to the upstream at once. Additional requests wait in an overflow queue, fail fast, or time out, depending on your settings and request headers.
Configure these settings under Pools → Concurrency. Plan limits cap the maximum values you can set (see Billing).
Concurrency mode
Fixed
You set Max concurrency. The pool never admits more than that many in-flight upstream requests.
When the pool is attached to a Capacity budget, this value is the pool's reserved share of the shared concurrency limit (or its reserved pin under Auto). Edit it here or on the budget. Both update the same allocation.
Adaptive
The pool learns a safe effective limit from upstream 429 responses and stays
at or below your max concurrency. Set a floor (min concurrent) so
the learned value does not collapse too far.
On an Auto capacity budget, adaptive remains an inner cap under the pool's budget entitlement. It never raises the pool above what the budget allows.
Capacity budgets
Interactive, batch, and partner pools that share one concurrency limit should use a Capacity budget. Fixed pins each pool's share. Auto keeps reserved pins local and moves float toward load under one shared concurrency limit. See Balancing.
Queue
When a slot is not free but the queue has room, a request holds its
connection open and waits: an open wait. It ends one of two ways, a slot
frees and the request proceeds, or the wait budget runs out and it gets
504 queue_timeout.
If the queue is already at max depth, the request never enters an open wait.
It gets 429 immediately instead, so you are not left waiting on a slot that
was never going to open in time.
| Setting | Meaning |
|---|---|
| Max queue depth | How many requests may be in an open wait at once |
| Queue timeout | Wait budget for an open wait before 504 queue_timeout |
Callers can override the wait budget with X-Nthpool-Queue-Timeout-Ms, or skip
the open wait entirely with X-Nthpool-No-Queue.
When the queue is full (or fail-fast is set), the pool returns 429 with
Retry-After when applicable. It is not a silent drop.
For callers that cannot hold an open wait, enable Async on the Concurrency tab (header opt-in, on open wait, or on queue timeout) and attach Webhooks.
Timeouts
Queue timeout waits for a slot. Request timeout holds it after admission. Total is the sum. Set your client timeout to that total.
| Setting | When | On expiry |
|---|---|---|
| Queue timeout | Waiting for a free slot | 504 queue_timeout |
| Request timeout | Holding a slot | 504 request_timeout, slot freed |
If the caller disconnects, the pool aborts upstream and frees the slot.
RPM (requests per minute)
Optional token-bucket rate gate on top of concurrency:
- Fixed: govern to
max RPM - Adaptive: learn a safe RPM from upstream
429s - Header: follow upstream
x-ratelimit-*headers when the provider exposes them
Retry on 429
When enabled, the pool retries upstream 429 responses a small number of times
(with short backoff) for eligible request bodies, recording throttle telemetry
that also feeds adaptive control.
Response cache
nthbouncer can cache successful GET / HEAD responses at the edge so later
matching requests skip admission and the upstream fetch. The feature is
on per pool by default. Callers still opt each response in with
X-Nthpool-Cache-TTL.
Turn the feature off on Concurrency if a pool should never store responses.
curl -sS "https://your-pool.nthpool.cloud/v1/items" \
-H "X-Nthpool-Key: $POOL_KEY" \
-H "X-Nthpool-Cache-TTL: 60"TTL is a positive integer in seconds, capped at 300. Lookup runs only for
unconditional GET/HEAD without inbound HMAC. A miss is stored only for
successful non-partial 2xx responses without Set-Cookie, Content-Range,
Vary, or cache-forbidding Cache-Control. Hits still count toward billing.
See the response cache cookbook for a worked example.