# Concurrency

## Introduction

Every pool enforces a hard **concurrency limit**: at most N requests may be
in flight to the upstream at once. Additional requests wait in an overflow
queue, fail fast, or time out, depending on your settings and request headers.

Configure these settings under **Pools → Concurrency**. Plan limits cap the maximum
values you can set (see [Billing](/docs/billing)).

## Concurrency mode

### Fixed

You set **Max concurrency**. The pool never admits more than that many in-flight
upstream requests.

When the pool is attached to a [Capacity](/docs/capacity) budget, this value is
the pool's reserved share of the shared concurrency limit (or its reserved pin under
Auto). Edit it here or on the budget. Both update the same allocation.

### Adaptive

The pool learns a safe effective limit from upstream `429` responses and stays
at or below your **max concurrency**. Set a **floor** (`min concurrent`) so
the learned value does not collapse too far.

On an Auto capacity budget, adaptive remains an inner cap under the pool's
budget entitlement. It never raises the pool above what the budget allows.

## Capacity budgets

Interactive, batch, and partner pools that share one concurrency limit should use a
[Capacity budget](/docs/capacity). **Fixed** pins each pool's share. **Auto**
keeps reserved pins local and moves float toward load under one shared
concurrency limit. See [Balancing](/docs/capacity/balancing).

## Queue

When a slot is not free but the queue has room, a request holds its
connection open and waits: an **open wait**. It ends one of two ways, a slot
frees and the request proceeds, or the wait budget runs out and it gets
`504 queue_timeout`.

If the queue is already at max depth, the request never enters an open wait.
It gets `429` immediately instead, so you are not left waiting on a slot that
was never going to open in time.

| Setting         | Meaning                                                 |
| --------------- | ------------------------------------------------------- |
| Max queue depth | How many requests may be in an open wait at once        |
| Queue timeout   | Wait budget for an open wait before `504 queue_timeout` |

Callers can override the wait budget with `X-Nthpool-Queue-Timeout-Ms`, or skip
the open wait entirely with `X-Nthpool-No-Queue`.

When the queue is full (or fail-fast is set), the pool returns **`429`** with
`Retry-After` when applicable. It is not a silent drop.

For callers that cannot hold an open wait, enable [Async](/docs/pools/async) on
the Concurrency tab (header opt-in, on open wait, or on queue timeout)
and attach [Webhooks](/docs/webhooks).

## Timeouts

Queue timeout waits for a slot. Request timeout holds it after admission.
Total is the sum. Set your client timeout to that total.

| Setting         | When                    | On expiry                         |
| --------------- | ----------------------- | --------------------------------- |
| Queue timeout   | Waiting for a free slot | `504 queue_timeout`               |
| Request timeout | Holding a slot          | `504 request_timeout`, slot freed |

If the caller disconnects, the pool aborts upstream and frees the slot.

## RPM (requests per minute)

Optional token-bucket rate gate on top of concurrency:

- **Fixed:** govern to `max RPM`
- **Adaptive:** learn a safe RPM from upstream `429`s
- **Header:** follow upstream `x-ratelimit-*` headers when the provider exposes them

## Retry on 429

When enabled, the pool retries upstream `429` responses a small number of times
(with short backoff) for eligible request bodies, recording throttle telemetry
that also feeds adaptive control.

## Response cache

nthbouncer can cache successful `GET` / `HEAD` responses at the edge so later
matching requests skip admission and the upstream fetch. The feature is
**on per pool by default**. Callers still opt each response in with
`X-Nthpool-Cache-TTL`.

Turn the feature off on Concurrency if a pool should never store responses.

```bash
curl -sS "https://your-pool.nthpool.cloud/v1/items" \
  -H "X-Nthpool-Key: $POOL_KEY" \
  -H "X-Nthpool-Cache-TTL: 60"
```

TTL is a positive integer in seconds, capped at **300**. Lookup runs only for
unconditional `GET`/`HEAD` without inbound HMAC. A miss is stored only for
successful non-partial `2xx` responses without `Set-Cookie`, `Content-Range`,
`Vary`, or cache-forbidding `Cache-Control`. Hits still count toward billing.

See the [response cache cookbook](/docs/knowledge-base/response-cache) for a
worked example.

## Next steps

- [Pools](/docs/pools)
- [Capacity](/docs/capacity)
- [Concurrency locks API](/docs/api/locks)
- [Admission errors](/docs/knowledge-base/admission-errors)