Rate limits

Limits protect shared GPU capacity. They are enforced per virtual key by the router. Set request and token limits on each key so one workload cannot starve another.

LayersLink to this section

Reading remaining capacityLink to this section

When the router supplies it, responses include x-mainai-remaining-requests for the current window. It is a hint, not a reservation. A 429 may include Retry-After.

curl -sS -D - -o /dev/null https://cloud.mainai.dk/v1/models \
  -H "Authorization: Bearer $MAINAI_API_KEY" \
  | grep -iE "^(x-request-id|x-mainai-remaining-requests|retry-after):"

BackoffLink to this section

1. 01

   Stop on 429

   Do not hammer through the window. Pause the worker or queue the job.

2. 02

   Honor Retry-After

   If present, wait at least that long. Otherwise use exponential backoff with jitter.

3. 03

   Retry once the window opens

   Keep the retry budget small and surface exhaustion to your own caller.

4. 04

   Reduce the base rate

   If 429s recur, lower worker concurrency or raise the key limit.

const wait = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function requestWithBackoff(init: RequestInit, attempts = 4) {
  for (let i = 0; i < attempts; i++) {
    const res = await fetch(`https://cloud.mainai.dk/v1/embeddings`, init);
    if (res.status !== 429) return res;

    const retryAfter = Number(res.headers.get("retry-after"));
    await wait(Number.isFinite(retryAfter) ? retryAfter * 1000 : 500 * 2 ** i + Math.random() * 250);
  }
  throw new Error("rate limit retries exhausted");
}

Design for the limitLink to this section

• Batch embeddings instead of sending one document per request.
• Queue bulk backfills separately from interactive traffic.
• Reuse HTTP connections; do not open a client per call.
• Use streaming for long generations so the user sees progress before the first full response completes.
• Use max_tokens so one runaway response cannot consume a whole window.
• Split workloads across keys when they need independent limits, then cap each key's budget.

Rate limits versus availabilityLink to this section

A 429 is a throughput decision, not an outage. A 503 with upstream_unavailable means the API could not reach an engine. Both are retryable, but only the second belongs on an availability dashboard.