Platform

Reliability, retries and timeouts

Every request is watched from start to first token. If a path to the model stalls or fails, the request moves to another before anything reaches you.

Self-healing

  • Most models are served over several paths. A request starts on the healthiest one.
  • A path that errors or can’t be reached is dropped for that request and retried elsewhere. A path that’s working is waited for, however long a big request takes; one that stays completely silent for 3 minutes (5 at high reasoning effort) counts as failed.
  • Paths that keep failing are moved to the back of the line for everyone. One that goes silent on a fifth or more of its requests goes behind a path that’s answering cleanly. Each is tried again once it recovers.
  • Nothing reaches you until a real token has arrived, so all of this is invisible: you see a slightly slower answer, not an error.

The models page shows each model’s live status (operational, degraded or down), from real requests and our own checks: every chat model is tested at least every 10 minutes.

Timeouts

Limit
Waiting for the first token3 minutes on each path before the next is tried (5 at high reasoning effort), plus 1 second per 5,000 prompt tokens (up to 30); 10 minutes in all
A quiet stream (no data)90 seconds; keep-alive comments arrive every 15 seconds, from 20 seconds into the wait for a first token
A whole streamNo limit while data keeps coming
A whole non-streaming request10 minutes
An image3 minutes per attempt, retried on another path if it stalls

A stream sends nothing for its first 20 seconds without a first token, then opens with keep-alive comments until the answer starts. A non-streaming request sends nothing until it’s done, so streaming suits clients that give up on slow headers (Node’s fetch waits 5 minutes for them). Set your client’s timeout to at least 10 minutes for long reasoning, long outputs and images. The OpenAI SDKs default to 10 minutes; some HTTP clients default to 30 or 60 seconds.

Rate limits

There’s no global rate limit on your account: keys are limited only if you set a rate limit on them. A workspace can run 32 requests at the same time; one more gets 429 concurrency_limit_exceeded with retry-after. A model that’s overloaded is retried on its other paths; if none answers, you get 503 model_unavailable with retry-after. Back off and retry, or turn on smart routing.

When a model is down

Rarely, every path to a model fails. Without smart routing, the request ends with 503 model_unavailable and retry-after. When the paths error while the model is still answering other requests, that’s a blip that usually passes in seconds, so we keep retrying them for up to 2 minutes first. When the model is down for everyone, the error comes at once, and when paths stay silent, within 10 minutes. With smart routing, a close alternative answers instead.

Questions, or something missing? Ask our support team and we’ll answer by email, usually within a minute.