Platform
Reliability, retries and timeouts
Every request is watched from start to first token. If a path to the model stalls or fails, the request moves to another before anything reaches you.
Self-healing
- Most models are served over several paths. A request starts on the healthiest one.
- A path that errors or can’t be reached is dropped for that request and retried elsewhere. A path that’s working is waited for, however long a big request takes; one that stays completely silent for 3 minutes (5 at high reasoning effort) counts as failed.
- Paths that keep failing are moved to the back of the line for everyone. One that goes silent on a fifth or more of its requests goes behind a path that’s answering cleanly. Each is tried again once it recovers.
- Nothing reaches you until a real token has arrived, so all of this is invisible: you see a slightly slower answer, not an error.
The models page shows each model’s live status (operational, degraded or down), from real requests and our own checks: every chat model is tested at least every 10 minutes.
Timeouts
A stream sends nothing for its first 20 seconds without a first token, then opens with keep-alive comments until the answer starts. A non-streaming request sends nothing until it’s done, so streaming suits clients that give up on slow headers (Node’s fetch waits 5 minutes for them). Set your client’s timeout to at least 10 minutes for long reasoning, long outputs and images. The OpenAI SDKs default to 10 minutes; some HTTP clients default to 30 or 60 seconds.
Rate limits
There’s no global rate limit on your account: keys are limited only if you set a rate limit on them. A workspace can run 32 requests at the same time; one more gets 429 concurrency_limit_exceeded with retry-after. A model that’s overloaded is retried on its other paths; if none answers, you get 503 model_unavailable with retry-after. Back off and retry, or turn on smart routing.
When a model is down
Rarely, every path to a model fails. Without smart routing, the request ends with 503 model_unavailable and retry-after. When the paths error while the model is still answering other requests, that’s a blip that usually passes in seconds, so we keep retrying them for up to 2 minutes first. When the model is down for everyone, the error comes at once, and when paths stay silent, within 10 minutes. With smart routing, a close alternative answers instead.
Questions, or something missing? Ask our support team and we’ll answer by email, usually within a minute.