Gateway API — Chat Completions

Phase 2 AI Gateway. Base path: /api/v1/gateway. An OpenAI client points its baseURL at https://<host>/api/v1/gateway.

Related references: provider connections, virtual keys, budgets, usage analytics.

Auth for POST /chat/completions (gatewayAuth): a virtual key (Authorization: Bearer agh_sk_…) is the primary machine path; a session cookie or personal/team API key also works but requires role owner/admin/editor (viewers get 403). Response metadata is returned in x-gateway-* headers.

All examples below were verified with real curl output against a running server using live OpenAI and Gemini connections.

POST /api/v1/gateway/chat/completions (non-streaming)

OpenAI-compatible chat completion. The gateway resolves the team's connection for the model's provider, calls it, prices the call, and records a request-log row.

curl
TypeScript
Python

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Content-Type: application/json" \
  --cookie "connect.sid=<session>" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":10}'

const result = await hub.gateway.chat({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Say hi in one word.' }],
  maxTokens: 10,
});
console.log(result.content);

result = await hub.gateway.chat(
    'gpt-4o-mini',
    [{'role': 'user', 'content': 'Say hi in one word.'}],
    max_tokens=10,
)
print(result.content)

Response headers (status 200):

x-gateway-request-id: 561e701d-d775-44b8-ab85-f9a7e75b3b3b
x-gateway-provider: openai
x-gateway-model: gpt-4o-mini-2024-07-18
x-gateway-cost-usd: 0.00000315
x-gateway-cache: miss

Response body (status 200):

{
  "id": "chatcmpl-DxoMTfx6c4OtbeOrvFXf9m6qKi2P6",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1783147313,
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 13, "completion_tokens": 2, "total_tokens": 15 }
}

POST /api/v1/gateway/chat/completions (Gemini — multi-provider routing)

The same endpoint routes to Gemini when the model belongs to a gemini connection. Note x-gateway-cost-usd is empty when the model has no configured price.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Content-Type: application/json" \
  --cookie "connect.sid=<session>" \
  -d '{"model":"gemini-2.5-flash-lite","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":10}'

Response headers (status 200):

x-gateway-request-id: ae680e52-1d75-4ceb-9447-b6d69312e00c
x-gateway-provider: gemini
x-gateway-model: gemini-2.5-flash-lite
x-gateway-cost-usd:
x-gateway-cache: miss

Response body (status 200):

{
  "id": "chatcmpl-eaa42e14-7bf5-4c5f-ad95-7ae5df61ccc8",
  "model": "gemini-2.5-flash-lite",
  "object": "chat.completion",
  "created": 1783147330,
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 7, "completion_tokens": 1, "total_tokens": 8 }
}

POST /api/v1/gateway/chat/completions (via virtual key)

Machine credential path — authenticate with a virtual key instead of a session. The virtual key was created via POST /gateway/keys; it is shown here redacted as agh_sk_...REDACTED (the real key was used and worked).

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer agh_sk_...REDACTED" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in one word."}],"max_tokens":10}'

Response headers (status 200):

x-gateway-request-id: be005d49-e3b1-4611-82c2-b568ddc9a083
x-gateway-provider: openai
x-gateway-model: gpt-4o-mini-2024-07-18
x-gateway-cost-usd: 0.00000315
x-gateway-cache: miss

Response body (status 200):

{
  "id": "chatcmpl-DxoN4uwS5rPd9w7RewNVjrDSx7nsD",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1783147350,
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 13, "completion_tokens": 2, "total_tokens": 15 }
}

POST /api/v1/gateway/chat/completions (streaming)

Set "stream": true. The response is text/event-stream: one OpenAI-compatible chat.completion.chunk per data: frame, terminated by data: [DONE]. x-gateway-cost-usd is intentionally omitted from stream headers (cost is not known until the stream ends).

curl
TypeScript
Python

curl --no-buffer -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Content-Type: application/json" \
  --cookie "connect.sid=<session>" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Count: one two three"}],"max_tokens":20,"stream":true}'

const stream = await hub.gateway.chat({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Count: one two three' }],
  maxTokens: 20,
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.delta.content ?? '');
}

stream = await hub.gateway.chat(
    'gpt-4o-mini',
    [{'role': 'user', 'content': 'Count: one two three'}],
    max_tokens=20,
    stream=True,
)
async for chunk in stream:
    print(chunk.delta.get('content', ''), end='')

Response headers (status 200):

Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
x-gateway-request-id: 9777fa85-2ace-4d96-8d97-acf8ac84be04
x-gateway-provider: openai
x-gateway-model: gpt-4o-mini

Response body (SSE stream):

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"One"},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" two"},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" three"},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"."},"finish_reason":null}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-9777fa85-2ace-4d96-8d97-acf8ac84be04","object":"chat.completion.chunk","created":1783147331,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":null}]}

data: [DONE]

Naming a trace across several calls

A trace can span many completions. Two headers name it, and the difference between them is whether you are giving an instruction or a fallback.

Header	On a new trace	On a trace that already has a name
`x-trace-name`	names it	overwrites it
`x-trace-name-if-unset`	names it	ignored

x-trace-name is last-explicit-write-wins on purpose: an agent that only works out what a run is on its second call can still name the trace after it. Omitting the header never resets a name.

x-trace-name-if-unset is for a fallback name — the one a client library reaches for when the caller expressed no preference. It fills in the default timestamp name and is otherwise ignored, so a call that merely joins a trace cannot rename it. Both SDKs send their runToolLoop default on this channel.

Both are free text, so send them percent-encoded (encodeURIComponent) — a raw non-ASCII header value is rejected before the request leaves most HTTP clients. The server decodes, and falls back to the raw value if it was not encoded.

A trace field the gateway could not use

The same values can travel in the body instead, as a trace object (traceId, parentSpanId, sessionId, capturePayloads, name, nameIfUnset, tags, metadata) or a span object (name, tags, metadata). A header always wins over the matching body field.

A field of the wrong type is dropped, never fatal. The completion is answered normally and x-gateway-context-ignored lists what was ignored, comma-separated, so a trace that arrives without your tags tells you why instead of leaving you to guess. The header is absent when everything was used.

tags must be an array of strings — the mistake below is sending one string:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi"}],"max_tokens":10,"trace":{"name":"refund-request","tags":"prod"}}' -i

Response (status 200) — the call runs, the trace is written, the name is kept and only tags is discarded:

x-gateway-context-ignored: trace.tags
x-gateway-model: openai/gpt-4o-mini
x-gateway-trace-id: 98b3ef57-aa27-4192-8fef-43742dbd34bf

{"id":"gen-1789540383-NI972N5mEVMmBs2PO3Jg","model":"openai/gpt-4o-mini","object":"chat.completion","created":1789540383,"choices":[{"index":0,"message":{"role":"assistant","content":"Hi there! How can I assist you today?"},"finish_reason":"length"}],"usage":{"prompt_tokens":9,"completion_tokens":10,"total_tokens":19}}

An x-trace-metadata or x-span-metadata header that is not a JSON object is listed under the header's own name rather than a trace. field.

Call A names the trace:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-trace-name: content-supervisor-flow" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}' -i

The response's x-gateway-trace-id is the trace to join. Reading it back:

{ "trace": { "name": "content-supervisor-flow", "spanCount": 1 } }

Call B joins it, sending only a fallback name:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-trace-id: b91a9709-d91a-4df5-b3fd-447918c8a552" \
  -H "x-trace-name-if-unset: runToolLoop" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi again"}]}'

{ "trace": { "name": "content-supervisor-flow", "spanCount": 2 } }

The span landed; the name did not move. Call C joins with an explicit name instead:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-trace-id: b91a9709-d91a-4df5-b3fd-447918c8a552" \
  -H "x-trace-name: refund-request" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi once more"}]}'

{ "trace": { "name": "refund-request", "spanCount": 3 } }

Both are also accepted on the request body as trace.name and trace.nameIfUnset, for clients that would rather not set headers.

Tools and prompt lineage on a completion

Three request fields decide which tools a completion sees and which prompt version its trace is attributed to. All three work on both the streaming and non-streaming path.

prompt — send a stored-prompt reference instead of messages and the gateway renders it, fills the model from the version's bound model when model is omitted, and auto-attaches the tools bound to that prompt alias. Definitions only: the model's tool_calls come back for you to run.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","prompt":{"name":"weather-brief","alias":"production","variables":{"city":"Lisbon"}}}'

Response (status 200) — the bound get_weather was attached without being named:

{
  "id": "chatcmpl-EEycj62PkeMuhzY0dJR3Q7yGnq6zC",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1787238337,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_pZdUAL1IodyxSgups7E1Sgbd",
            "type": "function",
            "function": { "name": "get_weather", "arguments": "{\"city\":\"Lisbon\"}" }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": { "prompt_tokens": 69, "completion_tokens": 15, "total_tokens": 84 }
}

tool_refs — name catalog tools by name when you send your own messages. Each ref follows an alias (default production) or pins one exact version; both on one ref is a 400. The gateway resolves them to schemas and merges them with any inline tools (a shared name is a 400). See tool resolve for the resolution rules.

prompt_version_id — for a caller that rendered the prompt itself and is sending the rendered messages. It stamps the request row and the llm span, so a client-side tool loop is as traceable back to its prompt as a server-rendered prompt reference is. It must be a prompt version in your own team.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [{"role":"user","content":"What'\''s the weather in Lisbon right now? Answer in one short sentence."}],
        "tool_refs": [{"name":"get_weather","alias":"production"}],
        "prompt_version_id": "d884bfc0-2f5f-4858-b119-682ff7caebbb"
      }'

Response (status 200):

{
  "id": "chatcmpl-EEyYkncDTfsaGVXWa5faoRUjKylu1",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1787238090,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "",
        "tool_calls": [
          {
            "id": "call_mpQNFOQ4A0ZQ0O8ydYAq2ZWv",
            "type": "function",
            "function": { "name": "get_weather", "arguments": "{\"city\":\"Lisbon\"}" }
          }
        ]
      },
      "finish_reason": "tool_calls"
    }
  ],
  "usage": { "prompt_tokens": 69, "completion_tokens": 15, "total_tokens": 84 }
}

An id that is not one of your team's prompt versions is rejected rather than dropped, so a caller asking for lineage finds out when it is not happening:

{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "prompt_version_id '11111111-1111-1111-1111-111111111111' is not a prompt version in this team."
  }
}

Sending prompt_version_id alongside prompt is allowed and ignored: the reference renders here, so the gateway already knows the exact version it used. Neither field ever reaches the provider.

Send `variables` with it

Add the values you rendered with as a top-level variables object. They are not re-rendered — you already rendered, and a second pass could rewrite model or user text that happens to contain {{ — but they are stored on the llm span as the run's replay lineage. An evaluation dataset example is those variables, so a client-rendered run that omits them can never seed one.

This is the shape to use when the stored version has no placeholders at all — a fixed system message with the user turn composed by your app:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [
          {"role": "system", "content": "You are a terse support agent. Answer in one sentence."},
          {"role": "user", "content": "How do I rotate an API key?"}
        ],
        "prompt_version_id": "792cf7a8-d061-4b2f-82a9-f9ad3200eb3b",
        "variables": {"question": "How do I rotate an API key?"}
      }'

Response (status 200):

{
  "id": "chatcmpl-ELkmu7PgavkoKukav8n0B6pGAXGVE",
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "created": 1788853448,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "To rotate an API key, generate a new key in your API management console and update your applications to use it while deactivating the old key after verification."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 31, "completion_tokens": 32, "total_tokens": 63 }
}

The span that call wrote carries both halves. From GET /traces/:id, the fields that matter here (the full span also carries timings, tokens and cost):

{
  "spanId": "90244880-496e-4021-90d6-8e4bd0aac682",
  "kind": "llm",
  "promptVersionId": "792cf7a8-d061-4b2f-82a9-f9ad3200eb3b",
  "payload": {
    "variables": { "question": "How do I rotate an API key?" }
  }
}

Feedback on that trace can then become a dataset example. Without variables the build skips the row with "no prompt variables were captured".

The SDKs fill all of this in for you — gateway.runPromptWithTools(rendered) / gateway.run_prompt_with_tools(rendered) derive the model, messages, refs and version id from one render result. See Call a prompt's tools from the SDK.

Error responses

Enforcement happens before any provider call. Observed shapes:

403 MODEL_NOT_ALLOWED — a virtual key with an allowedModels allow-list called a model outside it. See virtual keys.

{ "error": { "code": "MODEL_NOT_ALLOWED", "message": "Model 'gemini-2.5-flash-lite' is not allowed for this key." } }

402 BUDGET_EXCEEDED — the team-wide (or key-scoped) budget cap is reached. See budgets.

{ "error": { "code": "BUDGET_EXCEEDED", "message": "Team-wide budget exceeded." } }

When the provider is the one saying no

A rejection from the upstream provider comes back as 400 PROVIDER_BAD_REQUEST, carrying the provider's own message. That message is the part worth reading: it names what is actually wrong, which a status code cannot.

Two response_format requests that break different strict-mode rules, and therefore answer differently:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],
       "response_format":{"type":"json_schema","json_schema":{"name":"x","strict":true,
       "schema":{"type":"object","properties":{"a":{"type":"string"},"b":{"type":"string"}},
       "required":["a"],"additionalProperties":false}}}}'

Response (status 400) — b is in properties but missing from required:

{"error":{"code":"PROVIDER_BAD_REQUEST","message":"Provider rejected the request (400): Invalid schema for response_format 'x': In context=(), 'required' is required to be supplied and to be an array including every key in properties. Missing 'b'."}}

Now with required complete but additionalProperties dropped:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}],
       "response_format":{"type":"json_schema","json_schema":{"name":"x","strict":true,
       "schema":{"type":"object","properties":{"a":{"type":"string"},"b":{"type":"string"}},
       "required":["a","b"]}}}}'

Response (status 400):

{"error":{"code":"PROVIDER_BAD_REQUEST","message":"Provider rejected the request (400): Invalid schema for response_format 'x': In context=(), 'additionalProperties' is required to be supplied and to be false."}}

Which statuses forward the provider's message, and which do not. A 400, 404, 413, 422 or 429 describes something you can act on — your request, or the upstream's rate limit — so the provider's own message is passed through. A 401, 403 or any 5xx describes the connection instead: its credential or the provider's own internals, neither of which the caller of this gateway can do anything about, and a provider's 401 body can echo a masked copy of the key it was sent. Those are summarised:

{"error":{"code":"PROVIDER_ERROR","message":"Provider error (401): OpenAI request failed with status 401"}}

That one answers 502 PROVIDER_ERROR, and a provider timeout answers 504 PROVIDER_TIMEOUT. If you see a 401 here, the gateway connection's stored key is the thing to check, not the request.

A base URL the gateway will not call

A connection's config.base_url is checked again at the moment of the request, not only when you save it. A hostname cannot be resolved while the connection is being validated, so one that points at a private, loopback or cloud-metadata address is saved and then refused on every call:

curl -i -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"internal-model","messages":[{"role":"user","content":"hello"}]}'

Response (status 502):

{"error":{"code":"PROVIDER_ADDRESS_BLOCKED","message":"OpenAI request blocked: target address is not allowed. Check this connection's base URL — it must be a public address."}}

Nothing is sent upstream — the request is refused before the connection opens, so this is never a sign the provider is down. Point the connection at a public address.

If the model has fallbacks, the chain still runs and a working fallback answers the call normally — the response is an ordinary 200. The rescue is recorded on the call's trace rather than in the response: the llm span carries a warning of type model_fallback saying which model answered instead and why. You get the error above once every deployment has failed, and you get it even when a later one failed for some other reason, because a blocked address is the one fault in that list you can fix yourself.

Upstream rate limits

When the provider rate-limits or refuses on quota, the gateway answers 429 PROVIDER_RATE_LIMITED and forwards the provider's own Retry-After when it sends one. Streaming requests answer the same way, as JSON, because nothing has been written to the stream yet.

curl -i -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"free-glm","messages":[{"role":"user","content":"hi"}],"max_tokens":5}'

Response (status 429):

Retry-After: 5

{"error":{"code":"PROVIDER_RATE_LIMITED","message":"Provider rate limit (429): Provider returned error"}}

Do not confuse this with 429 RATE_LIMITED, which is your own virtual key's RPM/TPM window. The codes differ because the fixes differ: RATE_LIMITED is raised or lowered under Gateway → Virtual keys, while PROVIDER_RATE_LIMITED is the upstream account's limit and has to be resolved with that provider.

Retries and fallbacks (the `gateway` control field)

A completion body may carry a gateway object beside the OpenAI-shaped fields. It is stripped before the request reaches the provider, so the outgoing call stays OpenAI-compatible. It holds two keys:

maxRetries — how many extra times to call the same model on the same credential after a transient failure (network error, timeout, 5xx, 429). Integer 0–5, default 1. Ignored on a streaming call, where the first chunk commits the request to a model.
fallback — whether a failed model may hand off to the next model in its chain. Boolean, default true. false confines the call to the model named in model, so a failure is returned as an error rather than answered by a different model. It does not disable retries, which are the same model. Which models stand behind a name is set on the model itself as fallbackModelIds; this key only switches the chain off per call.

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"unstable-4o-mini","messages":[{"role":"user","content":"Say hello in five words."}],"gateway":{"maxRetries":2}}'

Response (status 200) — served by the fallback, because this model's credential is revoked:

{"id":"gen-1789322822-okaDxa8E8YEsYxi0WObY","model":"mistralai/mistral-small-3.2-24b-instruct","object":"chat.completion","created":1789322822,"choices":[{"index":0,"message":{"role":"assistant","content":"\"Hello, how are you today?\""},"finish_reason":"stop"}],"usage":{"prompt_tokens":9,"completion_tokens":9,"total_tokens":18}}

Outside 0–5 the call is rejected before it reaches a provider:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"unstable-4o-mini","messages":[{"role":"user","content":"hi"}],"gateway":{"maxRetries":9}}'

Response (status 400):

{"error":{"code":"VALIDATION_ERROR","message":"Number must be less than or equal to 5"}}

unstable-4o-mini has a revoked credential and mistral-small as its first fallback, so the default answers 200 from a different model. With fallback: false the same call returns the revoked credential's own failure:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"unstable-4o-mini","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5,"gateway":{"fallback":false}}'

Response (status 502):

{"error":{"code":"PROVIDER_ERROR","message":"Provider error (401): OpenAI request failed with status 401"}}

What a rescued call leaves behind

Nothing in a 200 says the model you asked for is the model that answered, so the gateway records the route on the call's llm span: attempts (provider calls across the whole chain) and trail (one entry per deployment tried, in order). Each trail entry names the model both ways — model is the public name you passed, upstreamModel is the provider's own — beside modelId and credentialId, carries its own attempts so a retried deployment is distinguishable from one that was only tried once, and on a failure carries error (the status code, as a string) and errorMessage (the provider's own sentence, truncated at 300 characters). A deployment that failed, was retried and then answered has no error of its own — it succeeded — so what it was failing on is kept as retriedAfter: {"error": "429", "errorMessage": "…"}.

A call that succeeded only because of a retry or a fallback also gets a warning object on that span — {"type": "model_fallback" | "provider_retry", "message": "…"}. The span stays ok, so the warning is what stops a primary model that fails every time from going unnoticed while its fallback keeps answering. The trace list exposes it as hasWarning, and has_warning=true filters on it.

The response header x-gateway-trace-id gives you the trace to read:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_VIRTUAL_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"unstable-4o-mini","messages":[{"role":"user","content":"Say hi"}],"max_tokens":10}'

Response (status 200) — served by the fallback, with these headers:

x-gateway-request-id: b1ab662c-93bb-41c5-9bc1-88c5f3003cc0
x-gateway-model: mistralai/mistral-small-3.2-24b-instruct
x-gateway-trace-id: ec60bcfb-ede3-4160-86bf-dc47f3966025

{"id":"gen-1789330077-IA7FaZdBoBo7XZFI5mBa","model":"mistralai/mistral-small-3.2-24b-instruct","object":"chat.completion","created":1789330077,"choices":[{"index":0,"message":{"role":"assistant","content":"Hi there! 😊 How can I help"},"finish_reason":"length"}],"usage":{"prompt_tokens":5,"completion_tokens":10,"total_tokens":15}}

The full trail and warning that call wrote are shown under GET /api/v1/traces/:id. See also Retries and fallbacks.

Prompt caching on Anthropic models

Anthropic models cache a prompt only when the request marks the end of the part to cache. A message carries the marker as "cache_control": {"type": "ephemeral"} beside its string content, either in raw messages or in a stored prompt version (see marking a message for caching). The gateway sends it on to Anthropic models only, on an Anthropic connection or OpenRouter, and drops it for every other model.

usage then reports the cache. cache_write_tokens counts tokens written to the cache and cached_tokens counts tokens read from it. Both are part of prompt_tokens, not added to it. The recorded cost bills a write at 1.25x and a read at 0.1x the model's input price, the rates Anthropic charges. Anthropic caches nothing below a minimum prompt length that depends on the model (Anthropic's limits). A shorter marked prompt runs uncached, and usage then has neither cache field.

The two calls below render the version from the link above. That version's marked system message is about 9,000 tokens long. They differ only in the question:

curl -X POST $ACRUXCORE_BASE_URL/gateway/chat/completions \
  -H "Authorization: Bearer $ACRUXCORE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt":{"name":"lamp-support","alias":"production","variables":{"question":"How long does SKU 1004 take to ship?"}},"max_tokens":60}'

{
  "id": "gen-1791623777-oHICdErEf6Srb6f0uExK",
  "model": "anthropic/claude-haiku-4.5",
  "object": "chat.completion",
  "created": 1791623777,
  "choices": [{"index":0,"message":{"role":"assistant","content":"SKU 1004 ships in **8 days**."},"finish_reason":"stop"}],
  "usage": {"prompt_tokens": 8999, "completion_tokens": 16, "total_tokens": 9015, "cache_write_tokens": 8980}
}

The second call, within five minutes, with "question":"What is the warranty on SKU 1200?":

{
  "id": "gen-1791623785-Q1rTR7GoqTg7S2vdlPtV",
  "model": "anthropic/claude-haiku-4.5",
  "object": "chat.completion",
  "created": 1791623785,
  "choices": [{"index":0,"message":{"role":"assistant","content":"SKU 1200 has a **2 years warranty**."},"finish_reason":"stop"}],
  "usage": {"prompt_tokens": 8997, "completion_tokens": 17, "total_tokens": 9014, "cached_tokens": 8980}
}

At $1 per million input tokens and $5 per million output tokens, the gateway recorded $0.011324 for the first call and $0.001000 for the second. Without the marker, each call costs $0.009084.

Response Cache (G6)

Per-team exact-match response cache. Opt-in per virtual key via cacheTtlSeconds; only requests with temperature: 0 are cached. A cache hit returns the stored body at zero cost with header x-gateway-cache: hit and does not increment any budget. Bypass per call with request header x-gateway-cache: no-store.

DELETE /api/v1/gateway/cache

Flush the calling team's response cache (owner/admin; editors/viewers 403). Returns the number of rows removed (0 when nothing was cached).

curl -X DELETE $ACRUXCORE_BASE_URL/gateway/cache \
  --cookie "connect.sid=<owner/admin session>"

Response (status 200):

{ "deleted": 0 }

POST /api/v1/gateway/chat/completions (non-streaming)​

POST /api/v1/gateway/chat/completions (Gemini — multi-provider routing)​

POST /api/v1/gateway/chat/completions (via virtual key)​

POST /api/v1/gateway/chat/completions (streaming)​

Naming a trace across several calls​

A trace field the gateway could not use​

Tools and prompt lineage on a completion​

Send variables with it​

Error responses​

When the provider is the one saying no​

A base URL the gateway will not call​

Upstream rate limits​

Retries and fallbacks (the gateway control field)​

What a rescued call leaves behind​

Prompt caching on Anthropic models​

Response Cache (G6)​

DELETE /api/v1/gateway/cache​

POST /api/v1/gateway/chat/completions (non-streaming)

POST /api/v1/gateway/chat/completions (Gemini — multi-provider routing)

POST /api/v1/gateway/chat/completions (via virtual key)

POST /api/v1/gateway/chat/completions (streaming)

Naming a trace across several calls

A trace field the gateway could not use

Tools and prompt lineage on a completion

Send `variables` with it

Error responses

When the provider is the one saying no

A base URL the gateway will not call

Upstream rate limits

Retries and fallbacks (the `gateway` control field)

What a rescued call leaves behind

Prompt caching on Anthropic models

Response Cache (G6)

DELETE /api/v1/gateway/cache