Chat

A client talks to the gateway in whichever provider dialect it already speaks. The gateway routes to whatever registry the application’s strategy picks, and translates in both directions. An Anthropic Messages request can land on an OpenAI registry and come back as an Anthropic response, streaming included. That independence is the point: moving traffic between providers becomes a routing change, not a client release.

Shapes the gateway accepts

Path after the application slug	Client dialect
`/v1/chat/completions`	OpenAI Chat Completions
`/v1/responses`	OpenAI Responses
`/v1/messages`	Anthropic Messages
`/v2/chat`	Cohere Chat v2
`/v1beta/models/{model}:generateContent` · `:streamGenerateContent`	Gemini
`/{v1\|v1beta1}/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent` · `:streamGenerateContent`	Vertex AI
`/{v1\|v1beta1}/publishers/google/models/{model}:generateContent` · `:streamGenerateContent`	Vertex AI express mode

The Vertex paths are what Google’s SDKs send with vertexai=True: the full path with a service account, the short one in express mode with an API key. Project and location in the path are ignored; the upstream call uses the registry’s own. Only these exact paths exist; a near miss such as /v2/chat/completions is a 404, not a guess. An AWS SDK talking to Amazon Bedrock does not need any of these dialects: the native Bedrock Runtime paths (/model/{modelId}/converse and its siblings) are accepted as well, and relayed to a Bedrock registry without translation. See Amazon Bedrock Runtime.

from openai import OpenAI

client = OpenAI(base_url="https://<llm-host>/<application-slug>/v1", api_key="<application key>")

client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Explain zero trust in one sentence."}],
)

import OpenAI from "openai"

const client = new OpenAI({ baseURL: "https://<llm-host>/<application-slug>/v1", apiKey: "<application key>" })

await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Explain zero trust in one sentence." }],
})

curl -X POST "https://<llm-host>/<application-slug>/v1/chat/completions" \
  -H "X-AG-API-Key: <application key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Explain zero trust in one sentence."}]}'

The key may travel as X-AG-API-Key, x-api-key, x-goog-api-key, or Authorization: Bearer, which is what lets stock SDKs point at the gateway unchanged. Copy the exact form from the application’s Connect tab; it already reflects the strategy and the deployment.

The model field

model is where the client meets routing: auto hands the choice to the gateway, a short name resolves to the first registry that serves it, @provider/model prefers a provider. Gemini puts the model in the path rather than the body; the rule is the same. Model restrictions on the application apply before any provider is called. The response carries X-Selected-Provider and X-Selected-Model, which is the first thing to read when a request went somewhere you did not expect.

What translation costs you

Because the dialect and the provider can differ, the gateway normalises a few things once it knows which model it is talking to:

Behaviour	What happens
Output cap clamp	A requested `max_tokens` above the model’s limit is clamped to it; the response says so in `X-Max-Tokens-Clamped`.
Context preflight	A prompt that cannot fit the model’s window is rejected with `400 context_length_exceeded` before any upstream call. Not applied where the window is unknown — OpenAI-compatible endpoints included.
Errors in your dialect	Upstream errors come back in the client’s envelope, mid-stream too. OpenAI-shaped clients get the upstream body as is.
Prompt-cache markers	Carried across the translation: `cache_control` from an OpenAI client reaches Claude as `cache_control`, or Bedrock as `cachePoint`. A target that takes none gets none, and the request still succeeds. Same-format traffic is untouched. See Prompt caching.
Provider-only features	An Anthropic server tool declared by `type` alone (web search, say) is dropped when routed to OpenAI or Gemini. Same-provider traffic is untouched.
Responses state	`previous_response_id`, `store` and server-side conversation state only work when the registry is OpenAI. On translated routes send the full `input` history each turn.
Stateful Responses	`previous_response_id` and `store: true` need an OpenAI registry. On a route that translates `/v1/responses` (Azure Foundry and other non-Responses registries) the gateway rejects them with `400 invalid_request_error` (`param` names the field) before any upstream call, so no failover happens. `store: false` is accepted.

The last three rows are the honest limit of translation: fields that exist in one provider’s API and nowhere else cannot be carried across. Everything in the common core — messages, tools with names, temperature, streaming — is.

Streaming

Set "stream": true on OpenAI, Anthropic, Cohere and Responses; on Gemini call :streamGenerateContent or add ?alt=sse. Events are re-encoded to the client’s format. If an upstream stream dies after the response has started, the HTTP status is already 200 and cannot change. The gateway emits a final error event in the client’s format instead — upstream_error for OpenAI-shaped clients, an event: error for Anthropic, an error object for Gemini — so a client that handles errors mid-stream sees one, and one that does not sees a truncated response rather than a hang.

Start here

Concepts

LLM Gateway

MCP Gateway

Policies

Operate

Shapes the gateway accepts

The model field

What translation costs you

Streaming

​Shapes the gateway accepts

​The model field

​What translation costs you

​Streaming

Shapes the gateway accepts

The model field

What translation costs you

Streaming