Models and Routing
Models are the LLMs your applications call through the Gateway, from any provider your organization has connected. Your organization approves the models once, and each application calls the ones it needs by model ID, with fallback models and inference defaults set outside its code.
Required role
Applies to
Why use it
When each team calls providers directly, nobody can list which models are approved, and every model change is a code change. Models puts that choice in one place, as part of the control and choice described in the Introduction.
- One approved list. An application can call only the models enabled for both the organization and the application. The Gateway checks this on every request.
- Switch models without a migration. Every model is called through the same API and named by a model ID, so moving to another model or provider is a one-line change.
- Give each application the right models. Restrict an application to a smaller selection from the organization's approved models.
- Requests survive provider failures. When a model fails, the Gateway retries the request with the application's fallback models.
- Manage inference defaults centrally. Set an application's system prompt and response length cap without repeating them in every request.
How it works
The organization's model catalog, in Company setup, lists the models your organization can use: the models of each provider you have a provider key for, and the models of your custom providers. For how organization and application settings fit together, see Organizations and Projects.
Which models an application can call is set in two places:
- The organization makes a model available. An Organization Admin enables the model in the organization's catalog. No application can call a model that isn't enabled there.
- The application narrows the list. An admin of the application selects which of the organization's models it can call. New applications start with no models enabled. An application explicitly configured to inherit uses the organization's enabled list, subject to provider scope and data governance rules.
This is the rule used across Optiak: the organization enables, the application narrows.
Choosing the model is one step of an LLM request through the Gateway, after policies, budgets and guardrails, and before the prompt is built and sent. See How the Gateway handles a request for every stage.
This page covers the highlighted steps: the Gateway checks that the requested model is available to the application and meets your data governance rules, then sends the request to the provider and tries the fallback models if it fails.
Organization model catalog
The catalog shows each model's provider, capabilities, data residency, region, context window, and input and output prices per million tokens. Badges identify no-training support and Smart Router eligibility. Expand a row to see its description, model ID, max input and output tokens, and full capabilities.
You can search by model name, filter by provider, data residency, region, capabilities or enabled status, and sort by fields such as context window or price to compare models before enabling them.
Model IDs
A model ID names the model a request calls, in the model field. To find the IDs an application can use, call GET /models with one of its application API keys. The response lists every model the key can call, including custom provider models, and starts with optiak/smart when Smart Router is on for the application. See List models in the API reference.
You can name a built-in model at three levels of detail; the more you name, the less the Gateway chooses for you. Custom provider models and Smart Router have their own forms:
| Form | Example | What the Gateway does |
|---|---|---|
model | gpt-5.6-luna | Calls the model at any provider enabled for the application that offers it. |
provider/model | openai/gpt-5.6-luna | Calls the model at that provider and chooses the region, preferring global. |
provider/region/model | openai/global/gpt-5.6-luna | Calls the model at that provider, in that region: eu, us or global. |
custom-provider/model | my-provider/my-model | Calls a model of one of your custom providers, named by the provider's identifier and the model's ID. |
optiak/smart | optiak/smart | Lets Smart Router choose the model for each request. The short form smart also works. |
For each model, GET /models also returns its context window, capabilities, data residency and price; see Using the API.
Inference settings
Inference settings are set per application and apply to every LLM request it sends: fallback models for when a model fails, and inference defaults that shape each request. In the console, open the application, then Configuration, then Inference.
Fallback models
Fallback models are the models the Gateway tries when the requested model fails. They are an ordered list, and the Gateway tries them from first to last. Add them under Fallback models and drag them to change the order. They must be models that the application can call.
| Request outcome | Fallback |
|---|---|
| The requested built-in model isn't enabled for the application | Uses fallback model |
| The provider fails, rejects the provider key or doesn't have the model | Uses fallback model |
| The request exceeds the model's context window | Uses fallback model |
| The request is invalid or uses a parameter the model doesn't support | Does not use fallback model |
| The request times out | Does not use fallback model |
| A streamed response has already started | Does not use fallback model |
In order to use fallbacks, you must enable them in the application's configuration. Data governance rules apply to fallback models too, and a request that names a region falls back only to models in that region. Each attempt appears in the request's trace, in the application's Requests tab.
Inference defaults
Inference defaults shape every request before it reaches the model. Today that is the application's system prompt.
System prompt
The application's system prompt is combined with the instructions a request carries: its system and developer messages, and instructions in the Responses API. Application prompt behavior sets how:
| Behavior | Result |
|---|---|
| Append (default) | The application's system prompt follows the request's instructions. |
| Prepend | The application's system prompt comes before the request's instructions. |
| Replace | Only the application's system prompt is sent. The request's instructions are discarded. |
A request with no instructions gets the application's system prompt as it is.
Set it up
An Organization Admin enables models for the organization, then each application enables its own. For who can do each step, see Members and Access.
In the organization
In the console, open Company setup, then Models. Find a model in the model catalog and turn on Enabled. If a provider's models are missing, add its provider key first; see Authentication.
In the application
In the console, open the application, then Configuration, then Models, and turn on the models the application can call. Then set its fallback models and inference defaults under Inference; see Inference settings.
An application can call a model only when it's enabled in Company setup for the organization and, if the application has its own selection, in the application's Models. If a request is rejected or a model is missing, check both.
Custom provider models
Creating a custom provider and adding its models does not enable those models for requests. Enable each model in Company setup → Models, then select it in the application's Configuration → Models. An application configured to inherit the organization's enabled models does not need a separate selection.
Custom models appear in the organization catalog even when the provider's Scope is limited to selected applications or projects. Scope controls which applications can use the provider's credentials. Enabling a model for the organization does not expand that scope.
For example, a provider scoped to Application A has a model named my-provider/my-model. That model appears in the organization catalog for approval. Once enabled there, it becomes available for Application A to select. Application B, outside the provider's scope, cannot select or call it.
Token limits
A custom model can declare a Context window and Max output tokens. Both are optional: leave a limit empty when the provider doesn't publish it, and Optiak won't check it. The context window covers prompt plus response tokens and is often listed as the context length. Use the limit your endpoint serves, which can be lower than the model's maximum. Max output tokens must be lower than the context window; if the provider publishes only a context length, set just the context window.
| Limit set | Effect |
|---|---|
| Context window | A request whose max_completion_tokens (Chat Completions) or max_output_tokens (Responses) is equal to or above the window is rejected with 400. |
| Max output tokens | A request asking for more output tokens than this is rejected with 400. |
| Context window | With Context Budgeting on, the prompt must fit in the model's context budget. An oversized prompt is trimmed or rejected with 422. |
| None | No local checks. The upstream endpoint decides. |
If the application has fallback models, Optiak tries them before returning one of these errors. Optiak estimates prompt sizes for custom models with an approximate tokenizer, so a prompt close to a limit may be counted slightly differently from the upstream model's own count.
Limits and behavior
- A request is rejected, without fallback, when its model ID doesn't exist or the model breaks your organization's data governance rules. See Using the API for the errors.
- When you turn off a model for the organization, no application can call it any more, as a requested model or as a fallback. The console will ask you to confirm the change if it would affect any application.
- Models from custom providers can be primary or fallback models, and can be mixed with built-in models in the fallback order. The provider must be active, available to the application, and meet its data governance rules. Optiak does not have complete capability metadata for custom models; ensure they support the request parameters and content.
- A request to a custom provider model that isn't enabled for the application, or whose provider is inactive, is rejected without fallback.
- Requests that name a region do not fall back to custom provider models.
- Requests with provider-owned conversation state or provider-specific tools/history do not switch to or from a custom provider for fallback.
- The catalog shows the retirement date when a provider announces that it will retire a model.
Related
- Smart Router: let Optiak choose the model for each request.
- Authentication: provider keys and custom providers.
- Data Governance: the organization rules that limit which models can be used.
- Using the API: endpoints, parameters and errors.