Model providers and call optimizations

Cameleer talks to 23 model providers out of the box: seven native components and 19 OpenAI-compatible services through provider= presets, plus any other OpenAI-compatible server through baseUrl. Around each model call it adds what usually needs a separate gateway: retries, fallback models, prompt caching, an exact response cache and coalescing of identical requests. These run in the same process as the routes, so they add no network hop.

Cargo feature: saas (openai, llm, anthropic, gemini) and cloud (azure-openai, aws-bedrock, google-vertexai) · Apache Camel: the openai/llm options mirror Camel 4.23; presets, caching and the reliability options are Cameleer extensions

Native components

All of them are models for the agent step and take the same tool options.

Provider presets

provider=NAME on openai: / llm: sets the base URL and the environment variable the key is read from. baseUrl and apiKey still override both.

- route:
    from:
      uri: direct:ask
      steps:
        - to: "llm:chat-completion?provider=groq&model=llama-3.3-70b-versatile&tags=support"

Any other OpenAI-compatible server works with baseUrl. Azure OpenAI's v1 API and Bedrock's OpenAI-compatible endpoint work the same way; the native components add Entra ID, SigV4 and guardrails.

What the component adjusts per provider:

• OpenAI's o-series and GPT-5 models get max_completion_tokens instead of max_tokens.
• DeepSeek, Qwen and Kimi return reasoning_content with tool calls and want it back on the next turn; Cameleer sends it back.
• Missing tool call ids (some Ollama versions) are generated.

The presets were checked against each provider's documentation, not against the live services; the tests run against mocks.

Reliability and cost

Every model component takes these options. They apply to the tool loop and the agent step.

- route:
    from:
      uri: direct:ask
      steps:
        - to: "openai:chat-completion?model=gpt-5.4-mini&tags=support&responseCache=exact&fallbackModels=anthropic:messages?model=claude-sonnet-4-5"

What these guarantee:

• A retried, cached or coalesced model turn never repeats a tool call: tools run in the ai-tool envelope after the model answers, with their own idempotency.
• A cached or coalesced answer reports zero tokens, so the usage headers show what was paid.
• A streamed call is not retried once text has been sent to the chunk processor.
• responseCache and coalesce are off by default: two users who send the same prompt get the same answer. Turn them on for classification, extraction and other deterministic calls.

Prompt caching

Providers bill cached input at a fraction of the normal price, but only for a prefix that is byte-for-byte the same as last time. Cameleer assembles the prompt itself (tools in a fixed order, then the system prompt, then the history), so the prefix is stable from turn to turn.

CamelAiCacheReadTokens and CamelAiCacheWriteTokens report the cached input tokens of a call when there were any. A prompt shorter than the provider's minimum (about 1024 tokens for most models) is not cached and costs nothing extra.

Why in-process

A hosted gateway adds a network hop to every model call; a sidecar gateway adds a local hop and another process. Cameleer's model calls go straight from the route to the provider over pooled keep-alive connections (no TCP and TLS handshake after the first call), and the cache and coalescing live in the same memory as the routes. The tool loop, approvals and the audit log are in the same process too, so a retried model turn can't run a tool twice.

Related

• The agent step and tool calling
• ai-tool: the envelope around every tool call
• MCP server and MCP client

Found a problem?

If something on this page doesn't match what Cameleer does, please open an issue, or fork the repository, fix it, add a test, run cargo test --workspace and open a pull request.