Cameleer / Use cases / Model calls without a gateway

LLM calls, in process

An LLM gateway that is not another hop

Q: Can a retried model call run a tool twice?

No. Tools run in the ai-tool envelope after the model answers, with their own idempotency, so a retried, cached or coalesced model turn never repeats a tool call.

Q: Is the response cache safe to turn on?

responseCache and coalesce are off by default, because two users who send the same prompt get the same answer. Turn them on for classification, extraction and other deterministic calls. A cached or coalesced answer reports zero tokens, so the usage headers show what was paid.

Retries, fallback models, prompt caching and a response cache usually mean a hosted gateway or a sidecar between your code and the model. In Cameleer they are options on the model endpoint, in the same process as the routes and the tools, over pooled keep-alive connections.

23model providers out of the box, plus any OpenAI-compatible server through baseUrl

0extra network hops: the call goes from the route to the provider

1 runper tool call, even when the model turn is retried, cached or coalesced

same optionson every model component, the tool loop and the agent step

One endpoint URI

A model call with an exact response cache and a fallback to another provider when the first still fails after its retries:

- route:
    from:
      uri: direct:ask
      steps:
        - to: "openai:chat-completion?model=gpt-5.4-mini&tags=support&responseCache=exact&fallbackModels=anthropic:messages?model=claude-sonnet-4-5"

A provider preset sets the base URL and the environment variable the key is read from:

        - to: "llm:chat-completion?provider=groq&model=llama-3.3-70b-versatile&tags=support"

tags=support offers the model the ai-tool routes with that tag and runs the tool loop; leave it out for a plain chat call. Inside a URI option, write a fallback's own & as %26.

Reliability and cost options

option	default	what it does
`maxRetries`	2	retries 408, 409, 425, 429, 5xx and 529 answers, timeouts and resets, with exponential backoff
`retryDelay`	500 ms	the first backoff; it doubles each retry, up to 10 s
`fallbackModels`		model endpoint URIs tried in order when the model still fails; they can be other providers
`responseCache`	off	`exact`: an identical request (model, messages, tools, system prompt) is answered from memory
`responseCacheTtl`, `responseCacheSize`	5 m, 1000	how long and how many answers are kept
`coalesce`	false	identical requests in flight at the same time share one model call
`promptCaching`	per component	`anthropic` adds `cache_control` breakpoints by default; `aws-bedrock` adds Converse `cachePoint`s with `promptCaching=true`

What they guarantee

A retried, cached or coalesced model turn never repeats a tool call: tools run in the ai-tool envelope after the model answers, with their own idempotency.
A cached or coalesced answer reports zero tokens, so the usage headers show what was paid.
A streamed call is not retried once text has been sent to the chunk processor.
responseCache and coalesce are off by default: two users who send the same prompt would get the same answer. Turn them on for classification, extraction and other deterministic calls.

Prompt caching

Providers bill cached input at a fraction of the normal price, but only for a prefix that is byte for byte the same as last time.
Cameleer assembles the prompt itself (tools in a fixed order, then the system prompt, then the history), so the prefix stays stable from turn to turn.
openai and the presets use the provider's automatic prefix caching; gemini and google-vertexai use Gemini's implicit caching.
CamelAiCacheReadTokens and CamelAiCacheWriteTokens report cached input tokens.

Providers

Native components

openai, llm: Chat Completions, the Responses API, embeddings; streaming in the tool loop; apiFormat=ollama for Ollama's /api/chat.
anthropic: the Messages API, with automatic prompt caching.
gemini: the Gemini API with AI Studio keys; thought signatures kept during tool calls.
azure-openai (Entra ID or keys), aws-bedrock (Converse, guardrails), google-vertexai (service accounts).

`provider=` presets on `openai` / `llm`

openai, gemini, mistral, deepseek, xai, groq, cerebras, together, fireworks, openrouter, perplexity, huggingface, dashscope (Qwen), moonshot (Kimi), cohere, nvidia (NIM), ollama, vllm, lmstudio, llamacpp.

Per provider, the component sends max_completion_tokens to OpenAI's o-series and GPT-5 models, sends DeepSeek's, Qwen's and Kimi's reasoning_content back during a tool loop, and generates missing tool call ids.

The presets were checked against each provider's documentation, not against the live services; the tests run against mock servers. Full table with base URLs and key variables: model providers.

Why in the same process

A separate gateway

A hosted gateway adds a network hop to every model call; a sidecar adds a local hop and another process to run.

Cameleer

Model calls go straight from the route to the provider over pooled keep-alive connections: no TCP or TLS handshake after the first call.
The cache and coalescing live in the same memory as the routes. The tool loop, approvals and the audit log are in the same process, so a retried model turn cannot run a tool twice.

No latency figures are published for model calls yet. The agent step uses the same options, and record / replay let cameleer test check an agent route without calling a model.

Common questions

Which model providers does Cameleer support?

23 out of the box: native components for OpenAI (Chat Completions and the Responses API), Ollama's /api/chat, Anthropic, Gemini, Azure OpenAI, AWS Bedrock Converse and Gemini on Vertex AI, plus provider= presets for OpenAI-compatible services such as Groq, Mistral, DeepSeek, xAI, OpenRouter, Together, Fireworks, Cerebras, Perplexity, Hugging Face, Qwen, Kimi, Cohere, NVIDIA NIM, vLLM, LM Studio and llama.cpp. Any other OpenAI-compatible server works with baseUrl.

Can a retried model call run a tool twice?

No. Tools run in the ai-tool envelope after the model answers, with their own idempotency, so a retried, cached or coalesced model turn never repeats a tool call.

Is the response cache safe to turn on?

responseCache and coalesce are off by default, because two users who send the same prompt get the same answer. Turn them on for classification, extraction and other deterministic calls. A cached or coalesced answer reports zero tokens, so the usage headers show what was paid.

Were the presets tested against the live services?

No. The presets were checked against each provider's documentation; the tests run against mock servers.

Try it

The tested fallback example fails over from a hosted model to a local OpenAI-compatible one (Ollama, vLLM) with the failover load balancer EIP; fallbackModels does the same as one option on the endpoint. Check it, then run it with the command in the file's header.

git clone https://github.com/pmsudhi/cameleer && cd cameleer/examples/agents
docker run --rm -v "$PWD:/routes:ro" pmsudhi/cameleer validate /routes/llm-multi-model-fallback.camel.yaml

Model providers guide The agent step Routes as MCP tools