Playbook 10 · Self-hosted

Run against your own LLM gateway

Goal: every model call leaves through one gateway that you run inside your own network, so prompts and customer data never touch a vendor endpoint you do not control. Zolva speaks the OpenAI chat-completions protocol, so any OpenAI-compatible gateway, LiteLLM, vLLM, or a proxy in front of Bedrock or Azure OpenAI, is a one-line config change, not a new adapter.

The shape: Zolva's OpenAI adapter posts to base_url + /chat/completions with a bearer token. Set base_url to your gateway and the bearer to the gateway's key instead of a vendor key, and the request never leaves your VPC. The gateway decides which real model serves it.

Prerequisites #

• An OpenAI-compatible gateway reachable on your network. For example, a LiteLLM proxy on http://llm-gateway.internal:4000, or a vLLM server on http://llm-gateway.internal:8000.
• The gateway's key (LiteLLM's master_key, or whatever bearer your proxy expects). Zolva reads it from OPENAI_API_KEY.
• A model name the gateway routes, such as the alias you configured in the proxy.
• Python ≥ 3.11 and pip install zolva.
• Environment variables, never inline config: LLM_GATEWAY_URL and OPENAI_API_KEY (the gateway key).

Step 1 · Stand up the gateway #

This is your infrastructure, not Zolva's; the two common paths, for reference:

# serves an OpenAI-compatible API on http://0.0.0.0:4000
# clients authenticate with the proxy master key, not the upstream provider key
litellm --config litellm.config.yaml

# exposes /v1/chat/completions on http://0.0.0.0:8000
vllm serve your-org/your-model --api-key "$GATEWAY_KEY"

Either way the base URL Zolva needs ends in /v1: LiteLLM's http://host:4000/v1 or vLLM's http://host:8000/v1.

Step 2 · Point Zolva at it #

One line in the agent's model block. Keep provider: openai (that selects the chat-completions protocol, not the vendor) and set base_url.

name: support-agent
instructions: support.md
model:
  provider: openai            # the wire protocol, not the vendor
  name: gpt-5.6-sol           # a model alias your gateway routes
  base_url: ${ENV:LLM_GATEWAY_URL}   # e.g. http://llm-gateway.internal:4000/v1
  timeout: 30
tools: [lookup_balance, recent_transactions]
handoffs: [human-escalation]

Set the two environment variables and validate:

export LLM_GATEWAY_URL="http://llm-gateway.internal:4000/v1"
export OPENAI_API_KEY="$LITELLM_MASTER_KEY"   # the gateway's key, not a vendor key
zolva validate agents/

The loader rejects any inline key, so the gateway credential stays in the environment, never in the YAML. Judge-backed rules and evals take the same base_url, so guardrails and the judge also run entirely through your gateway.

Step 3 · Timeouts and retries #

A gateway is one more hop that can be briefly busy. Zolva retries transient 429 and 5xx responses with bounded backoff, honouring Retry-After, before it gives up, so a momentary blip becomes a short wait rather than an escalated customer. The timeout above bounds each attempt, and each distinct base_url gets its own connection pool.

When the gateway is genuinely down, the call degrades to human handover, never to silence: the customer is escalated with the reason on the audit chain. Test that path with an adversarial synthetic that runs while the gateway is stopped.

Residency notes #

• One egress point: with the gateway inside your VPC, model traffic leaves through infrastructure you audit and log, not through each agent process.
• Vendor swap without code change: the gateway decides the real model, so moving from a hosted model to an open-weights vLLM deployment, or between clouds, never touches agent config beyond the alias.
• Keys stay put: agents hold the gateway key only; upstream provider keys live in the gateway, in one place, rotated independently.
• Protocol, not vendor: provider: openai means the OpenAI wire format. Anything that speaks it, including Azure OpenAI and Bedrock behind a compatible proxy, works through the same setting.

← All playbooks