Local LLMs (Ollama, vLLM, llama.cpp)
Run AI models on your own hardware and route them through Clevername. Model inference stays on your own hardware. Clevername still governs every call — CleverGuard scans content for security, and guardrails, drift detection, and audit logging apply the same as with cloud models.
What is BYOL?
Bring Your Own LLM (BYOL)lets you connect local AI models to Clevername. Instead of sending prompts to OpenAI or Anthropic, your model inference stays on your own hardware. Clevername still governs the agent — guardrail profiles, tool restrictions, drift detection, and audit logging all work the same way regardless of where the model runs.
Supported local LLM platforms
Easiest setup. Runs Llama, Mistral, Qwen, Gemma, and other open models. One-command install.
High-throughput serving for production deployments. GPU-optimized with PagedAttention.
Lightweight C++ inference. Runs on CPU or Metal (Apple Silicon). GGUF model format.
Any OpenAI-compatible API endpoint works — including LM Studio, Jan, LocalAI, and text-generation-webui.
How it works
Clevername connects to your local model via the LLM Relay— a lightweight WebSocket or HTTP bridge that forwards requests between Clevername and your local server. Your model endpoint never needs to be publicly accessible. The relay handles all communication securely through an authenticated tunnel.
Install and start Ollama
If you don't have Ollama yet, install it from ollama.com. Then pull a model and start the server:
ollama pull llama3.2 ollama serve
Ollama runs on http://localhost:11434 by default.
Register in Clevername
Go to Dashboard → Keys → Add Key and select Ollama as the provider. Enter your Ollama endpoint URL (usually http://localhost:11434). No API key is needed for local Ollama.
Assign the local model to an agent
In AI Company → Agents, open any agent and pick a local model from the model selector (local models appear under the “Local” section). Once bound, every governed call from that agent — through its cn-*gateway token — is routed to Ollama instead of a cloud provider, while still passing through CleverGuard scanning, the agent's guardrail profile, and audit logging.
Bind the local key to specific agents (optional)
From Settings → Keys, open the Ollama entry and use the agent-binding panel to scope it to specific agents. This works the same way as cloud BYOK keys — you can store multiple Ollama endpoints, label them, and isolate which agents can reach which endpoint.
What gets governed
Local models receive the same governance as cloud models. Clevername enforces your policies regardless of where the model runs.
Use cases
Route large batch jobs through local Qwen or Llama models during off-hours to save on API costs. Clevername still audits every interaction.
For regulated industries where data cannot leave the network. Local models + Clevername governance = compliant AI agents. Fully air-gapped operation requires self-hosted CleverGuard, since the managed service transmits content to Clevername's scanner.
Test new open-source models (Llama 3.2, Qwen 2.5, Mistral) through the same governance layer before committing to a provider.
Run routine classification and extraction tasks on local Haiku-class models. Reserve cloud models for complex reasoning.