A local model as your agent's brain

You can point a KDCube agent at a model running on your own machine — served by Ollama — and let a user pick it for a single conversation. No platform code changes: the existing provider: custom role path already speaks a small protocol; a standalone models gateway on the host translates it to Ollama.

The path

The agent never learns "Ollama." It asks its model router for a custom client, which posts to an endpoint. That endpoint is the gateway; the gateway speaks Ollama.

The agent asks for a custom client; the gateway is the only new piece, and it lives on the host.

Configured by descriptor, not code

Turning it on is a descriptor edit. services.llm.custom is a reserved platform-interpreted app property — the platform applies its endpoint, model_name, and num_ctx to the model router exactly the way it applies role_models. The gateway key, when set, is an ordinary bundle secret (services.llm.custom.api_key), resolved at the turn door.

services:
  llm:
    custom:
      endpoint: http://host.docker.internal:11500/generate
      model_name: qwen3.6:35b
      num_ctx: 65536

One gateway can serve every model you have pulled: the client transmits the picked model name and the gateway routes it, so each row selects its own weights.

Offer it as a pick

The endpoint above is the plumbing. To make the local model an option a user chooses, the agent declares it in the same place it declares any model or capability — its react config. A supported_models row is a pick offer; an instruction_profiles option is another.

react:
  default_agent:
    supported_models:
      - { model: qwen3.6:35b, provider: custom, label: Qwen3.6 35B (local) }
      - { model: qwen3:8b,    provider: custom, label: Qwen3 8B (local, fast) }
    instruction_profiles:
      default: full
      options:
        - { id: full,       label: Full }
        - { id: extra-lite, label: Extra Lite (local models), blocks: [ "xlite:workspace_exec" ] }

That is the whole app-side exposure. The admin declares the ceiling — which local models exist, which instruction sets are allowed — and the user picks a model and an instruction profile for a conversation from what the agent offered. Both are id-based inventories: the picker carries only labels and ids, never the endpoint or the instruction text.

Size the window to your prompts

The one setting that is load-bearing is num_ctx. An agent's decision prompt is large — tens of thousands of tokens once you count the protocol, the tool catalog, and any admin instructions. Ollama's default window silently truncates a longer prompt from the front, where the system instruction lives, and the model then answers as unstructured text the runtime cannot route. Set the window above your largest prompt; the only symptom otherwise is a truncating input prompt line in the Ollama log.

Give the small model a smaller prompt

A locally served model pays for every prompt token in seconds of evaluation. The lever is the instruction set. The agent's instructions are assembled from composable blocks, and the same machinery can compose a distilled set that keeps every hard signal and drops the restatements and long examples.

full instruction body   ≈ 27,000 tokens
extra-lite body         ≈  6,500 tokens   (same rules, distilled)

That distilled set is the extra-lite instruction profile offered above. Picking a local model and the extra-lite profile together is the difference between a several-minute first token and a usable one.

What you get

The pieces already existed — a router with a custom provider, a capability picker, composable instructions, descriptor props. The gateway and a reserved property tie them into one outcome: a model on your laptop, chosen for a conversation, driving the same agent as a hosted one — and told what it needs to know in a prompt it can afford to read.

One gateway, many models. The client sends the model name; the gateway routes it.
Descriptor-only. services.llm.custom plus the agent's supported_models and instruction_profiles.
Per-conversation. The user picks a local model and a lean instruction profile; nobody's defaults change.

Documentation on GitHub

The live docs behind this entry: