A local model as your agent's brain

You can point a KDCube agent at a model running on your own machine — served by Ollama — and let a user pick it for a single conversation. Once the runtime has custom-model routing, the app needs no model-specific code: a standalone models gateway on the host translates KDCube's small protocol to Ollama.

The path

The agent never learns "Ollama." It asks its model router for a custom client, which posts to an endpoint. That endpoint is the gateway; the gateway speaks Ollama.

The app selects a model; KDCube routes it through one shared host gateway.

Configured by descriptor, not code

Turning it on is a descriptor edit. services.llm.custom is a reserved platform-interpreted app property. It supplies one shared endpoint, a shared num_ctx fallback, and optional exact-model defaults. Model identity comes from the selected model row or role_models; there is no competing model_name under the service. The gateway key, when set, is an ordinary app secret (services.llm.custom.api_key), resolved at the turn door.

services:
  llm:
    custom:
      endpoint: http://host.docker.internal:11500/generate
      num_ctx: 65536
      model_overrides:
        qwen3:8b:
          num_ctx: 40960
        mistral:7b-instruct-v0.2-q4_K_M:
          num_ctx: 32768

One gateway can serve every model you have pulled: the client transmits the picked model name and the gateway routes it, so each row selects its own weights.

Offer it as a pick

The endpoint above is the plumbing. To make the local model an option a user chooses, the agent declares it in the same place it declares any model or capability — its react config. A supported_models row is a pick offer; an instruction_profiles option is another.

react:
  default_agent:
    supported_models:
      - model: qwen3.6:35b
        provider: custom
        label: Qwen3.6 35B (local)
        num_ctx: 65536
      - model: qwen3:8b
        provider: custom
        label: Qwen3 8B (local, fast)
        num_ctx: 40960
      - model: mistral:7b-instruct-v0.2-q4_K_M
        provider: custom
        label: Mistral 7B Instruct v0.2 (local)
        num_ctx: 32768
    instruction_profiles:
      default: full
      options:
        - { id: full,       label: Full }
        - { id: extra-lite, label: Extra Lite (local models), blocks: [ "xlite:workspace_exec" ] }

That is the whole app-side exposure. The admin declares the ceiling — which local models exist, their serving bounds, and which instruction sets are allowed — and the user picks a model and an instruction profile for a conversation. The saved choice contains only model/provider and profile id. Endpoint, context size, and instruction bodies remain admin-owned config.

Size the window to your prompts

The one setting that is load-bearing is num_ctx: Ollama's active context window in tokens. Instructions, tool definitions, conversation/input, and generated continuation share it. If the agent's prompt does not fit, Ollama can truncate from the front, where the system instruction lives, and the model may answer as text the runtime cannot route.

Choose a value large enough for the workload but no larger than the model's supported context. KDCube resolves it from the selected model row, then the exact-model service default, then the shared fallback. The saved user choice cannot inject a serving window. Watch the Ollama log for truncating input prompt.

Give the small model a smaller prompt

A locally served model pays for every prompt token in seconds of evaluation. The lever is the instruction set. The agent's instructions are assembled from composable blocks, and the same machinery can compose a distilled set that keeps every hard signal and drops the restatements and long examples.

full instruction body   ≈ 27,000 tokens
extra-lite body         ≈  6,500 tokens   (same rules, distilled)

That distilled set is the extra-lite instruction profile offered above. Picking a local model and the extra-lite profile together is the difference between a several-minute first token and a usable one.

What you get

The runtime pieces now meet at one path: a custom provider, per-request model routing, a capability picker, composable instructions, descriptor props, and the host gateway. The result is a model on your laptop, chosen for a conversation, driving the same agent as a hosted model — with no model-specific code in that app.

One gateway, many models. The client sends the model name; the gateway routes it.
Descriptor-only. services.llm.custom plus the agent's supported_models and instruction_profiles.
Per-conversation. The user picks a local model and a lean instruction profile; nobody's defaults change.

Documentation on GitHub

The live docs behind this entry: