Models and Inference Deployment¶

Every served model, the container image that serves it, and how the deployment differs between CPU, ROCm, and CUDA. The chart's values.yaml (chart defaults), values.k3s.yaml (the local k3d dev overlay cogniverse up applies by default), values.rocm.yaml, values.cuda.yaml, and the deploy/ sidecars are the underlying source of truth; this page flattens them into one reference.

Overview¶

Cogniverse runs five classes of inference services:

Each service lives in charts/cogniverse/values.yaml under the inference: block (or under top-level llm: for the LLM serving path). The chart renders one Deployment + Service per enabled service.

LLM serving¶

Student (vLLM chat / generation LLM)¶

The base default LLM engine is Ollama, not the vLLM student (see Optional: Ollama instead of vLLM below — despite the name, Ollama is what a fresh CPU install and the local k3d dev overlay (values.k3s.yaml) actually run). The vLLM student pod is the production path used once a ROCm (or manually configured CUDA) GPU is available: when enabled, it becomes the primary chat LLM used by every agent for DSPy/litellm calls. The Helm template (cogniverse.primaryLLMModel in templates/_helpers.tpl) always prepends the openai/ provider prefix and writes the resulting model id verbatim into config.json; create_dspy_lm() passes it through unchanged. The actual destination is determined by api_base, not the prefix.

Teacher (DSPy optimization only)¶

The teacher is scaled to zero by default because it's only needed when cogniverse_runtime.optimization_cli runs prompt optimization with a teacher configured (the larger model proposes few-shot demos for the optimizer to evaluate). Bring it up before optimization runs:

kubectl scale deployment/cogniverse-vllm-llm-teacher -n cogniverse --replicas=1
kubectl rollout status deployment/cogniverse-vllm-llm-teacher -n cogniverse --timeout=600s
# ... run optimization ...
kubectl scale deployment/cogniverse-vllm-llm-teacher -n cogniverse --replicas=0

The agent code wires it via LLMConfig.resolve_teacher() → create_dspy_lm() → teacher_settings on BootstrapFewShot in libs/agents/cogniverse_agents/optimizer/dspy_agent_optimizer.py, driven by cogniverse_runtime.optimization_cli. See the ColPali/ColQwen embedding flow for the runtime side that consumes the optimized program.

Optional: Ollama instead of vLLM¶

Ollama is what a fresh CPU install and local development actually run; "instead of vLLM" here means instead of the vllm_llm_student pod described above, which is disabled until the ROCm (or a manually configured CUDA) overlay turns it on. Use Ollama on machines without a vLLM-ready GPU. The Helm template (cogniverse.primaryLLMModel) writes openai/<model> into config.json regardless of engine; llm.engine only selects the api_base URL (pointing at the in-cluster Ollama /v1 endpoint in this case). Modern Ollama exposes /v1/chat/completions, so the OpenAI-compat wire contract routes to it unchanged.

Optional: external LLM endpoint¶

Set llm.engine: external and llm.external.url: <your-endpoint> to deploy nothing and route runtime LLM calls to a host-side or third-party endpoint (e.g. http://host.k3d.internal:11434 for a host-running Ollama).

Route through the vLLM Semantic Router¶

cogniverse up deploys the vLLM Semantic Router (Envoy front-end + the router) in front of the LLM backend, and the runtime routes every agent's LLM call through it. Envoy has one cluster per backend and routes on the catalog model the router selected (x-selected-model): basic-chat goes to the same LLM the runtime would otherwise call directly — the chart's srUpstream* helpers derive its host/port from primaryLLMEndpoint, so it tracks the llm.engine in use (ollama → the -llm service, vllm → the -vllm-llm-student service, external → the configured URL) — and pro-reasoning goes to the teacher, whose srTeacher* helpers derive from llmTeacherEndpoint (inference.vllm_llm_teacher, or its externalUrl). A render with the router on and no served teacher fails. Claim extraction during ingestion keeps the direct primary LLM endpoint. The division of labor:

• cogniverse sends only who the tenant is — the tenant identity (x-authz-user-id = tenant_id) and its tier (x-authz-user-groups, the tenant's stored tier). It does not classify the request.
• the router gates the tenant's allowed model set by tier (its authz signal — which requires the identity header and refuses to evaluate role bindings without it) and classifies the request content itself (domain/complexity) to pick the model + reasoning mode.

The router's own policy — model catalog, tier→role bindings, and the content-driven decisions — lives in the chart at charts/cogniverse/files/semantic-router/config.yaml (v0.3 schema). The default ships a pro/free tier split and a "technical domain → reasoning model" decision; extend the providers.models / routing.decisions there as you add models.

SemanticRouterConfig — the cogniverse side¶

SystemConfig.semantic_router (a SemanticRouterConfig) controls what cogniverse sends. The helper cogniverse_foundation.config.semantic_router.apply_semantic_routing(endpoint, config, tenant_id) returns a copy of the endpoint config with api_base rewritten to the router, model replaced by the router's auto alias, and the two authz headers merged onto extra_headers:

The resolved headers win on a key collision with any pre-existing extra_headers. The block is part of SystemConfig, which the runtime reads from the config store (Vespa) — not from config.json. A deployed runtime receives it from the chart via the SEMANTIC_ROUTER_ENABLED / SEMANTIC_ROUTER_URL env vars, which main.py folds into SystemConfig.semantic_router at boot. A tenant's tier is not deployment config: it is a per-tenant attribute, read per request and set through PUT /admin/tenants/{tenant_id}/tier (see docs/modules/runtime.md).

Agents build a router-aware LM through one shared helper, semantic_router.create_routed_lm(endpoint, config, tenant_id, tier) (apply_semantic_routing + create_dspy_lm); resolve_semantic_router_config(...) reads the block from a ConfigUtils-like accessor (a broken config store raises — no silent bypass). DynamicDSPyMixin uses these at LM-construction time, and the per-request paths (the orchestrator and the direct-build execution agents) build their LM the same way with the request's tenant_id.

Choosing a model means the router rebuilds the request body from its own parsed struct before forwarding it, so every field an agent sends has to survive that round trip. response_format is the one with teeth: agents bind their DSPy signature to a server-enforced json_schema (StructuredJSONAdapter), and a router build that keeps only {"type": "json_schema"} makes vLLM reject the request with 400 — the agent then serves its non-LM fallback on every structured call. semanticRouter.router.image.digest therefore pins a build that rebuilds response_format from the client bytes, and test_json_schema_response_format_survives_the_router asserts the schema the stub upstream receives is the payload production built, field for field.

The SemanticRouterConfig dataclass defaults to disabled, so unconfigured library use is a no-op — but cogniverse up turns routing on by default (semanticRouter.enabled: true). Opt out with cogniverse up ... --set semanticRouter.enabled=false. The router downloads its classifier bundle on first boot into a model-cache PVC. Its startup probe allows 30 minutes for that cold download because model pulls can be serialized on a shared development host; warm starts reuse the PVC. Coverage: tests/foundation/integration/test_semantic_router_e2e.py self-launches the real router+Envoy+stub via docker run (on the image digests charts/cogniverse/values.yaml deploys) and asserts the tier/content decisions; tests/e2e/deployment/test_semantic_router_deploy_e2e.py rides the deployed_stack fixture (its own isolated k3d cluster running the full chart) and asserts the routing decision per tenant tier + content against the deployed router's llm_decision_match_total metric; tests/charts/test_semantic_router_chart.py pins the rendered upstream endpoint and served model per llm.engine.

Visual embeddings¶

ColPali / ColQwen (visual multi-vector retrieval — one pod)¶

There is a single chart entry, inference.vllm_colpali, that serves both the ColPali and ColQwen retrieval families — both route to the same TomoroAI/tomoro-colqwen3-embed-4b (ColQwen3, 320-dim per-patch multi-vector) checkpoint. The video_colpali_smol500_mv_frame, image_colpali_mv, and video_colqwen_omni_mv_chunk_30s profiles all set inference_service: "vllm_colpali" — there is no separate vllm_colqwen chart key.

The k3s overlay changes neither this service's image nor its enabled state, so a CPU-only cogniverse up does not allocate a ColPali/ColQwen pod. On ROCm 7.12+ with gfx1151 and on CUDA, the device overlay enables the service and the official vLLM pooling runner serves the vllm_token_embed multi-vector contract.

X-CLIP (chunk-level video embeddings, video_embed sidecar)¶

Used by the video_xclip_sv_chunk_6s profile, which embeds 6-second clips and text queries into the same space so a text query retrieves clips directly. The runtime image ships no torch, so text-to-video search works only through this sidecar (inference_service_urls["video_embed"]). cogniverse up builds and imports the image from the repository root like the other custom sidecars (see the image table below).

Text embeddings¶

ColBERT (late-interaction, multi-vector text)¶

Serving: the PyLate sidecar (libs/cli/cogniverse_cli/modal_inference/servers/pylate.py) runs pylate.models.ColBERT itself and owns both encode directions: queries get PyLate's exact expansion — mask-token padding excluded from attention — and documents get the marker plus punctuation skiplist. Stock vLLM cannot serve LateOn exactly: its /pooling request accepts text or token IDs but no attention mask, so it attends to all query positions and every per-token vector drifts from the trained model's output. Clients send raw text plus is_query; no prefixes or token IDs are built client-side.

Code search (ColBERT variant)¶

DenseOn (single-vector dense text)¶

DenseOn uses the same vLLM pooling runner as the ColBERT path, but vllm_embed pools to a single dense vector per input (no per-token reshape), matching DenseOn's dense-retrieval semantics.

Audio (ASR + acoustic embeddings)¶

Whisper¶

vLLM's stock images don't ship the audio packages, so the derived image installs them at build time and the pod runs vllm serve directly. The service's device selects the repository and the upstream image it derives from. The endpoint is /v1/audio/transcriptions (OpenAI-compatible multipart upload). The remote clients send each 30 s chunk as verbose_json (timings) and, unless the timed segments run to the end of the chunk, as json (text), plus one request per answer that comes back empty or as a repetition loop.

CLAP acoustic embeddings (clap_embed sidecar)¶

When the sidecar is deployed, AudioEmbeddingGenerator routes acoustic embeddings to it via inference_service_urls["clap_embed"] (ingestion side injected by the embedding-generator factory; query side via AudioAnalysisDeps.clap_endpoint filled by the dispatcher). Without it, CLAP loads in-process — which requires torch and therefore only works in dev environments, never in the deployed runtime image; in that case the acoustic vector is skipped (best-effort) and audio chunks carry only transcript + semantic embeddings.

The runtime pods also need NUMBA_CACHE_DIR writable (set by the chart) or librosa's numba JIT crashes with "no locator available". If the pinned CLAP model cannot load, /health returns HTTP 503 and its detail matches clap_embed: model laion/clap-htsat-unfused load failed (<ExceptionType>): <cause>. Readiness therefore keeps the pod out of the Service while the TCP liveness probe leaves the running process available for diagnosis and recovery.

Vision/NLP sidecars¶

These two sidecars keep heavy ML dependencies (torch + gliner, torch + insightface) out of the slim runtime image. Both are FastAPI services with a model-backed /health readiness and model-identity endpoint plus one predict route. Their liveness probes check only the TCP serving socket, so a model load fault cannot create a Kubernetes restart loop.

GLiNER (zero-shot entity extraction)¶

GatewayAgent classifies queries by modality/generation-type using GLiNER zero-shot NER (urchade/gliner_large-v2.1), and EntityExtractionAgent / the knowledge-graph doc extractor use the same sidecar for entity extraction. The sidecar accepts only the canonical large model and its immutable revision; the optional request model field must match that identifier exactly. RemoteGlinerClient is the client-side loader that replaces the in-process GLiNER loader transparently. A pinned-model load failure returns HTTP 503; its detail matches gliner: model urchade/gliner_large-v2.1 load failed (<ExceptionType>): <cause>.

The export is DeBERTa-v3-large as an ONNX graph. ONNX Runtime's optimizer folds the relative-position projections that PyTorch recomputes in every layer of every request; they depend only on the weights and are more than half of the forward pass for a gateway-length query. The export writes source.json naming the checkpoint, and the server refuses to load an export of any other model or revision. transformers and onnxruntime are pinned in the Dockerfile because the graph traces one and runs on the other. tests/cli/integration/test_gliner_image_serving.py checks the e2e cluster's gliner pod: the baked export's source.json names the pinned checkpoint, the server runs it on ONNX Runtime with as many threads as the pod's CPU limit, and its entities match the PyTorch checkpoint's.

InsightFace (face embeddings, face_embed sidecar)¶

The Modal image downloads that official archive during its image build, verifies the digest before unpacking, and installs the required ONNX files at /opt/insightface/models/buffalo_l. Runtime initialization passes that root to InsightFace and fails if any required file is missing; it never downloads model data on the first inference request. A load failure returns HTTP 503; its detail matches face_embed: model buffalo_l load failed (<ExceptionType>): <cause>.

Only the detector and recogniser are loaded (the pack's landmark and gender/age models produce nothing /embed returns). On CPU each ONNX session runs FACE_EMBED_INTRA_OP_THREADS threads (chart value "1", with clients keeping one request in flight per sidecar CPU); ONNX Runtime's own default sizes its pool from the node's cores, not the pod's CPU limit. At the chart's 2-CPU limit a 640x480 frame of 5-7 faces takes 0.4-0.6 s, and at most 1.1 s with 4 requests in flight; with the default pool and all five models the same frames took 10-12 s each, and up to 43 s with 4 in flight. Boxes, vectors and scores are unchanged: test_face_embed_reference_parity compares the served faces with tests/fixtures/model_references/face_embed.json, recorded from the full pack with default sessions.

The service is stateless — no persistence. The knowledge-graph face pipeline (libs/agents/cogniverse_agents/graph/face_extractor.py) reads each keyframe KeyframeProcessor wrote from its path, POSTs it as image_b64, one keyframe per request with one request per sidecar CPU in flight (FACE_EMBED_MAX_CONCURRENCY, set by the chart from inference.face_embed.resources.limits.cpu), retries a failed request once, and clusters the returned 512-dim L2-normalized ArcFace vectors per source_doc_id to discover anonymous identity groups, attributing each cluster to a temporally-aligned named person where possible. It's wired the same way as CLAP: opt-in via inference_service_urls["face_embed"] (_lookup_face_embed_endpoint in libs/runtime/cogniverse_runtime/routers/ingestion.py), additive — a keyframe that still fails is logged as a warning naming its segment and cause and the rest of the video's faces proceed, and an unreachable sidecar (every keyframe failing) is logged as a warning rather than failing the whole ingest. Each run logs its counts at INFO: Face pipeline for source_doc_id=<id>: <n> faces in <k> keyframes, <f> failed keyframes, <c> clusters, <e> same_as edges, <a> anonymous face nodes, or no keyframes in the result / no faces in <k> keyframes, <f> failed keyframes.

Deployment style summary¶

Custom images are built locally by cogniverse up (which calls build_images() in libs/cli/cogniverse_cli/images.py) and imported into the k3d cluster. Each tag comes from the latest commit that touched that image's Dockerfile, local COPY/ADD sources, or .dockerignore. Images whose tag already exists on the host are not rebuilt, and tags already present in the k3d node are not re-imported. They are NOT published to a public registry — they're loaded from the host docker daemon into the cluster's containerd via k3d image import.

Starting without network access¶

Every inference pod starts from its image and the model cache alone, so a node that comes back without DNS still serves:

• No container installs packages or fetches from outside the cluster when it starts; packages are baked in at image build time.
• Each model server runs with HF_HUB_OFFLINE=1 and loads its pinned revision from the cache.
• The model-warm init container puts that revision in the cache first. It resolves the revision with local_files_only=True and makes no network call when it is cached; only a missing revision is fetched (from the MinIO mirror when configured, then the Hub). It covers every service that loads Hub weights (inference.<svc>.model and revision/modelRevision); GLiNER and InsightFace serve artifacts baked into their images.

tests/charts/test_model_pod_offline_startup.py enforces this for every inference pod the chart renders. scripts/check_offline_model_pods.py applies the same checks to a running cluster, read-only, and also fails a pod that is not Ready or has restarted:

uv run python scripts/check_offline_model_pods.py [deployment ...] \
    --context k3d-cogniverse-e2e --namespace cogniverse

Modal-hosted services (inference.<svc>.externalUrl)¶

Any non-LLM service can be served from Modal instead of an in-cluster pod. Setting a non-empty inference.<svc>.externalUrl (an absolute https URL, e.g. https://<workspace>--cogniverse-<svc>.modal.run):

• keeps the service's entry in INFERENCE_SERVICE_URLS, pointed at the external URL (for denseon, COGNIVERSE_SEMANTIC_EMBED_URL follows too);
• skips the local Deployment, Service, and model-cache PVC, drops the service from the local image build, and removes it as a startup-sequence gate;
• leaves every other service's rendering unchanged.

Callers authenticate to https://*.modal.run endpoints with COGNIVERSE_INFERENCE_API_KEY, delivered to the runtime and ingestor pods from Secret cogniverse-inference-api-key (synced by cogniverse secrets sync). Deploy and warm the Modal side with cogniverse inference modal deploy/warm <svc>.

externalUrl is rejected at render time on vllm_llm_student and vllm_llm_teacher: the LLM endpoints are derived by their own chart helpers and are overridden via runtime.primaryLLM.apiBase/model/apiKey instead.

Selected video profile (config.defaultProfiles.video)¶

The reference deployment is values.k3s.yaml + values.rocm.yaml + values.modal-llm.yaml on an AMD Strix Halo host: ColPali and ASR in-cluster, the student and teacher LMs on external endpoints.

With the Modal LLM apps undeployed the runtime stays ready and search keeps serving. Each worker's first LM call gets Modal's 404; after it, calls to that endpoint fail fast and one call rechecks it every 30s (LM endpoint availability). Searches report query_rewrite.degraded: query_rewrite_lm_not_serving, orchestrated queries answer 503 llm_unavailable with Retry-After, and /health answers degraded, naming the endpoint under dependencies.llm. A redeployed app is used again from the next recheck; its first request waits out the cold start.

config.defaultProfiles.video names the profile the chart writes to backend.default_profiles.video (profile only; search takes the ranking from the profile's schema) and active_video_profile, and the one profile the schema-deployment job deploys. The base values select none; values.rocm.yaml and values.cuda.yaml select video_colpali_smol500_mv_frame. The render fails, naming the profile and key, unless every service in the selected profile's inference_services has enabled: true or an externalUrl, and unless a description vlm_endpoint on the in-cluster vllm_llm_student Service has that service enabled. values.cuda.yaml deploys no student, so a CUDA composition sets runtime.primaryLLM.apiBase or composes values.modal-llm.yaml.

Device selection (device: per service)¶

Each vLLM-backed inference.<svc> block has a device: key (the FastAPI sidecars that are always CPU-only — clap_embed, face_embed, video_embed — omit it entirely; gliner sets device: cpu for documentation even though nothing reads it for GPU scheduling). Values:

cogniverse up auto-applies the right values overlay (values.rocm.yaml or values.cuda.yaml) and the node label when the host has the corresponding device — see scripts-operations.md.

For Strix Halo (gfx1151) iGPU specifically, the GPU "VRAM" IS host RAM (unified memory). The chart's ROCm overlay gives generation services bounded --gpu-memory-utilization fractions. The Tomoro pooling service uses an explicit 1 GiB KV cache because its transient image-profile allocation is larger than its steady-state cache budget; it also caps preprocessing at 1,048,576 pixels, matching the 1,024-patch Vespa document contract. See values.rocm.yaml for the complete per-service allocation. Its 0.18 utilization value is vLLM's initial free-memory guard; the explicit byte value controls the actual cache reservation.

vLLM sizes the vision batch from the step's token budget, as --max-num-batched-tokens // 1024 maximum-size images, both when it profiles memory at startup and when it schedules concurrent image requests. The ROCm overlay sets that budget to 1,536 tokens, so concurrent ingestion still encodes one image per step while sixteen text queries share a step. Chunked prefill is unsupported for this CLS-pooled model, so --max-model-len equals the budget; the largest request, one 1,024-patch image, is 1,031 tokens. --max-cudagraph-capture-size 512 pads every step of up to 512 tokens to one of 51 captured sizes, so query traffic runs a fixed set of GEMM shapes. tests/charts/test_rocm_startup_budget.py pins these relations.

Inference readiness probes begin immediately. A failed readiness probe only keeps the Service endpoint out of rotation; it does not restart the container. Cold-start protection remains on the separately budgeted liveness probe, so a model becomes routable on the first successful /health response instead of waiting through an additional fixed delay.

GEMM auto-tuning on ROCm (runtime.tunableOp)¶

runtime.tunableOp (false in values.yaml, true in values.rocm.yaml) enables PyTorch TunableOp on rocm-device inference pods. On gfx1151 the default hipBLASLt kernel heuristic mistunes many GEMM shapes; TunableOp benchmarks the candidate kernels once per shape and reuses the fastest. The key only takes effect when a pod's device is rocm — the cogniverse.tunableOpEnv helper gates on both the device and the toggle, so CPU/CUDA pods are unaffected.

Services on the pylate engine are excluded: their server faults the GPU (exit 139, Memory access fault by GPU node-1) while TunableOp benchmarks an untuned GEMM shape, and each new batch size or sequence length is a new shape. Set inference.<service>.tunableOp to override the engine-derived default in either direction.

Each participating rocm pod gets PYTORCH_TUNABLEOP_ENABLED=1, PYTORCH_TUNABLEOP_TUNING=1, and a per-service PYTORCH_TUNABLEOP_FILENAME=/root/.cache/huggingface/tunableop_<svc>_%d.csv. The results file lives in the persistent model-cache volume, so tuning survives pod restarts and rollouts — a shape is benchmarked once over the file's lifetime. The first request hitting a not-yet-tuned shape pays a one-time tuning latency; the persisted file means later pods skip it.

inference.<service>.tunableOpTuning: false renders PYTORCH_TUNABLEOP_TUNING=0: tuned shapes from the results file still apply, and a shape the file lacks runs the default kernel instead of being tuned on the request path. vllm_colpali sets it on the ROCm overlay. Its batched step lengths vary with the mix of concurrent queries and images, and tuning one new length stalls every queued request (31 s for a new 1,031-token image length, against 0.76 s once tuned).

Per-tenant Vespa schemas (separate from inference)¶

The seven schemas you'll see in Vespa per tenant — video_colpali_*, audio_content_*, document_text_*, image_colpali_mv_*, knowledge_graph_*, wiki_pages_*, agent_memories_* — are NOT served by the inference services. They're Vespa document schemas the runtime feeds into directly. The inference services produce the embeddings; Vespa stores and ranks them. See architecture/multi-tenant.md for the schema lifecycle.

Reusing k3d inference services in tests¶

The local k3d overlay publishes Tomoro and ASR on their canonical NodePorts, 29001 and 29005. The shared e2e cluster maps those ports to host ports 33901 and 33905 so it can coexist with the development cluster. Integration fixtures inspect the live workload and Service objects, resolve the load balancer's published host port, and accept an endpoint only when its OpenAI GET /v1/models response lists the requested model identifier exactly. A reachable endpoint serving another model is rejected; the fixture then checks the next configured cluster or starts its own exact-model sidecar.

Warm e2e reuse is tied to both working-tree deployment content and the exact effective deployment inputs: detected backend, selected device overlay, the ordered per-image tag tuple, a content digest of the Helm chart, Helm values-file paths, and --set overrides. A stopped shared cluster is started and polled for Kubernetes access, runtime readiness, exact Tomoro/ASR identities, and the matching fingerprint. That poll has a 40-minute hard deadline so a failed model start cannot hang the test session indefinitely.

See also¶

• docs/operations/setup-installation.md — local docker-style setup with port table
• docs/operations/kubernetes-deployment.md — chart structure, GPU passthrough, manual helm install
• docs/architecture/overview.md — service graph
• charts/cogniverse/values.yaml — canonical defaults
• charts/cogniverse/values.k3s.yaml — local k3d dev overlay (cogniverse up)
• charts/cogniverse/values.rocm.yaml — ROCm overlay
• charts/cogniverse/values.cuda.yaml — CUDA overlay