Troubleshooting Guide¶

Common errors and solutions for Cogniverse.

Table of Contents¶

Installation Issues
Deployment and Cluster Issues
Import Errors
Vespa Issues
Phoenix Issues
Embedding Issues
Agent Issues
Configuration Issues
Testing Issues
Runtime Issues

Installation Issues¶

uv Not Found¶

Error:

command not found: uv

Solution:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Or with pip
pip install uv

# Verify
uv --version

Package Installation Fails¶

Error:

error: Failed to resolve dependencies

Solution:

# Clear cache and reinstall
uv cache clean
uv sync --refresh

# If still failing, check Python version
python --version  # Should be 3.12+

# Install specific Python version
uv python install 3.12
uv sync

Missing System Dependencies¶

Error:

error: libffi.so not found

Solution:

# macOS
brew install libffi

# Ubuntu/Debian
sudo apt-get install libffi-dev

# Fedora/RHEL
sudo dnf install libffi-devel

Deployment and Cluster Issues¶

Cogniverse deploys as a Helm chart onto a Kubernetes cluster. cogniverse up (from the cogniverse CLI, libs/cli/) creates a local k3d cluster if none exists and no other Kubernetes context is already reachable, then installs the chart into the cogniverse namespace. Services are reached on the host through the k3d loadbalancer's NodePort mappings, not raw docker ps/docker logs.

Missing Prerequisites¶

Error:

Missing prerequisites:
  k3d: curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
  kubectl: ...
  helm: ...

Solution:

cogniverse up checks for docker, kubectl, helm, and (when no existing Kubernetes cluster is reachable) k3d, and offers to install what's missing.

# Let cogniverse up install missing tools interactively, or install manually
# with the printed commands, then re-run:
cogniverse up

Port Already Allocated¶

Error:

Error response from daemon: driver failed programming external connectivity:
Bind for 0.0.0.0:28000 failed: port is already allocated

Solution:

cogniverse up maps a fixed set of host ports through the k3d loadbalancer: 8080/19071 (Vespa), 28000 (runtime), 28400 (web client), 26006 (Phoenix), 4317 (OTLP), 11434 (Ollama), 2746 (Argo), plus 29001-29011 for inference sidecars.

cogniverse start checks the stored mappings of a stopped load balancer before starting any cluster node. If a port is occupied, it reports the host port and the Cogniverse service that needs it without leaving the cluster partially started.

# Find what's holding a conflicting port
lsof -nP -iTCP:11434 -sTCP:LISTEN

# Choice 1: free or reconfigure the host listener, then resume the cluster
cogniverse start --name cogniverse

k3d's experimental edit command can add a published port, but it cannot remove or remap one on an existing cluster. Setting COGNIVERSE_K3D_EXCLUDE_PORTS after creation therefore does not change its mappings. To keep a host Ollama listener on 11434, preserve any cluster-local data and recreate the cluster without the conflicting mapping:

# Choice 2: remove the existing cluster only after preserving required data
k3d cluster delete cogniverse
export COGNIVERSE_K3D_EXCLUDE_PORTS=11434
cogniverse up

k3d Load Balancer Detached from the Cluster Network¶

After k3d reports a successful start, cogniverse start verifies that the load balancer joined the cluster network. A detached load balancer cannot resolve the server container, so the Kubernetes API and every published service remain unavailable even though k3d reports the server as running.

The CLI reports the exact container and network names. Reattach and restart that load balancer without deleting the cluster:

docker network connect k3d-cogniverse k3d-cogniverse-serverlb
docker restart k3d-cogniverse-serverlb

Pods Stuck Pending / CrashLoopBackOff¶

Solution:

# Check overall status
cogniverse status

# List pods and their state directly
kubectl get pods -n cogniverse

# Tail logs for a specific service (runtime, web, vespa, phoenix, llm, argo)
cogniverse logs runtime -f

# Describe a pod for scheduling/image-pull failures
kubectl describe pod <pod-name> -n cogniverse

Import Errors¶

ModuleNotFoundError¶

Error:

ModuleNotFoundError: No module named 'cogniverse_core'

Solution:

# Ensure all packages are installed
uv sync

# Check package is installed
uv pip list | grep cogniverse

# Verify import works
uv run python -c "import cogniverse_core; print('OK')"

# If using IDE, ensure correct interpreter
which python
# Should point to .venv/bin/python

Circular Import¶

Error:

ImportError: cannot import name 'X' from partially initialized module

Solution:

Check import order in __init__.py
Use TYPE_CHECKING for type hints that cause cycles:

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from cogniverse_core.agents import AgentBase

# Use string annotation
def process(agent: "AgentBase") -> None:
    ...

Vespa Issues¶

Connection Refused¶

Error:

ConnectionRefusedError: [Errno 111] Connection refused
Backend: http://localhost:8080

Solution:

# Check Vespa pod status (cogniverse deploys to a k3d/Kubernetes cluster,
# not raw docker containers -- Vespa runs as the cogniverse-vespa StatefulSet)
cogniverse status

# Start Vespa (deploys the full stack, including Vespa, via k3d + Helm)
cogniverse up  # Starts all services including Vespa

# Check health
curl http://localhost:8080/ApplicationStatus

# Check logs
cogniverse logs vespa -f

Schema Deployment Failed¶

Error:

vespa.deployment.ApplicationError: Schema deployment failed

Solution:

# Check config server is ready
curl http://localhost:19071/ApplicationStatus

# Wait for config server (can take 30-60s after start)
sleep 60

# Schema deployment is handled by pyvespa ApplicationPackage
# See scripts/run_ingestion.py for schema deployment examples

# Check deployment status
curl http://localhost:8080/ApplicationStatus

Embedding Dimension Mismatch¶

Error:

Expected 768 values, got 1024

Solution:

Check schema embedding dimension matches model:
ColPali (Tomoro ColQwen3), video_colpali_smol500_mv_frame: multi-vector, 320-dim patches (float) or 40-dim (binary): tensor<bfloat16>(patch{}, v[320])
X-CLIP, single-vector profile video_xclip_sv_chunk_6s: 768-dim: tensor<float>(v[768])

# Check schema tensor types (schemas are JSON files in configs/schemas/)
cat configs/schemas/video_colpali_smol500_mv_frame_schema.json | grep -A2 '"embedding"'

# Embeddings use tensor type definitions with dimensions specified in brackets
# Example: tensor<bfloat16>(patch{}, v[320]) means multi-vector with 320-dim patches

Search Returns No Results¶

Possible Causes:

Wrong tenant_id
Schema doesn't exist
No documents ingested

Solution:

# Check tenant_id matches (content cluster id is "cogniverse_content",
# rendered by build_services_config in cogniverse_vespa.vespa_schema_manager)
curl http://localhost:8080/document/v1/?cluster=cogniverse_content

# List schemas
curl http://localhost:8080/ApplicationStatus | jq '.services'

# Check document count
curl http://localhost:8080/document/v1/<schema>/docid/?cluster=cogniverse_content | jq '.documentCount'

# Re-ingest if needed
uv run python scripts/run_ingestion.py --video_dir data/testset/evaluation/sample_videos --backend vespa

Phoenix Issues¶

Phoenix Won't Start¶

Error:

Phoenix server failed to start
Port 6006 already in use

Solution:

cogniverse up deploys Phoenix as the cogniverse-phoenix StatefulSet inside the k3d cluster; its container port (6006) is published on the host through the k3d loadbalancer's NodePort 26006, not 6006 directly.

# Check what's using the host port
lsof -i :26006

# Check pod status and logs
cogniverse status
cogniverse logs phoenix

# Restart services
cogniverse up

No Traces Visible¶

Possible Causes:

Wrong project name
Telemetry not enabled
Spans not being exported

Solution:

# Check project name format
# Should be: cogniverse-{tenant_id}-{project_name}

# Verify telemetry is enabled
from cogniverse_foundation.telemetry import get_telemetry_manager
telemetry = get_telemetry_manager()
print(telemetry.get_stats())

# Force flush spans
telemetry.force_flush(timeout_millis=30000)

# Check Phoenix UI (26006 is the k3d loadbalancer's host-published port
# for the cogniverse-phoenix service; its in-cluster port is 6006)
open http://localhost:26006

# Check OTLP endpoint is reachable
curl -s http://localhost:4317 || echo "Phoenix OTLP not reachable"

OTLP Export Fails¶

Error:

Failed to export spans: Connection refused to localhost:4317

Solution:

# Check OTLP endpoint is reachable
curl -s http://localhost:4317 || echo "Phoenix OTLP not reachable"

# Set the OTLP gRPC endpoint env var
export TELEMETRY_OTLP_ENDPOINT="localhost:4317"

# Or use the HTTP endpoint for Phoenix (26006 on the host under `cogniverse up`)
export TELEMETRY_HTTP_ENDPOINT="http://localhost:26006"

# Or configure per-project overrides in code
from cogniverse_foundation.telemetry import get_telemetry_manager

telemetry = get_telemetry_manager()
telemetry.register_project(
    tenant_id="acme",
    project_name="test",
    otlp_endpoint="http://localhost:4317",
    http_endpoint="http://localhost:26006",
)

Embedding Issues¶

Model Loading Failed¶

Error:

OSError: Unable to load weights from pytorch checkpoint

Solution:

# Clear huggingface cache
rm -rf ~/.cache/huggingface/hub

# Download model explicitly
uv run python -c "from transformers import AutoModel; AutoModel.from_pretrained('TomoroAI/tomoro-colqwen3-embed-4b')"

# Check disk space
df -h

CUDA Out of Memory¶

Error:

RuntimeError: CUDA out of memory

Solution:

# Use CPU instead
JAX_PLATFORM_NAME=cpu uv run python script.py

# Or limit frames per video
uv run python scripts/run_ingestion.py --tenant-id <tenant> --video_dir <dir> --max-frames 1

# Clear GPU memory
uv run python -c "import torch; torch.cuda.empty_cache()"

JAX Platform Error¶

Error:

RuntimeError: Unable to initialize JAX backend

Solution:

# Force CPU mode
export JAX_PLATFORM_NAME=cpu

# Or set in code (before any jax import)
import os
os.environ["JAX_PLATFORM_NAME"] = "cpu"

import jax  # Now uses CPU

Agent Issues¶

Agent Not Found¶

Error:

KeyError: Agent 'search_agent' not found in registry

Solution:

# Check agent is registered
from cogniverse_foundation.config.utils import create_default_config_manager
from cogniverse_core.registries.agent_registry import AgentRegistry
from cogniverse_core.common.agent_models import AgentEndpoint

config_manager = create_default_config_manager()
registry = AgentRegistry(tenant_id="your_org:production", config_manager=config_manager)
agents = registry.list_agents()
print(agents)

# Register agent endpoint
agent = AgentEndpoint(
    name="search_agent",
    url="http://localhost:8002",
    capabilities=["video_search"],
    health_endpoint="/health",
    process_endpoint="/process"
)
registry.register_agent(agent)

Process Method Timeout¶

Error:

asyncio.TimeoutError: Agent process exceeded timeout

Solution:

Agent timeout is controlled via AgentEndpoint.timeout (default: 30 seconds) or at the HTTP client level:

# Configure timeout when registering agent
from cogniverse_core.common.agent_models import AgentEndpoint

agent = AgentEndpoint(
    name="search_agent",
    url="http://localhost:8002",
    capabilities=["video_search"],
    timeout=300  # 5 minutes
)
registry.register_agent(agent)

# HTTP client timeout is set in AgentRegistry (default: 10 seconds)

Type Validation Error¶

Error:

pydantic.ValidationError: 1 validation error for SearchInput
query
  field required

Solution:

# Check required fields
from cogniverse_agents.search_agent import SearchInput

# All required fields must be provided
input = SearchInput(
    query="test",       # Required
    tenant_id="acme",   # Required
    modality="video",   # Has default value
    top_k=10            # Has default value
)

Configuration Issues¶

Config Not Found¶

Error:

KeyError: Configuration for tenant 'acme' not found

Solution:

ConfigManager.get_system_config() never raises for a missing config -- if nothing has been persisted yet it logs a warning and returns a SystemConfig() built from defaults. To make that default durable, write it back explicitly:

from cogniverse_foundation.config.utils import create_default_config_manager

config_manager = create_default_config_manager()
config = config_manager.get_system_config()  # returns defaults if unset

# Persist the (possibly default) config so future reads are explicit
config_manager.set_system_config(config)

Invalid Profile¶

Error:

ValueError: Profile 'video_colpali_frame' not found

Solution:

# List available profiles (port 8000 for a local `uv run uvicorn` runtime;
# 28000 if the runtime was deployed via `cogniverse up`)
curl http://localhost:8000/search/profiles

# Check config file directly
cat configs/config.json | jq '.backend.profiles'

# Get available profiles for a tenant. `service` defaults to "backend" --
# the same config service the runtime admin API and web client read/write.
from cogniverse_foundation.config.utils import create_default_config_manager

config_manager = create_default_config_manager()
profiles = config_manager.list_backend_profiles(tenant_id="acme")
print(profiles.keys())

Environment Variable Missing¶

Error:

KeyError: 'BACKEND_URL' not set

Solution:

# Set required variables
export TENANT_ID="acme"
export BACKEND_URL="http://localhost"
export BACKEND_PORT="8080"

# Or use .env file
cat > .env << EOF
TENANT_ID=acme
BACKEND_URL=http://localhost
BACKEND_PORT=8080
EOF

# Load .env in Python
from dotenv import load_dotenv
load_dotenv()

Testing Issues¶

Tests Fail with JAX Error¶

Error:

RuntimeError: jax.local_devices returned no devices

Solution:

# Always run tests with CPU mode
JAX_PLATFORM_NAME=cpu uv run pytest tests/ -v

# Add to conftest.py
import os
os.environ["JAX_PLATFORM_NAME"] = "cpu"

Async Test Not Running¶

Error:

PytestUnhandledCoroutineWarning: async def test_...

Solution:

The repo's root pytest.ini sets asyncio_mode = auto, so async def test_* functions run automatically -- no @pytest.mark.asyncio decorator is needed when tests run from the repo root. This warning almost always means either pytest-asyncio isn't installed in the active environment, or pytest was invoked from a directory where pytest.ini isn't discovered:

# Confirm pytest-asyncio is installed (pinned in pyproject.toml)
uv run python -c "import pytest_asyncio; print(pytest_asyncio.__version__)"

# Run from the repo root so pytest.ini (asyncio_mode = auto) is picked up
uv run pytest tests/path/to/test_module.py -v

Fixture Not Found¶

Error:

fixture 'config_manager' not found

Solution:

# Check conftest.py is in test directory
# tests/conftest.py

import pytest

@pytest.fixture
def config_manager_memory():
    """ConfigManager with in-memory store for unit testing."""
    from cogniverse_foundation.config.manager import ConfigManager
    from tests.utils.memory_store import InMemoryConfigStore

    store = InMemoryConfigStore()
    store.initialize()
    return ConfigManager(store=store)

Runtime Issues¶

Server Won't Start¶

Error:

uvicorn.error: Can't start server: Address already in use

Solution:

# Check what's using port
lsof -i :8000

# Kill process
kill -9 <PID>

# Or use different port
uv run uvicorn cogniverse_runtime.main:app --port 8001

API Returns 500 Error¶

Solution:

# Check server logs
uv run uvicorn cogniverse_runtime.main:app --log-level debug

# Check traceback in response (tenant_id is required)
curl -v http://localhost:8000/search/ -X POST \
  -H "Content-Type: application/json" \
  -d '{"query": "test", "tenant_id": "acme"}'

Web Client Shows "did not answer" or Is Unreachable¶

Error:

The Cogniverse runtime at http://cogniverse-runtime:8000 did not answer (TypeError).

Solution:

# The web server itself is up when /healthz answers
curl http://localhost:28400/healthz   # {"status":"ok"}

# Its logs name the runtime URL it calls
cogniverse logs web

# The runtime must be healthy and know the active tenant
curl http://localhost:28000/health
curl http://localhost:28000/admin/tenants/acme:production

A run that fails with "The runtime did not issue a harness key for tenant …" names the runtime's answer from /admin/harness/keys.

A k3d cluster created before port 28400 was published needs the mapping added: k3d cluster edit cogniverse --port-add 28400:28400@loadbalancer.

Getting Help¶

If you can't resolve an issue:

Check logs for detailed error messages
Search existing issues on GitHub
Create a minimal reproduction of the problem
Open an issue with:
Error message
Steps to reproduce
Environment details (OS, Python version)
Relevant config/code snippets