Ranking Strategy Testing Guide¶

This guide covers testing the production Vespa search path (VespaSearchBackend.search(query_dict)) with all available ranking strategies. Coverage lives in tests/runtime/integration/test_ranking_strategies_real.py — a parametrized integration test that drives every ranking strategy (each a rank-profile-name string) against a real Vespa container with seeded ColPali embeddings.

Run the test¶

uv run pytest tests/runtime/integration/test_ranking_strategies_real.py -v

What it covers¶

The test fixture chain builds the full real backend on demand:

• Real Vespa: vespa_instance fixture is a compatibility shim backed by the session-scoped shared_vespa container (see tests/conftest.py), with video_colpali_smol500_mv_frame_test_unit deployed at test setup.
• Real vLLM ColPali: remote_inference resolves the cogniverse-e2e cluster's vllm_colpali service serving TomoroAI/tomoro-colqwen3-embed-4b (pooling runner, embed convert mode, 320-dim per-token embeddings) and binds RemoteColPaliLoader against it.
• Real seed corpus: three documents with real per-token ColPali embeddings (sunset / ocean / forest scenes with matching transcripts) fed into Vespa via seeded_ranking_corpus.

There is also TestAutoSelectDefaultRanking, which omits strategy entirely and asserts the backend falls back to the schema's default rank profile (the contract SearchAgent relies on after dropping its hardcoded binary_binary default).

Each ranking strategy is a plain rank-profile-name string passed as the strategy key of the query_dict. Every strategy is exercised end-to-end through VespaSearchBackend.search:

Each case asserts on the returned List[SearchResult]: - Non-empty results from the seeded corpus. - Descending relevance order by result.score. - result.document.metadata["source_id"] falls within the seeded corpus.

Prerequisites¶

• docker available (Vespa runs in a container).
• The cogniverse-e2e cluster serving vllm_colpali (kubectl --context k3d-cogniverse-e2e -n cogniverse get deploy cogniverse-vllm-colpali); tests never start a model on this host.
• Environment variables BACKEND_URL / BACKEND_PORT set (or a configs/config.json with backend.url/backend.port) if constructing a ConfigManager via create_default_config_manager() outside the test fixtures, e.g. for the manual REPL usage below.

Strategy reference¶

The schema also defines a default rank profile (not in the table above, since no test parametrizes it by name) that TestAutoSelectDefaultRanking exercises indirectly by omitting strategy from the query dict.

Manual usage from a Python REPL¶

from cogniverse_vespa.search_backend import VespaSearchBackend
from cogniverse_foundation.config.utils import create_default_config_manager
from cogniverse_core.schemas.filesystem_loader import FilesystemSchemaLoader
from pathlib import Path

config_manager = create_default_config_manager()
schema_loader = FilesystemSchemaLoader(Path("configs/schemas"))

backend = VespaSearchBackend(
    config={
        "url": "http://localhost",
        "port": 8080,
        "profiles": {
            "test_colpali": {
                "type": "video",
                "schema_name": "video_colpali_smol500_mv_frame",
            }
        },
        # default_profiles[type] is a dict with "profile" (and optional
        # "strategy") keys — only consulted when >1 profile exists for a
        # type and query_dict omits "profile"/"strategy".
        "default_profiles": {"video": {"profile": "test_colpali"}},
    },
    config_manager=config_manager,
    schema_loader=schema_loader,
)

# Text-only — strategy is a rank-profile-name string. "schema_name" above
# is the base name; search() appends the tenant suffix internally, so
# tenant_id="test:unit" resolves to "video_colpali_smol500_mv_frame_test_unit".
results = backend.search({
    "query": "buck",
    "type": "video",
    "profile": "test_colpali",
    "strategy": "bm25_only",
    "top_k": 3,
    "tenant_id": "test:unit",
})

for result in results:
    print(result.score, result.document.metadata["source_id"])

# Visual / hybrid — pass pre-computed embeddings via `query_embeddings`
# (see test_ranking_strategies_real.py for how to encode a query through
# the cluster's vLLM service: RemoteColPaliLoader(...).load_model() returns a client
# whose .process_queries is bound to RemoteInferenceClient.process_queries_vllm).

Query tensor format¶

VespaSearchBackend formats query embeddings for Vespa via _format_query_vector_param (defined in cogniverse_vespa/search_backend.py):

• Single-vector schemas (LVT sv_chunk): numpy array flattened to a dense list — tensor(v[dim]) binding (e.g. tensor<float>(v[768])). A (1, dim) array is flattened before serialisation.
• Multi-vector schemas (ColPali, X-CLIP mv_chunk): numpy array converted to a {str(token_index): vector_list} dict — tensor<float>(querytoken{}, v[dim]) or tensor<int8>(querytoken{}, v[dim]) binding (the query-side input tensor; the stored document embedding field uses tensor<bfloat16>(patch{}, v[dim]), which is a different tensor from what queries bind to).

Schema arity is determined by _is_single_vector_schema(schema_name) from cogniverse_vespa.embedding_processor. Pass the raw numpy array from your encoder; the formatting is handled internally.

The bound input name depends on the rank profile's declared inputs (see configs/schemas/ranking_strategies.json):

Unrecognised input names are logged and skipped rather than raising, so a schema/strategy mismatch surfaces as missing ranking signal, not an error.

Related VespaSearchBackend coverage¶

This guide's real-Vespa/real-vLLM sweep in test_ranking_strategies_real.py covers rank-profile correctness only. Other aspects of VespaSearchBackend are covered by separate test files: