Skip to main content

Troubleshooting

First request after installing/publishing a model service fails, retry succeeds​

Symptom: right after installing the shared Gateway or publishing a new service provider, the first curl to the gateway returns upstream connect error or disconnect/reset before headers ... Connection refused, but an identical retry a few seconds later succeeds.

Cause: kubectl apply only means the Envoy Gateway controller has seen the desired Gateway/HTTPRoute resources — it still needs a moment to translate them into real Envoy xDS config and push it to the proxy. Lens polls the Gateway's status.conditions[Programmed] after apply and only reports the install as ready once that flips to True (default timeout 30s); if you still hit this within that window, wait a few seconds and retry, or check the install operation's log drawer for a "still programming the route" message.

Gateway stays Programmed=False (AddressNotAssigned)​

Cause: the Gateway's LoadBalancer Service has no external address, so Envoy Gateway never assigns one. Fix: on cloud clusters the provider assigns it; on local kind clusters install cloud-provider-kind or MetalLB. Check with kubectl get svc -n envoy-gateway-system.

HTTPRoute ResolvedRefs=False (InvalidKind) for InferencePool​

This repair is for Envoy AI Gateway, not the Istio provider selected by the cluster wizard. Ask the cluster administrator to install Envoy Gateway with its inference-pool addon. Without it, routes to InferencePool return 500.

Using the target cluster's kubeconfig, run:

helm upgrade -i eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.8.1 -n envoy-gateway-system --create-namespace \
  -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/v0.7.0/manifests/envoy-gateway-values.yaml \
  -f https://raw.githubusercontent.com/envoyproxy/ai-gateway/v0.7.0/examples/inference-pool/envoy-gateway-values-addon.yaml

Then open Model services, reconcile the affected cluster, and retry the request.

"no matches for kind ... ensure CRDs are installed first" during Gateway install​

Cause: kubectl's client-side discovery cache can report a CRD as unknown for a short window right after it becomes Established, or after an earlier failed install attempt cached a stale "unknown API group". Lens retries the apply a few times with a short delay and invalidates the discovery cache before applying; if it still fails, re-run the install — the CRDs themselves are already present.

Model Service permission change isn't visible in the UI​

Cause: the session's permission set is fetched once when the browser tab loads and cached in memory; it is not re-fetched automatically after a role or backend permission-catalog change. Fix: hard-refresh the page or sign out and back in.

Backend fails to start after a code/dependency change​

Check ~/.local/state/lens/logs/dev/backend.log for the actual exception. Common causes: a missing Python dependency after a [project.dependencies] change (python3 -m pip install -e '.[dev]'), or a port already in use — scripts/lens-local.sh picks free ports automatically and logs which ones it used.

The MCP tool catalog is missing a route you just added​

Regenerate it: node scripts/generate-mcp-tools.mjs. scripts/lens-local.sh start/restart do this automatically; only a manual backend restart (without going through scripts/lens-local.sh) can leave it stale.

Reuse map / task-start validation blocks scripts/lens-local.sh start​

The reuse-first workflow validates the capability map before starting services; an invalid registration or an unresolved pending decision blocks startup without stopping already-running services. See the reuse workflow for how to resolve it.

Benchmark Kubernetes proxy errors​

If a deployment is Ready but benchmarking fails during preparing-storage with Unable to connect to the server: Forbidden, inspect proxy routing before expanding RBAC permissions: a proxy rejection can also produce this error.

The runtime merges runtime.no_proxy with backend NO_PROXY / no_proxy, adds localhost, 127.0.0.1, .svc, .cluster.local and the resolved API-server host, and deduplicates entries. Harness Pods receive proxy settings through harness.extraEnvVars; Helm downloads use a separate wrapper. The benchmark CLI and kubectl subprocesses do not inherit HTTP/HTTPS/ALL proxy variables, including lowercase variants. Explicit kubeconfig proxy-url settings remain unchanged.

Inspect the actual kubeconfig and harness/Helm bypass list if cluster access still fails. Restart the backend after process-environment changes and retry the task; historical failure records do not become successful automatically.

Benchmark client OOM​

inference-perf-.../harness (terminated: OOMKilled, exit_code=137) indicates the benchmark worker exceeded host memory, not model GPU memory. Current API and fresh UI presets request 8 GiB per worker; harness_memory_gib accepts integers from 1 to 512 and overrides the worker memory request and limit.

Increase worker memory based on observations, or reduce input lengths, dataset size, concurrency or arrival rate. Total memory reservation grows with parallelism; the node must have enough schedulable memory. Configuration changes do not resize existing Pods: launch a new task. Open-loop duration excludes request draining, and a rate above serving capacity can accumulate backlog. Do not interpret an OOM-terminated run as a complete successful measurement.