Skip to main content

Deployments

Open Deployments to check whether a model is ready, inspect a failure, or manage an existing deployment.

P/D deployments using Router 0.11 automatically adapt legacy plugin settings when created or retried. The original configuration is retained, and deployment diagnostics record the effective Router values. A legacy disagg-headers-handler entry is replaced by the profile handler's built-in header handling; the saved prefill/decode decision is preserved. A conflicting old and new decider setting must be corrected before deployment.

Hardware profiles and guide variants​

Deployment rendering resolves hardware through the profile registry. The selected profile supplies the runtime image, scheduling labels, accelerator resource request and PCI-selection capability. Custom runtime images remain user-controlled; unknown hardware does not silently select another vendor's defaults.

Guide variants come from actual Kustomization entry points in the selected llm-d repository revision, including nested overlays. Changing hardware or source can change the available variants. A listed overlay can still require cluster resources such as NIC device classes, storage or runtime support; planning validates these requirements before deployment.

The deployment list cancels superseded filter requests so older results cannot replace the current selection. Cluster names are shown when available, with IDs as a fallback. Deletion still requires the displayed confirmation phrase and preserves the namespace choice; while deletion is pending, submission and dialog dismissal are disabled.

When specifying an operator preference for deployment planning, use English, such as prefer prefix cache, more GPU for performance, or use fewer GPU resources. The deterministic planner matches English preference keywords; Chinese keyword aliases are no longer supported.

Open the deployment creation flow​

Deployment creation requires models registered in Model Cache. Select a registered cache volume containing the ready model and its revision on the same cluster. The Advanced configuration editor has only the Model Cache source. For local-disk storage, synchronize new nodes before deployment. Ordinary provider rendering configures offline serving with a read-only cache mount. Download or repair the cache in Model Cache first, then create or retry the deployment.

Cache checks follow the active serving roles: P/D checks Prefill and Decode independently; other providers use Decode. Each role defaults to revision main. An explicit tokenizer must also be ready in the same cache volume; its revision defaults to the role's model revision unless tokenizer-revision is supplied. Conflicting dependency arguments are rejected. The rendered model and tokenizer arguments must match the checked dependencies before deployment proceeds.

Ordinary optimized-baseline rendering does not apply custom revision, tokenizer, or tokenizer-revision arguments, so those inputs are rejected. Configuration-source overrides and custom model, model-path, code-revision, hf-config-path, config, and download-dir arguments are not supported by cache preflight.

Model Cache enforcement limitations​

Exact published YAML configurations (officialGuide) bypass ordinary provider rendering. Their final manifests do not yet enforce the same offline flags, read-only cache mounts, or backing-volume binding. Cache-only deployment is therefore not guaranteed for this path. Offline environment variables are also not a network isolation boundary.

  1. In the left navigation, open Deployments.
  2. In the page header, click Create.
  3. Follow Create a deployment. The new deployment appears in this list after submission.

In Manual, open Advanced settings in the configuration editor and select Replica placement. The default prefers one node but can use others when necessary. The presets group all model-server Pods in the same rendered deployment manifest, including separate Prefill and Decode Deployments; router Pods are excluded. Require one node keeps the group on the same node; optionally select a live Node label to restrict eligible nodes. Distribute evenly across nodes balances the group across eligible nodes using hostname topology spread (up to one Pod of skew); Pods may remain Pending if the balance cannot be maintained with available capacity. Require separate nodes needs one eligible node per Pod in the group and leaves extra Pods Pending if there are not enough nodes. Neither distribution preset supports LeaderWorkerSet workloads. The label list comes from the selected cluster. Agentic deployments use the soft same-node replica preference without an affinity selector in their forms. Tensor-parallel devices are requested by each model-server Pod, so TP stays within that Pod; multi-Pod LeaderWorkerSet workers prefer the same node but can spread when necessary.

Find a deployment in the list​

  1. Review the summary cards at the top of the page: Total, Ready, In progress, Failed, and Cleaned.
  2. In the list toolbar, type in Search deployments to match a deployment by name, description, model, namespace, guide, or ID.
  3. Open Status to filter by deployment state.
  4. Open the cluster selector and choose All clusters or one cluster.
  5. If needed, change Rows per page and use the pager to move through the list.
  6. If no rows match, click Clear filters.

View deployment details​

  1. In the deployment row, click the action with tooltip View details.
  2. In the drawer, review the sections Basic Information, Model & Serving Engine, Cluster & Environment, Network & Endpoints, and Audit & Timeline.
  3. Use the copy buttons beside values such as Namespace, Cluster Session ID, Container Image, Internal Endpoint, Forwarded Endpoint, or Execution ID when you need to reuse them elsewhere.
  4. If the deployment failed, read the failure banner before making changes.

Wait for Ready before calling the deployment. For In progress, check the pod logs for startup progress. For Failed, read the failure banner and the affected pod's logs.

Failed is not final. A rollout that ran past its progress deadline is recorded as failed, but the workload often recovers on its own -- a pod that kept crashing until its model cache was complete, for example. Lens keeps re-checking failed deployments against the cluster and flips them back to Ready once their pods serve again, so leave the deployment in place rather than recreating it immediately. If the namespace is gone, the deployment is corrected to Cleaned instead.

View pods and stream logs​

  1. To see pod status inline, click the row toggle with tooltip Show pods.
  2. In the expanded row, review the pod table columns Pod, Status, Restarts, Node, and Pod IP.
  3. For the full log viewer, click the action with tooltip View Pods & Logs.
  4. Use the node filter or search fields to find the pod.
  5. Click a pod card to open Pod Logs: for that pod.
  6. If a pod has multiple containers, choose the needed Container: ... entry.
  7. Select 100 lines, 200 lines, 500 lines, or 1000 lines to change the log tail length.
  8. Turn Previous on if you need the previous container log.
  9. Leave Auto-refresh (5s) enabled to keep the pod list and logs live.
  10. Use Filter log lines... to narrow the log output, or click Copy to copy the current log text.

Create a local port-forward for a ready deployment​

  1. Find a deployment that is already Ready.
  2. In that row, click the action with tooltip Create local port-forward.
  3. Wait for the success notice Local endpoint connected.
  4. Copy the updated local endpoint from the table if you want to call the service from your workstation.

Open deployment monitoring and profiling​

  1. In the deployment row, click the action with tooltip Deployment monitoring.

  2. If monitoring has not been installed yet, click Install.

  3. If monitoring is present but paused, click Enable.

    Install and Enable stay unavailable until the deployment itself is ready -- monitoring scrapes the deployment's Pods, so there is nothing to collect before then. The menu says so while you wait.

  4. When monitoring is enabled, use Open Grafana or Open Prometheus to open the dashboards in a new tab.

  5. Click Profiling to jump to the profiling view for that deployment.

  6. Use Disable to stop monitoring without uninstalling it.

  7. Use Uninstall to remove deployment monitoring from that service.

Edit the deployment name and description​

  1. In the deployment row, click the action with tooltip Edit name and description.
  2. In Edit deployment, update Display name and Description.
  3. Click Save changes.
  4. If you decide not to keep the edit, click Cancel. To change the model, image, resources or cluster, create a new deployment; this dialog only edits its name and description.

Model inference access​

Deployment sharing is no longer supported. Admin and maintainers manage deployments within their cluster scope. Consumers do not get deployment pages or direct deployment access; publish a model service and use its Manage access drawer to grant inference to users or groups. Old deployment shares are not automatically converted into model grants.

Delete a deployment​

  1. In the deployment row, click the action with tooltip Delete deployment.
  2. In Delete deployment, review Execution ID, Cluster, and Namespace.
  3. Leave Also delete the Kubernetes namespace checked if you want Lens to remove the namespace and everything inside it, or clear it to remove only the deployment record.
  4. In Type <phrase> to confirm, enter the exact confirmation phrase shown in the dialog.
  5. Click Delete deployment.
  6. If you do not want to continue, click Cancel.

Cannot find a deployment or action?​

Clear the search, status and cluster filters first. If the deployment or action is still unavailable, ask an administrator to check your role's deployment permissions and cluster scope.

To let applications call a ready deployment, publish a model service.

In the deployment creation flow, expand Why this option under a validated alternative to see its AIC TTFT, TPOT, and throughput estimate when an exact-topology estimate is available. The estimate can appear even when the AIC candidate search finds no configuration satisfying the original SLO; in that case, the alternative still shows SLO not met. Alternatives without a matching estimate show that no exact AIC prediction is available.

The AI score shown beside an agentic candidate is a relative ranking score out of 100; the highest-scoring candidate is normalized to 100. It is not a benchmark result, confidence level, or probability of meeting the SLO. A deterministic fallback shows Deterministic score instead.

Recalculate selection works without an AI provider. It refreshes the recommendation using deterministic planning and the new preference; if AI scoring fails, the alternatives also retain deterministic scores.

Agentic alternatives are checked using the selected vLLM GPU memory utilization and max model length. The same values are applied when the deployment starts; actual free KV cache is measured by vLLM at startup and may still be lower than the planning estimate.

If you use the agentic planner from the deployment creation flow, see the Architecture overview for how workload-guided planning works.

Verify profile-driven Guide configuration​

Choose a cluster with downloaded Software Versions and refresh the page. The hardware profile selects the Guide source root; the selected llm-d checkout supplies its model servers and Kustomization variants, including nested paths. Root overlays display as Default (vllm) or Default (vllm-rdma) instead of .. Selecting a sibling runtime variant remains stable while editing other fields.

For official sources, cache and NIC controls appear after source preparation: CPU capacity requires a supported cache connector in the rendered containers; NIC count requires a claim matching the profile's NIC device class. Inspect the generated YAML and source provenance before deployment. Access-mode settings come from deployment.modes; DRA uses a device claim and plugin mode uses the declared extended resource. User-selected runtime images remain overrides.

In Evaluation results, open Resources during benchmark. Device allocation failures display a reason even when engine metrics are available. Plugin device metrics require kubelet-reported, profile-recognized device identities; an exporter alone cannot establish which devices belong to the workload. Profile file edits require restarting the development services and refreshing the page; this is not a live JSON editor. Existing saved results are not recollected.