Skip to main content

Observability

Open Observability to install monitoring or inspect an existing installation. Choose Cluster for cluster-wide dashboards, Accelerators for device metrics, or Deployments for one model deployment.

Inspect cluster observability status​

  1. Open Observability → Cluster.
  2. Use Target cluster to choose the cluster you want to inspect.
  3. In Namespace, enter the namespace you want to check and click Inspect.
  4. Review Live state, Installation operation, and Stack components.
  5. Use Refresh, Retry, or Recheck when those buttons are shown for the current state.

Install the cluster observability stack​

  1. In the Cluster tab, choose the target cluster.
  2. If the page shows Install stack or Install observability stack, click it.
  3. Confirm Namespace and keep Installation mode on Central (recommended).
  4. Click Run preflight.
  5. If preflight fails, fix the listed checks and run it again.
  6. When preflight succeeds, click Install stack.
  7. Follow progress in Installation operation until the page reports Cluster observability stack installed successfully.

Closing an installation dialog cancels its pending preflight. Reopening starts a fresh form and requires a new preflight; reinstall consent is not retained.

Reinstall the cluster observability stack​

  1. In the Cluster tab, click Reinstall when that action is available.
  2. In Reinstall observability stack, review the warning that reinstalling deletes stored metrics.
  3. Select I understand existing metrics data will be permanently deleted.
  4. Click Run preflight.
  5. After preflight passes, click Reinstall stack.

Open the cluster dashboards​

  1. Stay on the Cluster tab.
  2. In GUIs, click Prometheus or Grafana.
  3. If the dashboard cannot be opened, read the inline error message shown beside the buttons.

Inspect accelerator observability​

  1. Open the Accelerators tab.
  2. Choose the cluster and its accelerator under Accelerator type.
  3. Review GPU access mode and, if needed, switch between the modes that accelerator supports (for example DRA and Device plugin).
  4. Review Observability UI and Stack components for the selected accelerator.
  5. Use Prometheus, Grafana, Refresh, or Retry when those actions are available.

Install or repair accelerator observability​

  1. Open the Accelerators tab and pick the target cluster.
  2. If the page shows Install accelerator observability, Repair / reinstall, or Reinstall, click the action that matches your current state.
  3. In Install accelerator observability, set Namespace and GPU access mode.
  4. Click Run preflight.
  5. Review the preflight checks, warnings, and Command preview.
  6. When preflight succeeds, click Install.
  7. Watch Installation operation until the page reports the selected accelerator installed successfully (for example NVIDIA GPU observability installed successfully).

Manage deployment monitoring for a deployment​

  1. Open the Deployments tab.
  2. Choose the cluster in Target cluster.
  3. In Deployment monitoring, open the Monitoring menu for the deployment you want to manage.
  4. Choose Install when monitoring is not present, Enable when it is disabled, Disable when it is enabled, or Uninstall to remove it. Install and Enable stay unavailable until the deployment itself is ready, because monitoring scrapes that deployment's Pods.
  5. Wait for the Monitoring status to update before opening a dashboard.

Open deployment dashboards​

  1. In the Deployments tab, open the Monitoring menu for a deployment whose monitoring is enabled.
  2. Click Open Grafana or Open Prometheus.

Inspect deployment profiling​

  1. In the Deployments tab, open a deployment's Monitoring menu and click Profiling.
  2. In Deployment profiling, switch the Flow metric between Request and Token as needed.
  3. Keep Flow Map selected to view the current pipeline map.
  4. Use the node buttons under Inspect live flow nodes to open per-node metrics in the side drawer.
  5. In the flow-map toolbar, use Zoom in, Zoom out, Reset view, and Fit view.
  6. Click Back to deployments to return to the list.

Installation failed or dashboard unavailable​

Read the failed check or Installation operation log, correct the reported problem, then use Retry or Recheck. Do not use Reinstall merely to refresh the page: reinstalling the cluster stack permanently deletes its stored metrics.