Guide: Add a new hardware (Hardware Provider)

Status: the mechanism is designed and is being rolled out in phases. This guide describes the target workflow: once the mechanism lands, adding a hardware follows these steps.
Audience: hardware vendors and Lens developers adding support for a new accelerator type.

In one sentence: adding a hardware = one profiles/<id>.json + an entry point; optionally one provider code file. Deployment overlays are reused from upstream llm-d. No core code branches on vendor.

Deployment and evaluation configuration

For Deploy and Evaluation, extend the existing profile instead of adding vendor branches to domain code. deployment.overlay_root accepts {guide}, {variant} (the profile's upstream_variant) and {model_server}. Actual guide variants are discovered from the selected source's Kustomization files; do not enumerate them in JSON. Nested overlay paths are preserved.

Configure deployment.runtime_image, managed_image_repositories, runtime_class_matchers, node_selector, and modes. Each mode declares request_model and either DRA device_class / claim_request_name, or extended resource_names. PCI filtering additionally requires supports_pci_allowlist, pci_attribute_domain and pci_attribute. Defaults do not override custom images.

In telemetry, declare device_metric_sources with metric names, matchers, units, scales, output_unit, aggregation and tile_aggregation. Configure label_schema, allocation_join, allocation (drivers, PCI attributes, node labels and optional device-name pattern), source_label, and an optional scrape_path for direct snapshots. Keep Prometheus expressions in device_metrics consistent with these sources. planning.discovery owns the discovery command and parser settings; planning.memory_models contains hardware-specific capacity fallbacks.

Run the hardware/deploy/evaluate tests after changing a profile. A new profile must also pass schema.json; verify guide rendering against its intended llm-d revision.

0. Two integration models: built-in vs. external plugin

There are exactly two ways to add a hardware. Both use the same JSON contract, schema validation and fail-fast registration; they differ only in where the files live, who declares the entry points, and how the plugin is shipped.

Aspect	Built-in (in this repo)	External plugin (your package)
Intent	first-party support shipped with Lens	a vendor/add-on distributed separately
Profile JSON	`llm_d_lens/hardware/profiles/<id>.json` (packaged as data, auto-loaded)	JSON inside your package, returned by your `load_profiles()`
Provider code	`llm_d_lens/hardware/providers/<id>.py`	a module inside your package
Entry points	declared in this repo's `pyproject.toml`	declared in your package's `pyproject.toml`
Discovery source	the bundled `profiles/.json` and* the `llm_d_lens.hardware.*` entry-point groups	the `llm_d_lens.hardware.*` entry-point groups only
Installation	ships with Lens (no extra install)	`pip install <your-package>` into the backend's Python environment, then restart
Release coupling	released with Lens	your own version — adding/updating it changes no Lens source
Tests	`llm_d_lens/hardware/test_*.py`	your package's own tests (mirror the contract)
Upstream overlay	required (shared)	required (shared)

Neither model touches a domain: the profile + provider are the only vendor surface. Choose built-in only for hardware Lens intends to support first-party; otherwise ship an external plugin and leave this repository untouched.

How discovery works (both models)

llm_d_lens/hardware/discovery.py loads from two sources through one path:

the bundled llm_d_lens/hardware/profiles/*.json package data (built-ins), and
the llm_d_lens.hardware.profiles and llm_d_lens.hardware.providers entry-point groups — declared here for built-ins, and by any installed plugin.

Every payload is validated against schema.json before registration. A duplicate id with identical content is de-duplicated, and any identity conflict (the same device_classes, resource_prefixes, accelerator key or upstream_variant claimed by another profile) fails fast instead of silently shadowing. An external plugin is only visible once its package is installed in the backend's Python environment and Lens is restarted.

Deliverables

Artifact	Built-in location	External plugin location	Required?
Hardware profile (JSON)	`llm_d_lens/hardware/profiles/<id>.json`	your package (returned by `load_profiles()`)	Yes
entry point declaration	this repo's `pyproject.toml`	your package's `pyproject.toml`	Yes
Python provider	`llm_d_lens/hardware/providers/<id>.py`	your package	Optional (needed for auto driver install / telemetry)
Deployment overlay	upstream llm-d `guides/*/modelserver/<variant>/...`	same (shared)	Yes (usually already present)
Tests	`llm_d_lens/hardware/test_*.py`	your package's tests	Yes

1. Decide the hardware identity and naming

Fill in this table first. The canonical id must not collide with an existing profile; all three naming layers map to one profile.

Field	Your hardware	Example: Intel XPU	Example: NVIDIA
`id` (canonical)	?	`intel-xpu`	`nvidia`
`accelerator_keys` (Lens/AIC)	?	`["xpu","intel_gpu"]`	`["cuda","nvidia"]`
`benchmark_profile` (llm-d-benchmark)	?	`intel-xpu`	`nvidia`
`upstream_variant` (upstream dir/label)	?	`xpu`	`gpu`
`upstream_vendor` (upstream label)	?	`intel`	`nvidia`
`request_model`	?	`dra`	`extended-resource`
`resource_prefixes`	?	`["gpu.intel.com/"]`	`["nvidia.com/gpu"]`
`device_classes` (DRA)	?	`["gpu.intel.com"]`	`["gpu.nvidia.com"]`
`node_label_selector`	?	`{"intel.feature.node.kubernetes.io/gpu":"true"}`	`{"feature.node.kubernetes.io/pci-10de.present":"true"}`

How to choose request_model: use dra when the cluster selects devices via DRA ResourceClaimTemplate / deviceClassName; use extended-resource when it uses an extended resource (container resources.limits, e.g. xxx.com/gpu). The two shapes are not interchangeable.

benchmark_profile must be a value llm-d-benchmark already supports (nvidia/intel-xpu/amd/…, see ACCELERATOR_PROFILES in llm-d-benchmark/llmdbenchmark/parser/cluster_resource_resolver.py). If your hardware has no profile there, add one in that repo first.

2. Check whether an upstream deployment overlay exists

Upstream llm-d organizes overlays by vendor directory: guides/<guide>/modelserver/<variant>/<server>/. Check whether your hardware already exists (for example NVIDIA is gpu, Intel is xpu):

# use the llm-d checkout the cluster actually uses
ls "$LLM_D_ROOT/guides/optimized-baseline/modelserver"

Exists → record upstream_variant (the directory name) and the llm-d.ai/accelerator-variant / llm-d.ai/accelerator-vendor label values from the kustomization, and put them in the profile.
Missing → contribute the overlay upstream to llm-d first (out of scope here); Lens does not author overlays.

Also confirm the overlay's resource-request shape (patch-*.yaml: claims:/deviceClassName vs resources.limits: <vendor>/gpu) to decide request_model.

3. Write the profile JSON

Place it at llm_d_lens/hardware/profiles/<id>.json (built-in) or package it with your plugin and return it from load_profiles() (external). Minimal template (declarative only: no driver install, no telemetry):

{
  "id": "myvendor",
  "display_name": "MyVendor GPU",
  "vendor": "myvendor",
  "accelerator_keys": ["myvendor"],
  "benchmark_profile": "myvendor",       // must be a value llm-d-benchmark supports
  "upstream_variant": "myvendor",
  "upstream_vendor": "myvendor",
  "request_model": "extended-resource",
  "device_classes": [],
  "resource_prefixes": ["myvendor.com/gpu"],
  "node_label_selector": {"myvendor.com/gpu.present": "true"},
  "access_modes": ["plugin"],
  "deployment": {
    "overlay_root": "guides/{guide}/modelserver/{variant}/vllm",
    "arch": "myvendor",
    "device_class": "",
    "claim_request_name": "",
    "node_selector": {"myvendor.com/gpu.present": "true"},
    "resource_name": "myvendor.com/gpu",
    "supports_pci_allowlist": false
  },
  "evaluation_profile": "myvendor",
  "planning": {"aic_system_patterns": ["myvendor"]},
  "ui": {"icon": "Cpu", "labels": {"vendor": "MyVendor", "metric_group": "GPU"}}
}

Full template with driver install / telemetry / preflight (mirroring NVIDIA):

{
  // ... fields above ...
  "access_modes": ["plugin", "dra"],            // any names; a "modes" entry per name below
  "driver": {
    "installer": "helm",                          // "helm" | "kustomize"
    "modes": {
      "plugin": {
        "manifest_ref": "my-device-plugin",       // Helm chart name (or kustomize manifest ref)
        "chart_version": "0.17.4",                // pin to a released version
        "repo": "https://myvendor.github.io/charts",
        "daemonset_matchers": [{"name_contains": "my-device-plugin"}],
        "options": {"myvendor_flag": true}        // vendor-specific; only your provider parses these
      },
      "dra": {
        "manifest_ref": "my-dra-driver",
        "chart_version": "25.8.0",
        "repo": "https://helm.myvendor.com",
        "min_kubernetes_version": "1.32.0"        // preflight/install reject older clusters
      }
      // a hardware with only one mode just declares that one entry
    }
  },
  "presence": {                                   // blocking hardware-presence preflight
    "node_label_selector": {"myvendor.com/gpu.present": "true"},
    "ensure_refs": [],                            // NFD/NodeFeatureRule refs to apply first (optional)
    "timeout_seconds": 90
  },
  "telemetry": {
    "provider_id": "myvendor",                    // must be registered in monitoring/accelerator
    "device_metrics": {"utilization": "myvendor_gpu_util"},
    "label_schema": {"node": "node", "device": "gpu"},
    "allocation_join": "none",                    // "dra" | "extended-resource" | "none"
    "pod_metrics": {"gpu_cache_usage": "vllm:gpu_cache_usage_perc"}
  }
}

Field reference:

Field	Purpose	Consumed by
`accelerator_keys`	hardware key in Lens/AIC	`server/candidateSearch.ts`, `deploy/capabilities.py`
`benchmark_profile` / `evaluation_profile`	benchmark `accelerator.profile` value	`evaluate/router.py`
`upstream_variant` / `upstream_vendor`	locate overlay, resolve identity	`deploy/runtime/composition.py`, resolver
`request_model`	DRA vs extended resource	`deploy/providers/*`, `server/guidePlanning.ts`
`device_classes` / `resource_prefixes`	device counting, DRA/expanded-resource detection	`cluster/service.py`
`node_label_selector`	node filtering, GPU counting	`cluster/service.py`, the driver
`driver`	driver/plugin install source	`monitoring/gpu_driver/`
`presence`	blocking hardware-presence preflight	`gpu_driver` + `accelerator` preflight
`telemetry`	metric queries and attribution	`monitoring/profiling/`, `evaluate/router.py`
`planning.aic_system_patterns`	AIC system-name matching	`configuration/capability.py`, `server/candidateSearch.ts`
`ui`	vendor tab, panel, chart titles	`src/components/**`

4. Register the entry point

Both models declare the same two entry-point groups; only the file that declares them differs. The entry-point name must equal the profile id.

Built-in — add to this repo's pyproject.toml (the JSON is already auto-loaded from profiles/; the entry point makes the provider resolvable):

[project.entry-points."llm_d_lens.hardware.profiles"]
myvendor = "llm_d_lens.hardware.providers.myvendor:load_profiles"

[project.entry-points."llm_d_lens.hardware.providers"]
myvendor = "llm_d_lens.hardware.providers.myvendor:MyVendorProvider"

External plugin — declare the same groups in your package's pyproject.toml, ship the JSON as package data, and return it from load_profiles() (importable from your own namespace):

[project.entry-points."llm_d_lens.hardware.profiles"]
myvendor = "myvendor_lens_hardware:load_profiles"

[project.entry-points."llm_d_lens.hardware.providers"]
myvendor = "myvendor_lens_hardware:MyVendorProvider"

[tool.setuptools.package-data]
myvendor_lens_hardware = ["profiles/*.json"]

Rules for both:

load_profiles() returns one or more .json profile objects; the loader may live in the Lens package (built-in) or your plugin package (external).
Built-in profile JSON under profiles/ is auto-loaded; every provider and every external profile is discovered through these entry-point groups. There is no other implicit registration — an unregistered hardware is rejected when it is explicitly used.
Install the plugin into the backend's Python environment; reinstall/restart after changing an entry point (pip install -e . for built-ins, pip install <your-package> for external plugins) or it will not be discovered.

5. Implement the provider (optional)

Only needed when you want "auto driver install" or "custom telemetry". A built-in places it at llm_d_lens/hardware/providers/<id>.py; an external plugin puts the same code in its own package. Apart from the module path and where load_profiles() reads the JSON from, the code is identical:

import json
from importlib.resources import files

from llm_d_lens.hardware.models import HardwareProfile
from llm_d_lens.hardware.registry import get_profile


def load_profiles() -> list[dict]:
    # built-in: package data of llm_d_lens.hardware
    text = files("llm_d_lens.hardware").joinpath("profiles", "myvendor.json").read_text()
    # external plugin: read from your own package, e.g.
    # text = files("myvendor_lens_hardware").joinpath("profiles", "myvendor.json").read_text()
    return [json.loads(text)]


class MyVendorProvider:
    def profile(self) -> HardwareProfile:
        return get_profile("myvendor")                  # resolves the discovered profile
    async def ensure_presence(self, *, cluster_id): ...  # defaults to profile.presence; may override
    async def driver_status(self, access_mode, *, cluster_id): ...
    async def install_driver(self, access_mode, *, cluster_id): ...

Do not call register_provider() yourself: the provider entry point names this class (or an instance), and discovery instantiates and registers it — a self-registration would raise "already registered".

Conventions:

Prefer the vendor's official Helm chart / kustomize for driver install, pinned to the latest validated version — never a floating latest.
Do not install the host kernel driver (assume nodes are ready); install only the device plugin / DRA driver.
Telemetry providers reuse the interface in llm_d_lens/monitoring/accelerator/registry.py, following the pattern of intel_gpu.py.
Route every kubectl / helm call through the existing CommandRunner; do not call subprocess directly.

6. Prepare driver / telemetry refs

Need	Where to get it	Note
device plugin / DRA driver	vendor's official Helm chart or kustomize	pin the version; `driver.installer` picks the invocation
telemetry exporter	vendor's official chart (e.g. DCGM exporter)	integrate with the existing monitoring stack; do not redeploy Prometheus/Grafana
NFD / NodeFeatureRule	vendor or upstream manifest	put in `presence.ensure_refs` to label hardware presence

7. Tests

Add tests under llm_d_lens/hardware/ (built-in) or your plugin package (external), covering:

Contract: the profile passes JSON Schema; id is unique; device_classes/resource_prefixes do not collide with other profiles; every alias resolves to this profile.
Discovery: use an importlib.metadata fixture to simulate an entry point; verify load, validation, and failure isolation.
Resolution: upstream labels (accelerator-variant/vendor), device class, extended-resource prefix, and accelerator_keys all resolve.
request_model: dra requires a ResourceClaimTemplate; extended-resource skips the claim and parses container resources.
Presence preflight: fail-fast when no label is present; passes when present.
Driver/telemetry: use a fake kubectl/helm runner (see FakeRunner in monitoring/accelerator/test_accelerator.py).

Run:

.venv/bin/python -m pytest llm_d_lens/hardware -q
npm run build   # if you changed frontend/gateway consumers

8. Verification checklist

GET /api/v1/hardware/capabilities returns your profile with complete fields.
The cluster overview shows this hardware's GPU count / VRAM (bucketed per vendor).
The frontend "Accelerator" area shows your vendor tab; panel defaults (type/namespace) are correct.
Deploy: the hardware's guide resolves to the correct overlay <variant> and request shape.
Evaluate: accelerator.profile carries your benchmark_profile.
With no hardware present, preflight fails fast instead of waiting.
A heterogeneous cluster (multiple vendors) resolves correctly without cross-talk.

9. Common errors

Symptom	Cause	Fix
"registration conflict" on startup	two profiles declare the same `device_classes`/prefix	make `id` and identity fields unique
Hardware does not appear	entry point not installed/reinstalled, or the external plugin is not in the backend's environment	`pip install -e .` (built-in) or `pip install <plugin>` (external), then restart
`ACCELERATOR_NOT_REGISTERED` when explicitly used	no profile registered, or `accelerator_keys` misses the key	add the profile or alias
Deployment stuck at 0 ready	overlay directory missing / wrong `arch` (e.g. `cuda` but the real dir is `gpu`)	use `upstream_variant`
`ResourceClaimTemplate is missing`	the hardware is extended-resource but treated as DRA	set `request_model="extended-resource"`
Preflight waits 5 minutes	no `presence` configured, or no node label	add `presence`; install the host driver first
Metrics missing	wrong `device_metrics`/`pod_metrics` names or `label_schema`	match the vendor exporter's metric names

10. References

Existing implementation examples: llm_d_lens/monitoring/accelerator/intel_gpu.py, llm_d_lens/monitoring/gpu_driver/
Contribution process and validation: CONTRIBUTING.md

Deployment and telemetry access modes

Declare each supported access mode in deployment.modes, including its request_model and either DRA device_class / claim_request_name or extended resource_names. Consumers resolve the selected mode rather than assuming that top-level deployment defaults apply to every driver.

telemetry.modes maps the same mode names to their allocation_join strategy. For exporters without Pod labels, pod-status uses kubelet status.containerStatuses[].allocatedResourcesStatus and telemetry.allocation.resource_id_pattern, a regular expression containing a named pci capture. Resource IDs must actually contain that device identity; opaque card indices cannot be guessed into PCI addresses. Missing identities and duplicate owners in the namespace are unavailable, not node-wide samples. Use this strategy with exclusive plugin device resources. DRA continues to use ResourceClaims and ResourceSlices. Mixed request models remain unattributed.

Guide variants are discovered from Kustomization entry points under deployment.overlay_root in the selected llm-d source. Do not maintain a second hardware-specific runtime or variant whitelist in deployment metadata.

Deployment and evaluation configuration​

0. Two integration models: built-in vs. external plugin​

How discovery works (both models)​

Deliverables​

1. Decide the hardware identity and naming​

2. Check whether an upstream deployment overlay exists​

3. Write the profile JSON​

4. Register the entry point​

5. Implement the provider (optional)​

6. Prepare driver / telemetry refs​

7. Tests​

8. Verification checklist​

9. Common errors​

10. References​

Deployment and telemetry access modes​