Model services

Backend API

Model Service routes are split across llm_d_lens/model_service/router.py's user, admin, public, and internal routers.

List model access tokens

List the caller's active model access tokens. The response never includes plaintext token values, only the metadata needed to identify and revoke them later.

REST API

GET /api/v1/model-service/tokens

Request: no request body. The caller must be authenticated with a Lens user session.

Response: returns an items array of ModelAccessToken records projected through ModelAccessToken.api_payload(), so tokenHash is omitted.

{
  "items": [
    {
      "id": "mat-7f2c4a19b6d1",
      "userId": "user-123",
      "name": "laptop",
      "tokenHint": "9mQx",
      "status": "active",
      "lastUsedAt": "2026-09-23T08:15:00Z",
      "lastUsedIp": "10.0.0.25",
      "createdAt": "2026-09-21T12:00:00Z"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
tokens = service.tokens.list_for_user("user-123", include_revoked=False)
for token in tokens:
    print(token.api_payload())

Create a model access token

Create a new model access token for the caller. The plaintext token is returned exactly once together with the persisted token metadata.

REST API

POST /api/v1/model-service/tokens

Request body (TokenCreateRequest):

• name (string): display name for the token.

{
  "name": "automation"
}

Response: one ModelAccessToken payload plus the one-time token string.

{
  "id": "mat-2c47de48a35b",
  "userId": "user-123",
  "name": "automation",
  "tokenHint": "K8xP",
  "status": "active",
  "createdAt": "2026-09-23T09:00:00Z",
  "token": "lens-mk-bTj9X7mW4zP1qN8sR6uC2yF5hL0aV3dK1mQ"
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
token, raw_token = service.tokens.create("user-123", name="automation")

print(token.api_payload())
print(raw_token)

Regenerate a model access token

Revoke the caller's current active token or tokens, then issue a fresh token. You can optionally give the replacement token a name.

REST API

POST /api/v1/model-service/tokens/regenerate

Request body: optional TokenCreateRequest. If omitted, the route uses "default".

{
  "name": "default"
}

Response: the newly issued token metadata plus the new plaintext token.

{
  "id": "mat-54b2a51d9d93",
  "userId": "user-123",
  "name": "default",
  "tokenHint": "F6rN",
  "status": "active",
  "createdAt": "2026-09-23T09:05:00Z",
  "token": "lens-mk-Gq4nP1sT8uV2xY5zA7bC9dE3fH6jK8mN0pR"
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
token, raw_token = service.tokens.regenerate("user-123", name="default")

print(token.id, raw_token)

Revoke a model access token

Revoke one previously issued model access token for the caller. After revocation, the token can no longer be used for gateway access.

REST API

DELETE /api/v1/model-service/tokens/{token_id}

Request: no request body.

• token_id (string, path): token id such as mat-7f2c4a19b6d1.

Response: 204 No Content. This endpoint does not return a JSON body.

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
revoked = service.tokens.revoke("user-123", "mat-7f2c4a19b6d1")
print(revoked)

List accessible models

List callable active models with online positive-weight providers. Consumers need inference permission in the model's cluster plus a direct or group model grant; non-self-scoped operators retain inference access in their binding's scope. Deployment shares and ownership do not grant consumer inference. This policy also applies to /v1/models and Gateway authorization. Entries include clusterName to distinguish identical public names in different clusters.

REST API

GET /api/v1/model-service/models

Request: no request body. The caller must be authenticated with a Lens user session.

Response: returns an items array of ModelServiceGroup payloads, each with an added clusterName.

{
  "items": [
    {
      "id": "msg-4a8b41f3dd6c",
      "name": "qwen3-8b",
      "modelRef": "Qwen/Qwen3-8B",
      "displayName": "Qwen3 8B",
      "description": "Shared Qwen 3 8B service",
      "selectionPolicy": "random",
      "status": "active",
      "clusterId": "cluster-a",
      "clusterName": "Cluster A",
      "servedName": "Qwen/Qwen3-8B",
      "baseModel": "Qwen/Qwen3-8B",
      "createdByUserId": "user-123",
      "createdAt": "2026-09-20T15:00:00Z",
      "updatedAt": "2026-09-22T11:30:00Z"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
entries = service.list_model_entries_for_user("user-123")
for entry in entries:
    print(entry["name"], entry["clusterName"])

Get gateway connection info

Return the Gateway connection info a client copies into its OpenAI client. Each cluster runs its own Gateway, so the response lists one connection per cluster the caller can access; LENS_MODEL_GATEWAY_PUBLIC_URL, when set, is a single override applied to every cluster.

REST API

GET /api/v1/model-service/connection

Request: no request body.

Response: the global override (empty string when unset) plus a per-cluster list.

{
  "baseUrl": "",
  "clusters": [
    {
      "clusterId": "cluster-a",
      "clusterName": "Cluster A",
      "provider": "istio",
      "gatewayName": "lens-inference-gateway",
      "address": "10.0.0.10",
      "publicUrl": null,
      "port": null,
      "baseUrl": "http://10.0.0.10/v1",
      "ready": true
    }
  ]
}

Python API

import os

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService, gateway_base_url

global_url = os.environ.get("LENS_MODEL_GATEWAY_PUBLIC_URL", "").strip()
public_host = os.environ.get("LENS_PUBLIC_HOST", "").strip()
status = asyncio.run(GatewayOpsService().status())
clusters = [
    {
        "clusterId": cluster.cluster_id,
        "clusterName": cluster.cluster_name,
        "provider": cluster.gateway_provider,
        "gatewayName": cluster.gateway_name,
        "address": cluster.gateway_address,
        "publicUrl": cluster.gateway_public_url,
        "port": cluster.gateway_port,
        "baseUrl": gateway_base_url(cluster, public_host, global_url),
        "ready": bool(cluster.gateway_ready),
    }
    for cluster in status.clusters
]

print({"baseUrl": global_url, "clusters": clusters})

List OpenAI-compatible models

Return an OpenAI-style GET /v1/models response for a valid model token. The model list is filtered to the groups visible to that token's owning user.

REST API

GET /v1/models

Request: no request body. Pass the model token with either Authorization: Bearer <token> or x-api-key: <token>.

Response: standard OpenAI-compatible model-list payload.

{
  "object": "list",
  "data": [
    {
      "id": "qwen3-8b",
      "object": "model",
      "created": 1727107200,
      "owned_by": "lens"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
record = service.tokens.validate("lens-mk-bTj9X7mW4zP1qN8sR6uC2yF5hL0aV3dK1mQ", ip="10.0.0.25")
if record is None:
    raise RuntimeError("token is missing, expired, or revoked")

payload = {
    "object": "list",
    "data": [
        {
            "id": group.name,
            "object": "model",
            "created": int(group.created_at.timestamp()),
            "owned_by": "lens",
        }
        for group in service.list_models_for_user(record.user_id)
    ],
}
print(payload)

List usage records

List the caller's recent usage records, with simple paging through limit and offset. Each item is the stored UsageRecord projected with its camelCase API aliases. The llm-d EPP reports only input/output tokens, so cachedInputTokens is currently always 0 (the field is kept for engines that expose it).

REST API

GET /api/v1/model-service/usage/records

Request: no request body.

• limit (integer, query, default 100, max 500): number of records to return.
• offset (integer, query, default 0): zero-based offset into the caller's ledger.

Response: an items array of usage records.

{
  "items": [
    {
      "id": "usg-8c2d9e1fa410",
      "requestId": "req-20260923-0001",
      "userId": "user-123",
      "tokenId": "mat-2c47de48a35b",
      "groupId": "msg-4a8b41f3dd6c",
      "groupName": "qwen3-8b",
      "clusterId": "cluster-a",
      "executionId": "exec-qwen3-8b",
      "modelRef": "Qwen/Qwen3-8B",
      "provider": "vllm",
      "inputTokens": 512,
      "cachedInputTokens": 128,
      "cacheWriteTokens": 0,
      "outputTokens": 196,
      "usageSource": "engine",
      "status": "success",
      "streaming": false,
      "ttftMs": 240,
      "durationMs": 1310,
      "createdAt": "2026-09-23T08:40:00Z"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
records = service.usage.list_for_user("user-123", limit=100, offset=0)
for record in records:
    print(record.api_payload())

Get daily usage totals

Aggregate the caller's recent usage records into daily token totals. The handler reuses the same usage ledger and buckets it with _daily_totals.

REST API

GET /api/v1/model-service/usage/timeseries

Request: no request body.

• days (integer, query, default 30, range 1..365): number of recent days to include.

Response: daily usage buckets with totals for each token class.

{
  "items": [
    {
      "date": "2026-09-22",
      "requests": 4,
      "inputTokens": 1200,
      "cachedInputTokens": 300,
      "cacheWriteTokens": 0,
      "outputTokens": 640
    },
    {
      "date": "2026-09-23",
      "requests": 2,
      "inputTokens": 530,
      "cachedInputTokens": 64,
      "cacheWriteTokens": 0,
      "outputTokens": 211
    }
  ]
}

Python API

from datetime import datetime, timedelta

from llm_d_lens.model_service.router import _daily_totals
from llm_d_lens.model_service.service import default_service

service = default_service()
since = datetime.now() - timedelta(days=30)
records = service.usage.list_for_user("user-123", since=since, limit=10000)
print(_daily_totals(records))

List model routing groups

List configured model groups in the caller's cluster-scoped read authority. Each group is a public model name routing to deployment-owned InferencePools. GET /api/v1/model-service/admin/groups/{group_id} enforces the same scoped model-service:group:read permission. Group payloads also include nullable ownerUserId and ownerGroupId; ownership alone does not grant consumer inference.

Model inference grants

All access-management endpoints require model-service:group:share in the model's cluster. Admin and maintainer have this permission; it does not imply group or Gateway management. Grants cover every provider of the model, not its management APIs. Model deletion revokes grants; renaming preserves them. An empty model cannot move clusters while grants remain.

REST API

Method	Path below /api/v1/model-service/admin/groups/{group_id}	Result
GET	/access	{"items":[...]} with grant IDs, subjectType, subjectId, subjectName, permission, createdAt
GET	/access/options	{"users":[{"id":"user-123","username":"alice"}],"groups":[{"id":"group-123","name":"team"}]}
POST	/access	HTTP 201 with grant ID, recipient and inference permission
DELETE	/access/{binding_id}	HTTP 204, empty body

POST body:

{"subjectType":"user","subjectId":"user-123"}

Response example:

{"id":"binding-123","subjectType":"user","subjectId":"user-123","permission":"model-service:inference:use"}

Use subjectType: "group" for a group. No role, cluster or owner field is accepted. The server fixes the grant role and derives the cluster from the stored model. Recipients must already have inference permission in that cluster. The options endpoint exposes only eligible recipient IDs and display names, not full identity records. Disabled/protected users are not selectable.

Insufficient scope returns 403, missing resources or a mismatched revoke binding return 404, duplicates return 409, and invalid/extra fields return 422. Revocation re-evaluates subsequent requests using existing API keys without interrupting already-running streams or changing HTTPRoute weights.

Python API

from llm_d_lens.model_service.contracts import ModelGrantRequest
from llm_d_lens.model_service.service import default_service

def grant_model(model_id: str, user_id: str, principal):
    service = default_service()
    return service.grant_access(
        model_id,
        ModelGrantRequest(subject_type="user", subject_id=user_id),
        principal=principal,
    )

def read_model_access(model_id: str, principal):
    service = default_service()
    return (
        service.list_access(model_id, principal=principal),
        service.access_options(model_id, principal=principal),
    )

def revoke_model(model_id: str, binding_id: str, principal):
    default_service().revoke_access(model_id, binding_id, principal=principal)

Pass the authenticated caller's resolved principal. These methods enforce grant-management action and model cluster scope, including for internal callers.

List model routing groups: examples

REST API

GET /api/v1/model-service/admin/groups

Request: no request body.

Response: returns an items array of ModelServiceGroup payloads.

{
  "items": [
    {
      "id": "msg-4a8b41f3dd6c",
      "name": "qwen3-8b",
      "modelRef": "Qwen/Qwen3-8B",
      "clusterId": "cluster-a",
      "servedName": "Qwen/Qwen3-8B",
      "baseModel": "Qwen/Qwen3-8B",
      "displayName": "Qwen3 8B",
      "description": "Shared Qwen 3 8B service",
      "selectionPolicy": "random",
      "status": "active",
      "createdByUserId": "user-123",
      "createdAt": "2026-09-20T15:00:00Z",
      "updatedAt": "2026-09-22T11:30:00Z"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
for group in service.list_groups():
    print(group.api_payload())

Create a model routing group

Create a routing group that exposes one user-facing model name in a single cluster. name and clusterId are required; the public name is unique per cluster (clusterId) and every provider must be a deployment in that same cluster. Supplying executionId creates the group and its initial member in one transaction, as used by Add model. The deployment must be ready and have a model name. Lens derives modelRef and servedName from that deployment and sets baseModel to the public name, ignoring caller overrides for those three fields. Without executionId, standalone group creation remains supported.

REST API

POST /api/v1/model-service/admin/groups

Request body (GroupCreateRequest):

• name (string): the public model name clients use in Chat Completions API requests.
• clusterId (string): the cluster the service belongs to (required); its providers must be in this cluster and the name is unique within it.
• displayName (string, optional): UI-friendly label; defaults to name.
• description (string, optional): free-form description.
• executionId (string, optional): ready deployment to publish as the first member. Requires both model-service:group:manage and model-service:member:manage in the deployment's cluster.
• modelRef / servedName / baseModel (string, optional, advanced): with executionId, derived server-side. Without it, modelRef defaults to name, baseModel to modelRef, and servedName to null.
• selectionPolicy (string, legacy): accepted for compatibility; the llm-d data plane ignores it.

{
  "name": "qwen3-8b",
  "clusterId": "cluster-a",
  "executionId": "exec-123",
  "displayName": "Qwen3 8B",
  "description": "Shared Qwen 3 8B service"
}

Response: 201 with the created ModelServiceGroup. When executionId is supplied, its member is also committed and gateway reconciliation is scheduled in the background. Errors: 403 for missing management permissions or scope, 404 for a missing deployment target, 409 for a duplicate name, cluster mismatch, non-ready deployment or missing deployment model name, and 422 for invalid input. A failed transaction leaves neither record behind.

{
  "id": "msg-4a8b41f3dd6c",
  "name": "qwen3-8b",
  "modelRef": "Qwen/Qwen3-8B",
  "clusterId": "cluster-a",
  "servedName": "Qwen/Qwen3-8B",
  "baseModel": "qwen3-8b",
  "displayName": "Qwen3 8B",
  "description": "Shared Qwen 3 8B service",
  "selectionPolicy": "random",
  "status": "active",
  "createdByUserId": "user-123",
  "createdAt": "2026-09-23T09:10:00Z",
  "updatedAt": "2026-09-23T09:10:00Z"
}

Python API

from llm_d_lens.model_service.contracts import GroupCreateRequest
from llm_d_lens.model_service.service import default_service

service = default_service()
group = service.create_group(
    GroupCreateRequest(
        name="qwen3-8b",
        clusterId="cluster-a",
        executionId="exec-123",
        displayName="Qwen3 8B",
        description="Shared Qwen 3 8B service",
    ),
    created_by="user-123",
)
print(group.api_payload())
print([member.api_payload() for member in service.list_members(group.id)])

exec-123 must identify an existing ready deployment in cluster-a. Trusted Python callers may omit principal; request adapters must pass the authenticated principal to enforce both permissions and cluster scope. create_group commits records only; the HTTP route additionally schedules gateway reconciliation.

Publish a ready deployment as a public model service

Publish one ready deployment as a new cluster-scoped public model name without granting full group-management rights. This is the route used by Add model when the public name does not exist yet. The caller needs model-service:member:manage in the deployment's cluster. The service uses the default random selection policy; editing group metadata or selection policy still requires model-service:group:manage.

POST /api/v1/model-service/admin/groups/publish

Request body (GroupPublishRequest):

• name (string): public model name.
• clusterId (string): deployment's cluster.
• executionId (string): ready deployment to publish.
• displayName / description (string, optional).

Response: 201 with the created ModelServiceGroup; the deployment is committed as its first member and gateway reconciliation is scheduled. Errors match group creation (403, 404, 409, 422).

{
  "name": "qwen3-8b",
  "clusterId": "cluster-a",
  "executionId": "exec-123"
}

List published model service members

List the published members that back routing groups. The route can optionally filter by one group id and adds a human-friendly deployment name field derived from execution_name().

REST API

GET /api/v1/model-service/admin/members

Request: no request body.

• group_id (string, query, optional): only return members for one routing group.

Response: returns an items array of member payloads, including integer weight (0 through 1000000).

{
  "items": [
    {
      "id": "msm-32d55cb7f9a1",
      "groupId": "msg-4a8b41f3dd6c",
      "executionId": "exec-qwen3-8b",
      "clusterId": "cluster-a",
      "targetNamespace": "ns",
      "targetService": "vllm",
      "targetPort": 8000,
      "endpointKind": "vllm",
      "weight": 1,
      "status": "active",
      "publishedByUserId": "user-123",
      "createdAt": "2026-09-23T09:12:00Z",
      "updatedAt": "2026-09-23T09:12:00Z",
      "name": "qwen-qwen3-8b-202609210558"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service
from llm_d_lens.model_service.targets import execution_name

service = default_service()
items = []
for member in service.list_members("msg-4a8b41f3dd6c"):
    payload = member.api_payload()
    payload["name"] = execution_name(member.execution_id) or member.execution_id
    items.append(payload)

print(items)

Publish a deployment as a member

Publish one ready deployment execution into a routing group. The route derives the actual namespace, service, port, and endpoint kind from the execution id. The execution's cluster must be the group's cluster, otherwise the call is rejected with 409 (model_member_conflict). Every new provider defaults to weight 1; existing provider weights are preserved. Use the traffic-distribution endpoint to change the allocation. The HTTP route schedules reconciliation; the direct Python call only persists.

REST API

POST /api/v1/model-service/admin/members

Request body (MemberCreateRequest):

• groupId (string): target routing group id.
• executionId (string): ready deployment execution id.
• poolName (string, optional, advanced): the deployment-owned InferencePool name; defaults to the deployment's pool stored as namespace/pool (unique across namespaces).

{
  "groupId": "msg-4a8b41f3dd6c",
  "executionId": "exec-qwen3-8b",
  "poolName": "ns/qwen"
}

Response: the created ModelServiceMember.

{
  "id": "msm-32d55cb7f9a1",
  "groupId": "msg-4a8b41f3dd6c",
  "executionId": "exec-qwen3-8b",
  "clusterId": "cluster-a",
  "targetNamespace": "ns",
  "targetService": "qwen-epp",
  "targetPort": 8000,
  "endpointKind": "llm-d-epp",
  "weight": 1,
  "poolName": "qwen",
  "eppRef": "qwen-epp",
  "status": "active",
  "ownerUserId": "user-123",
  "publishedByUserId": "user-123",
  "createdAt": "2026-09-23T09:12:00Z",
  "updatedAt": "2026-09-23T09:12:00Z"
}

Python API

from llm_d_lens.model_service.contracts import MemberCreateRequest
from llm_d_lens.model_service.service import default_service

service = default_service()
member = service.create_member(
    MemberCreateRequest(
        groupId="msg-4a8b41f3dd6c",
        executionId="exec-qwen3-8b",
        poolName="ns/qwen",
    ),
    published_by="user-123",
)
print(member.api_payload())

Save and apply traffic distribution

Atomically replace all weights and optionally add deployment providers, then reconcile the cluster's shared Gateway and weighted HTTPRoute. The model-service:member:manage permission is required in the group's cluster.

REST API

PUT /api/v1/model-service/admin/groups/{group_id}/traffic

Request body (TrafficUpdateRequest):

• members: complete list of 1 to 16 entries with executionId and integer weight in 0..1000000. Existing providers cannot be omitted; use member deletion separately.
• expectedWeights: previously read execution-ID-to-weight map, including every existing provider and excluding new providers.

{
  "members": [
    {"executionId": "stable", "weight": 90},
    {"executionId": "canary", "weight": 10}
  ],
  "expectedWeights": {"stable": 1}
}

New deployments and zero-weight providers being activated must be ready, in the same cluster, backed by distinct InferencePools/EPPs, and use the group's provider model name. At least one non-disabled provider must have positive weight. Zero-weight members remain stored but are not rendered in backendRefs; disabled members also receive no traffic.

Response: 200 with items (full updated member payloads, including weight) and operation (a GatewayOperation with status, message, and detailJson). A failed operation does not roll back the saved weights. succeeded means the configuration was submitted, not that request counts have verified the split.

Validation returns 422 for malformed/range-invalid values, 409 (model_traffic_conflict) for stale snapshots or invalid allocations, 404 for missing targets, and 403 for insufficient permissions. A transport/domain exception after saving returns 502 (model_traffic_apply_failed) explicitly stating that configuration was saved. Reload the allocation and retry Reconcile after resolving the failure.

Python API

ModelService.update_traffic() validates and persists; call GatewayOpsService.reconcile_cluster_gateway() separately to submit it. Obtain the principal from the authenticated request, not user-supplied identity.

from llm_d_lens.auth.access import current_principal
from llm_d_lens.model_service.contracts import TrafficUpdateRequest
from llm_d_lens.model_service.gateway_ops import GatewayOpsService
from llm_d_lens.model_service.service import default_service

async def configure_traffic(request, group_id, canary_execution_id):
    service = default_service()
    principal = current_principal(request)
    existing = service.list_members(group_id)
    weights = {member.execution_id: member.weight for member in existing}
    members = [{"executionId": member.execution_id, "weight": member.weight}
               for member in existing]
    members[0]["weight"] = 90
    members.append({"executionId": canary_execution_id, "weight": 10})
    saved = service.update_traffic(
        group_id,
        TrafficUpdateRequest(members=members, expectedWeights=weights),
        principal=principal,
    )
    operation = await GatewayOpsService().reconcile_cluster_gateway(
        saved[0].cluster_id, actor=principal.user_id,
    )
    return {"items": [member.api_payload() for member in saved],
            "operation": operation.api_payload()}

The renderer render_model_route(..., pools=[PoolBinding(...)]) emits exact relative weights, one IPP model map, and cross-namespace ReferenceGrants. Its existing pool=PoolBinding(...) single-backend calling convention remains supported. Do not pass both forms. PoolBinding.weight defaults to 1.

Get usage analytics

Return aggregated model-service usage analytics for the admin dashboard. The payload includes overall totals, a time series, grouped breakdowns, and filter-facet options.

REST API

GET /api/v1/model-service/admin/usage/analytics

Request: no request body.

• since (string, query, optional): ISO timestamp lower bound.
• until (string, query, optional): ISO timestamp upper bound.
• interval (string, query, default day): day or hour.
• group_by (string, query, default model): model, user, or cluster.
• user_id (string[], query, optional): repeated user ids.
• group_id (string[], query, optional): repeated group ids.
• cluster_id (string[], query, optional): repeated cluster ids.

Response: analytics built by build_usage_analytics().

{
  "interval": "day",
  "groupBy": "model",
  "totals": {
    "requests": 3,
    "inputTokens": 17,
    "cachedInputTokens": 2,
    "outputTokens": 6,
    "tokens": 25
  },
  "series": [
    {
      "bucket": "2026-09-21",
      "requests": 2,
      "inputTokens": 13,
      "cachedInputTokens": 2,
      "outputTokens": 6,
      "tokens": 21
    },
    {
      "bucket": "2026-09-22",
      "requests": 1,
      "inputTokens": 4,
      "cachedInputTokens": 0,
      "outputTokens": 0,
      "tokens": 4
    }
  ],
  "groups": [
    {
      "key": "m1",
      "label": "m1",
      "totals": {
        "requests": 2,
        "inputTokens": 13,
        "cachedInputTokens": 2,
        "outputTokens": 6,
        "tokens": 21
      },
      "series": [
        {
          "bucket": "2026-09-21",
          "requests": 2,
          "inputTokens": 13,
          "cachedInputTokens": 2,
          "outputTokens": 6,
          "tokens": 21
        }
      ]
    }
  ],
  "facets": {
    "models": [
      {
        "key": "g-m1",
        "label": "m1"
      }
    ],
    "users": [
      {
        "key": "u1",
        "label": "u1"
      }
    ],
    "clusters": [
      {
        "key": "c1",
        "label": "c1"
      }
    ]
  }
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
analytics = service.usage_analytics(
    since=None,
    until=None,
    interval="day",
    group_by="model",
    user_ids=None,
    group_ids=None,
    cluster_ids=["cluster-a"],
    principal=None,
)
print(analytics)

List usage records across clusters

List recent usage records across users/models/clusters for the admin dashboard, optionally filtered by routing group or cluster and capped by limit. Cluster visibility is still narrowed to the caller's accessible clusters.

REST API

GET /api/v1/model-service/admin/usage

Request: no request body.

• group_id (string[], query, optional): repeated routing-group ids.
• cluster_id (string[], query, optional): repeated cluster ids.
• limit (integer, query, default 100, max 500): number of records to return.

Response: an items array of usage records (same shape as /usage/records).

{
  "items": [
    {
      "id": "usg-8c2d9e1fa410",
      "requestId": "req-20260923-0001",
      "userId": "user-123",
      "tokenId": "mat-2c47de48a35b",
      "groupId": "msg-4a8b41f3dd6c",
      "groupName": "qwen3-8b",
      "clusterId": "cluster-a",
      "executionId": "exec-qwen3-8b",
      "modelRef": "Qwen/Qwen3-8B",
      "provider": "vllm",
      "inputTokens": 512,
      "cachedInputTokens": 0,
      "cacheWriteTokens": 0,
      "outputTokens": 196,
      "usageSource": "engine",
      "status": "success",
      "streaming": false,
      "ttftMs": 240,
      "durationMs": 1310,
      "createdAt": "2026-09-23T08:40:00Z"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
records = service.usage.list_all(group_ids=None, cluster_ids=["cluster-a"], limit=100)
for record in records:
    print(record.api_payload())

Get gateway status

Get the current gateway status across clusters: published members, the pinned provider, and the shared Gateway's programmed state. gatewayNamespace and ippNamespace contain the effective namespace including the stable cluster-registration short ID.

REST API

GET /api/v1/model-service/admin/gateway/status

Request: no request body.

Response: serialized GatewayStatus payload.

{
  "clusters": [
    {
      "clusterId": "1234abcd",
      "clusterName": "Primary cluster",
      "members": [
        {
          "groupId": "msg-4a8b41f3dd6c",
          "groupName": "qwen3-8b",
          "executionId": "exec-qwen3-8b",
          "endpointKind": "llm-d-epp",
          "poolName": "qwen",
          "target": "qwen-epp.ns:8000",
          "status": "active"
        }
      ],
      "gatewayProvider": "istio",
      "gatewayState": "installed",
      "gatewayNamespace": "lens-gateway-1234abcd",
      "gatewayName": "lens-inference-gateway",
      "gatewayReady": true,
      "gatewayRunning": true,
      "ippState": "installed",
      "ippReady": false,
      "ippName": "lens-ipp",
      "ippNamespace": "lens-gateway-1234abcd",
      "lastOperation": {
        "id": "gop-2a7f0d9c0c11",
        "kind": "reconcile-gateway",
        "status": "succeeded",
        "clusterId": "1234abcd",
        "message": "gateway reconciled (istio); 1 model route(s)",
        "createdAt": "2026-09-23T09:15:00Z",
        "finishedAt": "2026-09-23T09:15:18Z"
      }
    }
  ]
}

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

async def main() -> None:
    status = await GatewayOpsService().status()
    print(status.api_payload())

asyncio.run(main())

Reconcile gateway clusters

Reconcile one gateway cluster with logs

Re-render and reapply one cluster's shared Gateway, HTTPRoutes and provider wiring. This is available to scoped maintainers and streams operation logs as SSE.

POST /api/v1/model-service/admin/gateway/reconcile/stream

Request body: { "clusterId": "cluster-a" }. Requires model-service:gateway:reconcile.

Schedule a full data-plane reconciliation for every cluster that currently has active members. The route discovers the affected cluster ids, then runs _reconcile_clusters() in the background.

REST API

POST /api/v1/model-service/admin/gateway/reconcile

Request: no request body.

Response: confirms that reconciliation was scheduled and lists the target clusters.

{
  "scheduled": true,
  "clusters": [
    "cluster-a",
    "cluster-b"
  ]
}

Python API

import asyncio

from llm_d_lens.model_service.router import _reconcile_clusters

async def main() -> None:
    await _reconcile_clusters(["cluster-a", "cluster-b"])

asyncio.run(main())

Install or reuse the shared Gateway

Install (or reuse) the cluster's shared llm-d Gateway. The provider is user-selectable (istio, gke, agentgateway, envoy-ai-gateway); when omitted, the cluster record's pinned provider is used. The namespace defaults to lens-gateway-<cluster-id>; custom prefixes receive the same stable short-ID suffix. Gateway and IPP operations resolve that namespace consistently. Istio authentication uses a namespace-local EnvoyFilter scoped to this Gateway's pods, with fail-closed HTTP external authorization; installation does not register a shared mesh-wide auth provider.

Existing unsuffixed resources are not adopted or removed. A NodePort owned by the old Gateway remains a conflict until an operator selects an available port or deliberately retires that old Gateway. Uninstall through the operation stream deletes only this registration's labelled namespace-local Gateway resources, never another namespace, a shared GatewayClass, or deployment-owned InferencePools.

REST API

POST /api/v1/model-service/admin/gateway/install

Request body (ClusterGatewayRequest):

• clusterId (string): target cluster id.
• provider (string, optional): istio | gke | agentgateway | envoy-ai-gateway.

{ "clusterId": "1234abcd", "provider": "istio" }

Response: recorded GatewayOperation for the install.

{
  "id": "gop-9c1a2b3d4e5f",
  "kind": "install-gateway",
  "status": "succeeded",
  "clusterId": "1234abcd",
  "message": "shared Gateway ready (istio)",
  "detailJson": { "provider": "istio", "namespace": "lens-gateway-1234abcd", "name": "lens-inference-gateway", "programmed": true },
  "createdAt": "2026-09-24T16:20:00Z",
  "finishedAt": "2026-09-24T16:20:12Z"
}

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

async def main() -> None:
    operation = await GatewayOpsService().install_cluster_gateway("1234abcd", provider="istio")
    print(operation.api_payload())

asyncio.run(main())

Get a model routing group

Fetch one routing group by id.

REST API

GET /api/v1/model-service/admin/groups/{group_id}

Request: no request body.

Response: the ModelServiceGroup payload (same shape as the list endpoint).

{
  "id": "msg-4a8b41f3dd6c",
  "name": "qwen3-8b",
  "modelRef": "Qwen/Qwen3-8B",
  "clusterId": "cluster-a",
  "servedName": "Qwen/Qwen3-8B",
  "baseModel": "Qwen/Qwen3-8B",
  "selectionPolicy": "random",
  "status": "active"
}

Python API

from llm_d_lens.model_service.service import default_service

group = default_service().get_group("msg-4a8b41f3dd6c")
print(group.api_payload() if group else None)

Update a model routing group

Update mutable fields of a routing group (name, model references, display name, description, status, cluster, served name, base model). The HTTP handler saves configuration and awaits reconciliation of the affected clusters. status: "disabled" stops the public model and removes its route; "active" restores routes to enabled, positive-weight providers. Provider status, weights and running Pods are unchanged. The caller needs group-management permission in every affected cluster. Controller convergence is asynchronous.

REST API

PATCH /api/v1/model-service/admin/groups/{group_id}

Request body (GroupUpdateRequest; all fields optional): name, modelRef, displayName, description, status (active | disabled), clusterId, servedName, baseModel.

{ "displayName": "Qwen3 8B (prod)", "status": "active" }

Response: the updated ModelServiceGroup after successful reconciliation submission. 502 model_publication_apply_failed means configuration was saved but routing could not be applied; use Reconcile to retry. 403 rejects out-of-scope changes.

Python API

from llm_d_lens.model_service.contracts import GroupUpdateRequest
from llm_d_lens.model_service.service import default_service

group = default_service().update_group(
    "msg-4a8b41f3dd6c",
    GroupUpdateRequest(displayName="Qwen3 8B (prod)", status="active"),
)
print(group.api_payload())

update_group(..., principal=principal) enforces the caller's cluster-scoped management permission. This service method only persists configuration; direct Python callers must also await GatewayOpsService().reconcile_cluster(group.cluster_id) to apply routing and check the returned operation status. Trusted internal callers may omit principal; HTTP requests always supply it.

Enable or disable a model service

Start or stop serving a public model without changing provider states, weights or running Pods.

POST /api/v1/model-service/admin/groups/{group_id}/status

Request body: { "status": "active" } or { "status": "disabled" }. Requires model-service:group:toggle in the model's cluster.

Delete a model routing group

Remove an empty routing group with cluster-scoped group-management permission. Any attached provider, even offline or weight 0, returns 409 model_group_conflict. The handler serializes with Gateway reconciliation, removes the model's Lens-managed HTTPRoute and IPP mapping, and confirms deletion before deleting the record. Missing models return 404; cleanup failure returns 502 model_removal_failed and retains the record for retry. Other models, shared Gateway resources, deployments and Pods are not removed.

REST API

DELETE /api/v1/model-service/admin/groups/{group_id}

Requires model-service:group:remove. Request: no request body. Response: 204 No Content.

Python API

from llm_d_lens.cluster.registry import gateway_lock
from llm_d_lens.model_service.gateway_ops import GatewayOpsService
from llm_d_lens.model_service.service import default_service

async def remove_model(group_id, principal):
    service = default_service()
    group = service.require_empty_group(group_id, principal=principal)
    async with gateway_lock(group.cluster_id):
        group = service.require_empty_group(group_id, principal=principal)
        await GatewayOpsService().remove_model_route(group)
        service.delete_group(group_id, principal=principal)

delete_group() alone only removes the database record and rechecks empty membership transactionally. Use the complete sequence above for live models; cleanup exceptions must stop deletion. Trusted internal callers may omit the principal in the service calls; HTTP requests always supply it.

Start or stop all providers

Requires member-management permission in the model's cluster. Changes all provider statuses atomically, then reconciles routing once per affected cluster. Weights, parent model status and Pods are unchanged. Starting requires every deployment to be ready in the registered cluster. An empty model, unready deployment or concurrent provider change returns 409 model_member_conflict without partial status updates; missing models return 404. A 502 model_publication_apply_failed means statuses were saved but route application failed: retry Reconcile.

REST API

PUT /api/v1/model-service/admin/groups/{group_id}/providers/status

{ "status": "disabled" }

Use "active" to start all providers or "disabled" to stop all providers. Success: 200 with {"items": [...]} containing the updated member payloads.

Python API

from llm_d_lens.model_service.contracts import ProvidersStatusRequest
from llm_d_lens.model_service.gateway_ops import GatewayOpsService
from llm_d_lens.model_service.service import default_service

async def stop_providers(group_id, principal):
    members = default_service().set_providers_status(
        group_id, ProvidersStatusRequest(status="disabled"), principal=principal
    )
    for cluster_id in {member.cluster_id for member in members}:
        operation = await GatewayOpsService().reconcile_cluster(
            cluster_id, actor=principal.user_id
        )
        if operation.status != "succeeded":
            raise RuntimeError(operation.message)
    return members

set_providers_status() persists only; direct callers must reconcile and inspect the operation result as above.

List publishable deployments

List ready deployments (executions) that can be published as members, with the derived target already filled in.

REST API

GET /api/v1/model-service/admin/deployments

Request: no request body.

Response: returns an items array of PublishableDeployment payloads.

{
  "items": [
    {
      "executionId": "exec-qwen3-8b",
      "name": "qwen-qwen3-8b-202609210558",
      "displayName": "Qwen3 8B",
      "model": "Qwen/Qwen3-8B",
      "clusterId": "cluster-a",
      "clusterName": "Cluster A",
      "namespace": "ns",
      "service": "qwen-epp",
      "port": 8000,
      "endpointKind": "llm-d-epp",
      "modelRef": "Qwen/Qwen3-8B",
      "status": "ready"
    }
  ]
}

Python API

from llm_d_lens.model_service.service import default_service

for item in default_service().list_publishable_deployments():
    print(item.api_payload())

Update a member

Update a published member's status and await cluster routing reconciliation. disabled takes the provider offline without changing its saved weight or scaling its Pods. Re-enabling requires a ready deployment in its registered cluster. Weight 0 remains standby, and a disabled parent group stays stopped. All providers may be offline; with no enabled positive weights the HTTPRoute is removed. Otherwise it references all enabled positive-weight InferencePools. Health probes cannot overwrite an explicit offline decision.

REST API

PATCH /api/v1/model-service/admin/members/{member_id}

Request body (MemberUpdateRequest; all fields optional): status (active | disabled | unhealthy).

{ "status": "disabled" }

Response: the updated ModelServiceMember after successful reconciliation submission. 409 model_member_conflict rejects an unready provider being brought online; 403 rejects callers without member-management permission in its cluster. 502 model_publication_apply_failed means the state was saved but routing could not be applied; use Reconcile to retry.

Python API

from llm_d_lens.model_service.contracts import MemberUpdateRequest
from llm_d_lens.model_service.service import default_service

member = default_service().update_member("msm-32d55cb7f9a1", MemberUpdateRequest(status="disabled"))
print(member.api_payload())

Pass principal=principal for caller-scoped authorization. This method persists state only; direct Python callers must also await GatewayOpsService().reconcile_cluster(member.cluster_id) and check its operation status. Neither method scales the provider's Pods.

Remove a member

Remove a published member. The handler schedules a reconcile of its cluster.

REST API

DELETE /api/v1/model-service/admin/members/{member_id}

Request: no request body. Response: 204 No Content.

Python API

from llm_d_lens.model_service.service import default_service

default_service().delete_member("msm-32d55cb7f9a1")

Probe member backends

Probe non-disabled members of enabled models and refresh health_json / last_health_at, flipping status between active and unhealthy. Stopped models and offline providers are skipped; concurrent health writes cannot re-enable an offline provider. When no provider has an enabled positive weight, the missing HTTPRoute is expected, Gateway serving checks are skipped, and the model does not trigger auto-healing. An existing zero-weight InferencePool without Gateway parent status reports healthJson.components.inferencePool: "standby". This expected absence of routing does not make an otherwise healthy provider unhealthy or trigger Gateway/IPP auto-healing. Missing resources, explicit pool rejection, and EPP/model-server endpoint failures still report unhealthy; positive-weight pools still require Gateway acceptance.

REST API

POST /api/v1/model-service/admin/gateway/probe

Request: no request body.

Response:

{ "updated": 2 }

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

updated = asyncio.run(GatewayOpsService().probe_members())
print(updated)

Sync gateway usage

Read each cluster's llm-d EPP token counters from Prometheus and write the delta since the last sync into the usage ledger, attributed to the token/user behind the request's fairness_id (the Gateway injects x-llm-d-inference-fairness-id from the ext_authz decision). Idempotent: unchanged counters add nothing.

REST API

POST /api/v1/model-service/admin/gateway/usage/sync

Request: no request body.

Response: records written per cluster.

{ "written": { "cluster-a": 3 } }

Python API

import asyncio

from llm_d_lens.model_service.service import default_service
from llm_d_lens.model_service.usage_sync import GatewayUsageSync

written = asyncio.run(GatewayUsageSync(service=default_service()).sync())
print(written)

Read the Inference Payload Processor config

Read a cluster's live Inference Payload Processor config from its ConfigMap (Lens does not store it).

REST API

GET /api/v1/model-service/admin/gateway/ipp-config?clusterId=cluster-a

Response:

{ "config": "apiVersion: llm-d.ai/v1alpha1\nkind: PayloadProcessorConfig\nplugins: []\n", "custom": false, "default": "plugins: []\n" }

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

config = asyncio.run(GatewayOpsService().get_ipp_config("cluster-a"))
print(config["custom"], config["config"])

Save the Inference Payload Processor config

Apply a PayloadProcessorConfig body to the cluster and upgrade IPP (no Lens storage). Empty config restores the chart default.

REST API

POST /api/v1/model-service/admin/gateway/ipp-config

Request body (IppConfigRequest):

• clusterId (string): target cluster id.
• config (string): PayloadProcessorConfig body (starts at plugins:; the chart adds apiVersion/kind).

{ "clusterId": "cluster-a", "config": "plugins:\n- type: body-field-to-header\n" }

Response: a recorded GatewayOperation for the IPP install.

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

operation = asyncio.run(GatewayOpsService().set_ipp_config("cluster-a", "plugins:\n- type: body-field-to-header\n"))
print(operation.status)

Preflight a cluster

Check that the cluster is reachable and has active members before installing or reconciling the data plane.

REST API

POST /api/v1/model-service/admin/gateway/preflight

Request body (ClusterGatewayRequest):

• clusterId (string): target cluster id.

{ "clusterId": "cluster-a" }

Response: a recorded GatewayOperation for the preflight.

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

operation = asyncio.run(GatewayOpsService().preflight("cluster-a"))
print(operation.api_payload())

Scale a data-plane component

Scale a data-plane Deployment (Gateway data plane, EPP or model server) to a replica count.

REST API

POST /api/v1/model-service/admin/gateway/scale

Request body (ComponentScaleRequest):

• clusterId (string)
• namespace (string)
• name (string)
• replicas (integer, 0–100)
• kind (string, optional, default deployment)

{ "clusterId": "cluster-a", "namespace": "ns", "name": "qwen-epp", "replicas": 2 }

Response: the recorded GatewayOperation (detailJson carries namespace, name, replicas).

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

operation = asyncio.run(
    GatewayOpsService().scale_component("cluster-a", "ns", "qwen-epp", 2)
)
print(operation.api_payload())

Stream a data-plane operation

Run a data-plane operation and stream its log lines as Server-Sent Events. Used by the admin UI for install/uninstall/reconcile so the log drawer updates live.

REST API

POST /api/v1/model-service/admin/gateway/stream

Request body (GatewayStreamRequest):

• clusterId (string)
• op (string): install-gateway | uninstall-gateway | reconcile | scale | install-ipp | uninstall-ipp
• provider, namespace, name, replicas, kind, gatewayName, ippChart, ippVersion (optional, depending on op)

{ "clusterId": "cluster-a", "op": "reconcile" }

Response: text/event-stream; each event carries one log line, then the terminal GatewayOperation payload.

Python API

The stream is an HTTP transport; the equivalent Python calls run the operation directly.

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

async def main() -> None:
    ops = GatewayOpsService()
    await ops.reconcile_cluster_gateway("cluster-a")          # op=reconcile
    await ops.install_cluster_gateway("cluster-a")            # op=install-gateway
    await ops.uninstall_inference_gateway("cluster-a")        # op=uninstall-gateway
    await ops.install_ipp("cluster-a")                        # op=install-ipp
    await ops.uninstall_ipp("cluster-a")                      # op=uninstall-ipp

asyncio.run(main())

Stream a component's logs

Stream live logs (kubectl logs -f) for a data-plane component. Used by the admin data-plane diagram to watch Gateway / IPP / EPP output in real time.

REST API

POST /api/v1/model-service/admin/gateway/logs

Request body (ComponentLogsRequest):

• clusterId (string): target cluster.
• component (string, optional): gateway or ipp — the backend resolves the namespace/name.
• namespace / name (string, optional): explicit coordinates when no component is given.
• kind (string, default deployment), container (string, optional), tail (integer, default 200).

{ "clusterId": "cluster-a", "component": "gateway" }

Response: text/event-stream; each log event carries one line, then a terminal complete.

Python API

import asyncio

from llm_d_lens.model_service.gateway_ops import GatewayOpsService

async def main() -> None:
    namespace, name = await GatewayOpsService().resolve_log_target(
        "cluster-a", component="gateway", namespace=None, name=None
    )
    async for line in GatewayOpsService().stream_component_logs("cluster-a", namespace, name, tail=50):
        print(line)

asyncio.run(main())

Internal endpoints (Edge ext_authz)

These /api/v1/internal/model-gateway/* routes are called only by Edge Envoy's ext_authz filter and the llm-d EPP, never by Lens clients directly; they carry no Lens user session.

Authorize a gateway request

Validate the caller's model token, resolve the requested public model name to an active routing group the token may use, select one backend member, and return routing headers for Edge Envoy to attach upstream. GET /v1/models (no model) is token-checked only; every other HTTP method is accepted because Envoy forwards the client's original method.

REST API

POST /api/v1/internal/model-gateway/authorize (also GET/PUT/PATCH/DELETE, and /authorize/{rest:path} for Envoy's per-cluster path shape, e.g. .../authorize/cluster/<id>/...)

Request: a JSON body {"token": "...", "model": "..."}, or the raw OpenAI request body plus Authorization/x-api-key and x-lens-model/x-gateway-base-model-name headers.

Response: 200 with the AuthorizeResult payload and x-lens-* / x-ai-eg-model / x-llm-d-inference-fairness-id routing headers Edge forwards upstream; 401 (token_invalid) or 404 (model_not_found) on failure.

Python API

from llm_d_lens.model_service.contracts import AuthorizeRequest
from llm_d_lens.model_service.service import default_service

service = default_service()
result = service.authorize_request(
    AuthorizeRequest(token="lens-mk-bTj9X7mW4zP1qN8sR6uC2yF5hL0aV3dK1mQ", model="qwen3-8b"),
    cluster_id="cluster-a",
)
print(result.model_dump(mode="json", by_alias=True))

Record inference usage

Record one completed inference's token usage into the ledger. Called by the llm-d EPP/usage sync path; idempotent on requestId.

REST API

POST /api/v1/internal/model-gateway/record-usage

Request body (UsageRecordRequest): same fields as a UsageRecord (see "List usage records"), keyed by requestId.

Response:

{ "recorded": true, "id": "usg-8c2d9e1fa410" }

Python API

from llm_d_lens.model_service.contracts import UsageRecordRequest
from llm_d_lens.model_service.service import default_service

service = default_service()
record = service.record_usage(UsageRecordRequest(requestId="req-20260923-0001", inputTokens=512, outputTokens=196))
print(record.id)

List models a specific user may call

Return the model names a given user id may call, used to serve that user's /v1/models.

REST API

GET /api/v1/internal/model-gateway/models?user_id=user-123

Response: an items array of ModelServiceGroup payloads (same shape as "List accessible models").

Python API

from llm_d_lens.model_service.service import default_service

service = default_service()
for group in service.list_models_for_user("user-123"):
    print(group.api_payload())