Model cache

Backend API

Download, deletion and incremental node synchronization share launch_cache_jobs in llm_d_lens.model_cache.jobs. Public service calls below retain their existing signatures: callers select nodes and prepare credentials, while the launcher initializes progress and submits Jobs. Incremental synchronization preserves existing nodes; partial submission failures propagate without marking the whole batch in progress.

Routes live in llm_d_lens/model_cache/router.py. Each endpoint below documents the REST contract and the matching ModelCacheService or Hugging Face helper call it uses.

Model Cache owns deployment-model downloads. Shared creation preflight requires a ready model/revision in a registered cache on the selected cluster; local-disk entries must also be synchronized to the eligible nodes. llm_d_lens.deploy.providers.storage_mount.require_cached_model enforces this before rendering. During ordinary provider rendering, llm_d_lens.deploy.providers.model_cache_environment.model_cache_environment sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, including for legacy online inputs, and adds HF_HOME=/model-cache for the auto-cache source. Those serving mounts are read-only, and missing files fail offline resolution instead of being downloaded at startup. Exact published manifests bypass ordinary provider rendering and are not covered by the same final-manifest guarantees; see deployment limitations.

Hugging Face environment variables (HF_*, HUGGING*, TRANSFORMERS_*) are administrator-managed. The deployment planning API and configuration validation reject user-supplied customParameters entries that set them (see llm_d_lens/configuration/managed_environment.py), and the deployment editor blocks them with an "administrator-managed" message. The platform injects these values from the cluster's saved proxy, model cache and token configuration.

Deployment callers use require_cached_model(overrides, cluster_id=..., provider_id=...) before rendering. provider_id defaults to baseline-vllm; the shared deployment service supplies the actual provider. The check resolves revision, tokenizer, and tokenizer-revision separately for each active role, matches repository IDs and revisions exactly, and checks node synchronization for both model and tokenizer. Inactive-role arguments are ignored and conflicting dependency values are rejected. The manifest adapter then uses validate_cached_model_arguments to compare the rendered loading arguments against the checked dependencies, including published manifests. This argument check does not enforce published-manifest offline settings or storage binding. Unsupported dependency overrides are described in the deployment guide.

List Model Cache entries

List Model Cache entries, optionally filtered by cluster or storage volume. Use this to discover cached model ids, download state, and node sync drift.

REST API
Python API

GET /api/v1/model-cache/entries

Request: no request body. Optional query params: cluster_id (str) and storage_volume_id (str).

Response: returns an items array of Model Cache entries, with each item including the router-computed pendingSyncNodes field.

{
  "items": [
    {
      "id": "model-cache-7f3a2c1d",
      "clusterId": "cluster-123",
      "storageVolumeId": "storage-abc12345",
      "source": {
        "kind": "huggingface",
        "huggingface": {
          "repoId": "Qwen/Qwen2.5-7B-Instruct",
          "revision": "main"
        }
      },
      "tokenSource": {
        "mode": "none",
        "namespace": null,
        "name": null
      },
      "status": "ready",
      "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
      "sizeBytes": 15123042304,
      "nodeProgress": [
        {
          "node": "worker-1",
          "status": "ready",
          "failureDetail": null
        }
      ],
      "failureDetail": null,
      "createdAt": "2026-09-23T08:15:00Z",
      "updatedAt": "2026-09-23T08:22:11Z",
      "pendingSyncNodes": ["worker-2"]
    }
  ]
}

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entries = await service.list(cluster_id="cluster-123", storage_volume_id=None)
    pending = await service.compute_pending_sync(entries)
    payload = [
        {**entry.api_payload(), "pendingSyncNodes": pending.get(entry.id, [])}
        for entry in entries
    ]
    print(payload)


asyncio.run(main())

Get a Model Cache entry

Get one Model Cache entry by id, including source, token mode, node progress, and pending sync nodes.

REST API
Python API

GET /api/v1/model-cache/entries/{entry_id}

Request: no request body. Path param: entry_id (str).

Response: returns one Model Cache entry object plus the router-computed pendingSyncNodes list.

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "existing-secret",
    "namespace": "models",
    "name": "hf-token"
  },
  "status": "downloading",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": null,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "downloading",
      "failureDetail": null
    },
    {
      "node": "worker-2",
      "status": "pending",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:16:09Z",
  "pendingSyncNodes": []
}

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = await service.get("model-cache-7f3a2c1d")
    pending = await service.compute_pending_sync([entry])
    payload = {**entry.api_payload(), "pendingSyncNodes": pending.get(entry.id, [])}
    print(payload)


asyncio.run(main())

Request a model download

Request a model download into Lens Model Cache. Use this to seed a ready shared model cache before deployments or benchmarks.

REST API
Python API

POST /api/v1/model-cache/entries

Request: body fields are clusterId (str), storageVolumeId (str), source (ModelSource), and the optional force (bool, default false). tokenSource is accepted for backward compatibility but ignored: downloads always use the HuggingFace token saved with the cluster (the wizard's hf_token_secret), or run unauthenticated when the cluster has none. The frontend's Download model dialog no longer sends tokenSource at all.

If the storage volume already holds an entry for the same model and revision -- in any status -- the route answers 409 with code model_cache_entry_exists instead of downloading, so the caller can ask the user to confirm. The problem body carries entryId, entryStatus, model and storageVolumeId. Resend the same request with force: true to go ahead: a ready or failed entry is reset to pending and downloaded again in place, and a pending or downloading entry is returned unchanged so the confirmation joins the running download rather than starting a second one. No duplicate entry is ever created.

{
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  }
}

Response: returns the queued ModelCacheEntry payload that was created for the download request.

{
  "type": "about:blank",
  "status": 409,
  "title": "Model already on this storage volume",
  "detail": "'Qwen/Qwen2.5-7B-Instruct@main' is already on this storage volume (status: ready)",
  "code": "model_cache_entry_exists",
  "entryId": "model-cache-7f3a2c1d",
  "entryStatus": "ready",
  "model": "Qwen/Qwen2.5-7B-Instruct@main",
  "storageVolumeId": "storage-abc12345"
}

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "existing-secret",
    "namespace": "models",
    "name": "hf-token"
  },
  "status": "pending",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": null,
  "nodeProgress": [],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:15:00Z",
  "pendingSyncNodes": []
}

import asyncio

from llm_d_lens.model_cache.contracts import HuggingFaceSource, ModelCacheCreateRequest, ModelSource, ModelSourceKind
from llm_d_lens.model_cache.service import ModelCacheEntryExistsError, default_service


async def main() -> None:
    service = default_service()
    # tokenSource is intentionally omitted: request_download() always resolves the
    # HuggingFace token from the cluster's saved hf_token_secret instead.
    request = ModelCacheCreateRequest(
        clusterId="cluster-123",
        storageVolumeId="storage-abc12345",
        source=ModelSource(
            kind=ModelSourceKind.HUGGINGFACE,
            huggingface=HuggingFaceSource(repoId="Qwen/Qwen2.5-7B-Instruct", revision="main"),
        ),
    )
    try:
        entry = await service.request_download(request)
    except ModelCacheEntryExistsError as already_there:
        # Ask the operator whether to download it again, then retry with force.
        print(already_there.entry.id, already_there.entry.status)
        entry = await service.request_download(request.model_copy(update={"force": True}))
    await service.provision(entry.id)
    print(entry.id, entry.status)


asyncio.run(main())

Retry a model download

Retry a failed or retryable Model Cache entry download.

REST API
Python API

POST /api/v1/model-cache/entries/{entry_id}/retry

Request: no request body. Path param: entry_id (str).

Response: returns the updated entry after it has been moved back into a retryable download state.

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "none",
    "namespace": null,
    "name": null
  },
  "status": "pending",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": null,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "pending",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:30:00Z",
  "pendingSyncNodes": []
}

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = service.retry("model-cache-7f3a2c1d")
    await service.provision(entry.id)
    print(entry.api_payload())


asyncio.run(main())

Sync new nodes for all entries

Trigger sync for all local-disk Model Cache entries that are missing on newly added nodes. Optionally scope the work to one cluster.

REST API
Python API

POST /api/v1/model-cache/entries/sync-nodes

Request: no request body. Optional query param: cluster_id (str).

Response: returns the entries that were changed and queued for new-node provisioning.

{
  "items": [
    {
      "id": "model-cache-7f3a2c1d",
      "clusterId": "cluster-123",
      "storageVolumeId": "storage-abc12345",
      "source": {
        "kind": "huggingface",
        "huggingface": {
          "repoId": "Qwen/Qwen2.5-7B-Instruct",
          "revision": "main"
        }
      },
      "tokenSource": {
        "mode": "none",
        "namespace": null,
        "name": null
      },
      "status": "downloading",
      "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
      "sizeBytes": 15123042304,
      "nodeProgress": [
        {
          "node": "worker-1",
          "status": "ready",
          "failureDetail": null
        },
        {
          "node": "worker-2",
          "status": "pending",
          "failureDetail": null
        }
      ],
      "failureDetail": null,
      "createdAt": "2026-09-23T08:15:00Z",
      "updatedAt": "2026-09-23T08:35:00Z",
      "pendingSyncNodes": []
    }
  ]
}

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    changed = await service.sync_all_nodes(cluster_id="cluster-123")
    for entry in changed:
        await service.provision_new_nodes(entry.id)
    print([entry.id for entry in changed])


asyncio.run(main())

Sync new nodes for one entry

Trigger node sync for one specific Model Cache entry. Use this when a single cached model needs to be replicated onto newly added nodes.

REST API
Python API

POST /api/v1/model-cache/entries/{entry_id}/sync-nodes

Request: no request body. Path param: entry_id (str).

Response: returns the updated entry. When new nodes need provisioning, the entry status moves back to downloading.

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "none",
    "namespace": null,
    "name": null
  },
  "status": "downloading",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": 15123042304,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "ready",
      "failureDetail": null
    },
    {
      "node": "worker-2",
      "status": "pending",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:35:00Z",
  "pendingSyncNodes": []
}

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = await service.sync_nodes("model-cache-7f3a2c1d")
    if entry.status.value == "downloading":
        await service.provision_new_nodes(entry.id)
    print(entry.api_payload())


asyncio.run(main())

Delete a Model Cache entry

Delete a Model Cache entry and, when required, schedule cleanup of its downloaded files. Set keep_files=true to remove the record only.

REST API
Python API

DELETE /api/v1/model-cache/entries/{entry_id}

Request: no request body. Path param: entry_id (str). Optional query param: keep_files (bool, default false).

Response: usually returns the entry in deleting state while background cleanup runs. If the entry never reached a persisted cache path, the API may instead return a minimal {"id": ..., "status": "deleted"} acknowledgement.

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "none",
    "namespace": null,
    "name": null
  },
  "status": "deleting",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": 15123042304,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "ready",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T09:00:00Z",
  "pendingSyncNodes": []
}

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = await service.request_delete("model-cache-7f3a2c1d", keep_files=False)
    if entry is not None:
        await service.provision_delete(entry.id)
        print(entry.api_payload())
    else:
        print({"id": "model-cache-7f3a2c1d", "status": "deleted"})


asyncio.run(main())

Get Model Cache logs

Get logs for one Model Cache entry, optionally scoped to a specific node. Use this to debug download or cleanup failures.

REST API
Python API

GET /api/v1/model-cache/entries/{entry_id}/logs

Request: no request body. Path param: entry_id (str). Optional query param: node (str).

Response: returns the collected log text for the entry, or for the requested node only.

{
  "logs": "[worker-1] pulling Qwen/Qwen2.5-7B-Instruct@main\n[worker-1] download completed successfully"
}

import asyncio

from llm_d_lens.model_cache.service import default_service

async def main() -> None:
    service = default_service()
    logs = await service.get_logs("model-cache-7f3a2c1d", node="worker-1")
    print(logs)

asyncio.run(main())

Search Hugging Face models

Search Hugging Face model repositories from Lens Model Cache. Use this to discover repo ids before creating a cache entry.

REST API
Python API

GET /api/v1/model-cache/huggingface/search

Request: no request body. Query params: query (str) and optional limit (int, default 20).

Response: returns Hugging Face search matches as HuggingFaceModelSummary objects.

{
  "items": [
    {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "likes": 12345,
      "downloads": 987654,
      "pipelineTag": "text-generation",
      "tags": ["transformers", "text-generation", "safetensors"],
      "lastModified": "2026-09-01T12:00:00.000Z",
      "gated": false
    }
  ]
}

import asyncio

from llm_d_lens.model_cache import huggingface_hub

async def main() -> None:
    items = await huggingface_hub.search_models("Qwen2.5", limit=5)
    summaries = [item["id"] for item in items]
    print(summaries)

asyncio.run(main())

Get Hugging Face model details

Get detailed Hugging Face metadata for one model repository, including the README snapshot Lens fetched.

REST API
Python API

GET /api/v1/model-cache/huggingface/models/{repo_id:path}

Request: no request body. Path param: repo_id (str, for example Qwen/Qwen2.5-7B-Instruct).

Response: returns a HuggingFaceModelDetail object.

{
  "repoId": "Qwen/Qwen2.5-7B-Instruct",
  "likes": 12345,
  "downloads": 987654,
  "pipelineTag": "text-generation",
  "tags": ["transformers", "text-generation", "safetensors"],
  "lastModified": "2026-09-01T12:00:00.000Z",
  "gated": false,
  "readme": "---\nlibrary_name: transformers\n---\n# Qwen2.5-7B-Instruct"
}

import asyncio

from llm_d_lens.model_cache import huggingface_hub

async def main() -> None:
    detail = await huggingface_hub.get_model_detail("Qwen/Qwen2.5-7B-Instruct")
    print(detail["info"]["id"], bool(detail.get("readme")))

asyncio.run(main())

See the full request/response schemas in the FastAPI OpenAPI schema (regenerate with npm run docs:api).

Backend API​

List Model Cache entries​

Get a Model Cache entry​

Request a model download​

Retry a model download​

Sync new nodes for all entries​

Sync new nodes for one entry​

Delete a Model Cache entry​

Get Model Cache logs​

Search Hugging Face models​

Get Hugging Face model details​

Backend API

List Model Cache entries

Get a Model Cache entry

Request a model download

Retry a model download

Sync new nodes for all entries

Sync new nodes for one entry

Delete a Model Cache entry

Get Model Cache logs

Search Hugging Face models

Get Hugging Face model details