Model cache

后端 API

下载、删除和增量节点同步共用 llm_d_lens.model_cache.jobs 中的 launch_cache_jobs。下面的公共服务调用保留了其现有签名：调用方负责选择节点和准备凭据，而启动器负责初始化进度并提交 Jobs。增量同步会保留已有节点；部分提交失败会向外传播，而不会将整个批次标记为“进行中”。

路由位于 llm_d_lens/model_cache/router.py。下面的每个端点都记录了 REST 契约，以及它所使用的对应 ModelCacheService 或 Hugging Face 辅助调用。

部署会直接消费缓存。llm_d_lens.deploy.providers.model_cache_environment.model_cache_environment 会为每个基于缓存的模型服务器启用 Hugging Face 离线模式（HF_HUB_OFFLINE=1、TRANSFORMERS_OFFLINE=1），并为 auto-cache 来源添加 HF_HOME=/model-cache，使 vLLM 能够从挂载的快照中解析模型，而不是在启动时访问 huggingface.co。直接挂载模型目录的提供方（shared-path、mountPath，包括 P/D 分离、精确前缀缓存路由和分层前缀缓存）会获得离线标志但不设置 HF_HOME。这使得基于缓存的启动在受限或不稳定的网络环境下仍能正常工作（例如 repo_utils.py 中的 SSL unexpected_eof_while_reading 失败）。huggingface 来源仍会在启动时下载并保持在线。

Hugging Face 环境变量（HF_*、HUGGING*、TRANSFORMERS_*）由管理员统一管理。部署规划 API 和配置校验会拒绝用户提供的、试图设置这些变量的 customParameters 条目（参见 llm_d_lens/configuration/managed_environment.py），部署编辑器也会以“由管理员管理”的提示信息阻止这些操作。平台会根据集群已保存的代理、模型缓存和令牌配置自动注入这些值。

列出模型缓存条目

列出模型缓存条目，可选按集群或存储卷过滤。用它来发现已缓存的模型 id、下载状态以及节点同步偏差。

REST API

GET /api/v1/model-cache/entries

请求：无请求体。可选查询参数：cluster_id（str）和 storage_volume_id（str）。

响应：返回模型缓存条目的 items 数组，每个条目都包含由路由计算得出的 pendingSyncNodes 字段。

{
  "items": [
    {
      "id": "model-cache-7f3a2c1d",
      "clusterId": "cluster-123",
      "storageVolumeId": "storage-abc12345",
      "source": {
        "kind": "huggingface",
        "huggingface": {
          "repoId": "Qwen/Qwen2.5-7B-Instruct",
          "revision": "main"
        }
      },
      "tokenSource": {
        "mode": "none",
        "namespace": null,
        "name": null
      },
      "status": "ready",
      "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
      "sizeBytes": 15123042304,
      "nodeProgress": [
        {
          "node": "worker-1",
          "status": "ready",
          "failureDetail": null
        }
      ],
      "failureDetail": null,
      "createdAt": "2026-09-23T08:15:00Z",
      "updatedAt": "2026-09-23T08:22:11Z",
      "pendingSyncNodes": ["worker-2"]
    }
  ]
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entries = await service.list(cluster_id="cluster-123", storage_volume_id=None)
    pending = await service.compute_pending_sync(entries)
    payload = [
        {**entry.api_payload(), "pendingSyncNodes": pending.get(entry.id, [])}
        for entry in entries
    ]
    print(payload)


asyncio.run(main())

获取单个模型缓存条目

根据 id 获取单个模型缓存条目，包括来源、令牌模式、节点进度和待同步节点。

REST API

GET /api/v1/model-cache/entries/{entry_id}

请求：无请求体。路径参数：entry_id（str）。

响应：返回单个模型缓存条目对象，并附带由路由计算得出的 pendingSyncNodes 列表。

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "existing-secret",
    "namespace": "models",
    "name": "hf-token"
  },
  "status": "downloading",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": null,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "downloading",
      "failureDetail": null
    },
    {
      "node": "worker-2",
      "status": "pending",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:16:09Z",
  "pendingSyncNodes": []
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = await service.get("model-cache-7f3a2c1d")
    pending = await service.compute_pending_sync([entry])
    payload = {**entry.api_payload(), "pendingSyncNodes": pending.get(entry.id, [])}
    print(payload)


asyncio.run(main())

请求下载模型

请求将模型下载到 Lens 模型缓存中。用它可以在部署或基准测试之前预先准备好一份就绪的共享模型缓存。

REST API

POST /api/v1/model-cache/entries

请求：请求体字段包括 clusterId（str）、storageVolumeId（str）、source（ModelSource）以及可选的 force（bool，默认 false）。为了向后兼容仍接受 tokenSource，但会被忽略：下载始终使用与集群一起保存的 HuggingFace 令牌（向导中的 hf_token_secret），如果该集群没有令牌，则以未认证方式运行。前端的下载模型对话框现在已经完全不再发送 tokenSource。

如果该存储卷上已经存在同一模型、同一版本的条目（无论处于什么状态），路由不会开始下载，而是返回 409、错误码为 model_cache_entry_exists 的问题响应，以便调用方请用户确认。响应体中会带上 entryId、entryStatus、model 和 storageVolumeId。用 force: true 重新发送同一请求即可继续：处于 ready 或 failed 的条目会被重置为 pending 并就地重新下载，处于 pending 或 downloading 的条目则原样返回，因此确认操作是加入正在进行的下载，而不是再起一个。任何情况下都不会创建重复条目。

{
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  }
}

响应：返回为该下载请求创建的、处于排队状态的 ModelCacheEntry 数据。

{
  "type": "about:blank",
  "status": 409,
  "title": "Model already on this storage volume",
  "detail": "'Qwen/Qwen2.5-7B-Instruct@main' is already on this storage volume (status: ready)",
  "code": "model_cache_entry_exists",
  "entryId": "model-cache-7f3a2c1d",
  "entryStatus": "ready",
  "model": "Qwen/Qwen2.5-7B-Instruct@main",
  "storageVolumeId": "storage-abc12345"
}

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "existing-secret",
    "namespace": "models",
    "name": "hf-token"
  },
  "status": "pending",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": null,
  "nodeProgress": [],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:15:00Z",
  "pendingSyncNodes": []
}

Python API

import asyncio

from llm_d_lens.model_cache.contracts import HuggingFaceSource, ModelCacheCreateRequest, ModelSource, ModelSourceKind
from llm_d_lens.model_cache.service import ModelCacheEntryExistsError, default_service


async def main() -> None:
    service = default_service()
    # tokenSource is intentionally omitted: request_download() always resolves the
    # HuggingFace token from the cluster's saved hf_token_secret instead.
    request = ModelCacheCreateRequest(
        clusterId="cluster-123",
        storageVolumeId="storage-abc12345",
        source=ModelSource(
            kind=ModelSourceKind.HUGGINGFACE,
            huggingface=HuggingFaceSource(repoId="Qwen/Qwen2.5-7B-Instruct", revision="main"),
        ),
    )
    try:
        entry = await service.request_download(request)
    except ModelCacheEntryExistsError as already_there:
        # Ask the operator whether to download it again, then retry with force.
        print(already_there.entry.id, already_there.entry.status)
        entry = await service.request_download(request.model_copy(update={"force": True}))
    await service.provision(entry.id)
    print(entry.id, entry.status)


asyncio.run(main())

重试模型下载

重试一个失败或可重试的模型缓存条目下载。

REST API

POST /api/v1/model-cache/entries/{entry_id}/retry

请求：无请求体。路径参数：entry_id（str）。

响应：返回重新进入可重试下载状态后的更新后条目。

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "none",
    "namespace": null,
    "name": null
  },
  "status": "pending",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": null,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "pending",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:30:00Z",
  "pendingSyncNodes": []
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = service.retry("model-cache-7f3a2c1d")
    await service.provision(entry.id)
    print(entry.api_payload())


asyncio.run(main())

为所有条目同步新节点

为所有在新添加节点上缺失的本地磁盘模型缓存条目触发同步。可以选择将范围限定在某个集群。

REST API

POST /api/v1/model-cache/entries/sync-nodes

请求：无请求体。可选查询参数：cluster_id（str）。

响应：返回已被更改并加入新节点预配队列的条目。

{
  "items": [
    {
      "id": "model-cache-7f3a2c1d",
      "clusterId": "cluster-123",
      "storageVolumeId": "storage-abc12345",
      "source": {
        "kind": "huggingface",
        "huggingface": {
          "repoId": "Qwen/Qwen2.5-7B-Instruct",
          "revision": "main"
        }
      },
      "tokenSource": {
        "mode": "none",
        "namespace": null,
        "name": null
      },
      "status": "downloading",
      "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
      "sizeBytes": 15123042304,
      "nodeProgress": [
        {
          "node": "worker-1",
          "status": "ready",
          "failureDetail": null
        },
        {
          "node": "worker-2",
          "status": "pending",
          "failureDetail": null
        }
      ],
      "failureDetail": null,
      "createdAt": "2026-09-23T08:15:00Z",
      "updatedAt": "2026-09-23T08:35:00Z",
      "pendingSyncNodes": []
    }
  ]
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    changed = await service.sync_all_nodes(cluster_id="cluster-123")
    for entry in changed:
        await service.provision_new_nodes(entry.id)
    print([entry.id for entry in changed])


asyncio.run(main())

为单个条目同步新节点

为某一个特定的模型缓存条目触发节点同步。当某个已缓存的模型需要被复制到新添加的节点上时使用此端点。

REST API

POST /api/v1/model-cache/entries/{entry_id}/sync-nodes

请求：无请求体。路径参数：entry_id（str）。

响应：返回更新后的条目。当有新节点需要预配时，条目状态会回到 downloading。

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "none",
    "namespace": null,
    "name": null
  },
  "status": "downloading",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": 15123042304,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "ready",
      "failureDetail": null
    },
    {
      "node": "worker-2",
      "status": "pending",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T08:35:00Z",
  "pendingSyncNodes": []
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = await service.sync_nodes("model-cache-7f3a2c1d")
    if entry.status.value == "downloading":
        await service.provision_new_nodes(entry.id)
    print(entry.api_payload())


asyncio.run(main())

删除模型缓存条目

删除一个模型缓存条目，并在需要时安排清理其已下载的文件。设置 keep_files=true 可以只删除记录本身。

REST API

DELETE /api/v1/model-cache/entries/{entry_id}

请求：无请求体。路径参数：entry_id（str）。可选查询参数：keep_files（bool，默认 false）。

响应：通常在后台清理运行期间返回处于 deleting 状态的条目。如果该条目从未到达过已持久化的缓存路径，API 可能会改为返回一个最简的 {"id": ..., "status": "deleted"} 确认信息。

{
  "id": "model-cache-7f3a2c1d",
  "clusterId": "cluster-123",
  "storageVolumeId": "storage-abc12345",
  "source": {
    "kind": "huggingface",
    "huggingface": {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "revision": "main"
    }
  },
  "tokenSource": {
    "mode": "none",
    "namespace": null,
    "name": null
  },
  "status": "deleting",
  "cachePath": "hub/models--Qwen--Qwen2.5-7B-Instruct",
  "sizeBytes": 15123042304,
  "nodeProgress": [
    {
      "node": "worker-1",
      "status": "ready",
      "failureDetail": null
    }
  ],
  "failureDetail": null,
  "createdAt": "2026-09-23T08:15:00Z",
  "updatedAt": "2026-09-23T09:00:00Z",
  "pendingSyncNodes": []
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    entry = await service.request_delete("model-cache-7f3a2c1d", keep_files=False)
    if entry is not None:
        await service.provision_delete(entry.id)
        print(entry.api_payload())
    else:
        print({"id": "model-cache-7f3a2c1d", "status": "deleted"})


asyncio.run(main())

获取模型缓存日志

获取某个模型缓存条目的日志，可选限定在某个节点。用它来调试下载或清理失败的问题。

REST API

GET /api/v1/model-cache/entries/{entry_id}/logs

请求：无请求体。路径参数：entry_id（str）。可选查询参数：node（str）。

响应：返回该条目收集到的日志文本，或者仅限于所请求节点的日志文本。

{
  "logs": "[worker-1] pulling Qwen/Qwen2.5-7B-Instruct@main\n[worker-1] download completed successfully"
}

Python API

import asyncio

from llm_d_lens.model_cache.service import default_service


async def main() -> None:
    service = default_service()
    logs = await service.get_logs("model-cache-7f3a2c1d", node="worker-1")
    print(logs)


asyncio.run(main())

搜索 Hugging Face 模型

从 Lens 模型缓存中搜索 Hugging Face 模型仓库。用它在创建缓存条目之前先发现可用的仓库 id。

REST API

GET /api/v1/model-cache/huggingface/search

请求：无请求体。查询参数：query（str）和可选的 limit（int，默认 20）。

响应：以 HuggingFaceModelSummary 对象的形式返回 Hugging Face 的搜索匹配结果。

{
  "items": [
    {
      "repoId": "Qwen/Qwen2.5-7B-Instruct",
      "likes": 12345,
      "downloads": 987654,
      "pipelineTag": "text-generation",
      "tags": ["transformers", "text-generation", "safetensors"],
      "lastModified": "2026-09-01T12:00:00.000Z",
      "gated": false
    }
  ]
}

Python API

import asyncio

from llm_d_lens.model_cache import huggingface_hub


async def main() -> None:
    items = await huggingface_hub.search_models("Qwen2.5", limit=5)
    summaries = [item["id"] for item in items]
    print(summaries)


asyncio.run(main())

获取 Hugging Face 模型详情

获取某个模型仓库的详细 Hugging Face 元数据，包括 Lens 抓取到的 README 快照。

REST API

GET /api/v1/model-cache/huggingface/models/{repo_id:path}

请求：无请求体。路径参数：repo_id（str，例如 Qwen/Qwen2.5-7B-Instruct）。

响应：返回一个 HuggingFaceModelDetail 对象。

{
  "repoId": "Qwen/Qwen2.5-7B-Instruct",
  "likes": 12345,
  "downloads": 987654,
  "pipelineTag": "text-generation",
  "tags": ["transformers", "text-generation", "safetensors"],
  "lastModified": "2026-09-01T12:00:00.000Z",
  "gated": false,
  "readme": "---\nlibrary_name: transformers\n---\n# Qwen2.5-7B-Instruct"
}

Python API

import asyncio

from llm_d_lens.model_cache import huggingface_hub


async def main() -> None:
    detail = await huggingface_hub.get_model_detail("Qwen/Qwen2.5-7B-Instruct")
    print(detail["info"]["id"], bool(detail.get("readme")))


asyncio.run(main())

完整的请求/响应 schema 请参见 FastAPI OpenAPI schema（可通过 npm run docs:api 重新生成）。