跳到主要内容

评估

后端 API​

评估相关的路由位于 llm_d_lens/evaluation/router.py 中，前缀为 /api/v1/evaluate。请求模型定义在 llm_d_lens.evaluation.models 中；命名的 HTTP 响应模型定义在 llm_d_lens.evaluation.api_models 中。

运行、工作流、用例和详情响应会保留额外的已存储字段。记录中不存在的可选字段会保持省略状态；显式存储的 null 值会保持为 null。列表响应会保留其 { "items": [...] } 信封结构。Python 路由函数返回字典；FastAPI 会在 HTTP 边界处应用对应的命名响应模型。

错误使用应用的 application/problem+json 响应格式，包含 type、status、title、detail、code 以及可选的 requestId。detail 字段可以包含结构化的校验错误。认证和资源权限仍然适用；声明一个响应模型并不会授予访问权限。

获取基准测试默认值​

获取 Lens Evaluate 将使用的 llm-d-benchmark 默认仓库和运行时设置。

GET /api/v1/evaluate/benchmark-defaults

请求：无请求体。

响应：HTTP 200，BenchmarkDefaultsResponse。

{
  "mode": "managed-source",
  "repository": "https://github.com/llm-d/llm-d-benchmark.git",
  "revision": "main",
  "localRuntimeConfigured": false
}

启动一次基准测试运行​

配置驱动工作流和已有服务评测都调用 create_run() 并使用同一执行器；工作流另外负责部署编排和比较。 当前 UI 的已有目标入口选择已发布的 Model Service，而非任意 URL。 deployment_ownership 控制清理归属，不改变指标语义：取消已有服务的评测不会停止模型服务。

两者都使用 BenchmarkSpec 默认值：parallelism=1、wait_timeout_seconds=1800、 每个 worker 的 harness_memory_gib=8，以及矩阵的 warmup_requests=2。 显式设置优先，UI 预设的预热次数可能不同。旧调用方如需 7200 秒或 32 GiB，必须显式传入。

证据操作当前行为
监控准备优先复用健康的 Prometheus target（up == 1），否则尝试创建监控资源及配套 RBAC。发现、配置和就绪等待共用 30 秒时限；失败会记录，但不使压测失败。
Prometheus 采集即使准备失败，仍查询压测时间窗口，时限 120 秒。仅发现集群的中心 Prometheus，不根据模型端点推断外部 Prometheus URL。缺失序列保持不可用，不记为零。
KV capture使用已有探针的容器，通过 kubectl exec 启停采集。压测不会安装探针、修改模型清单或重启服务。不支持的布局、缺失探针或请求 ID、采集冲突、worker 并行度不为一都会产生不可用证据。
资源快照在完成时、清理前尽力采集，时限 30 秒。缺失的历史快照不会用当前资源补写。

探针安装属于独立的部署渲染步骤：符合条件的新 vLLM 清单会加入探针，当前注入器跳过 PD 清单。 已有探针的部署不需要重新部署即可采集。KV 工作集采集要求兼容的 vLLM V1、 prefix caching、单个 FullAttention cache group 和 parallelism=1。 测量范围是成功请求中完整 prompt 块去重后的逻辑工作集，不包括生成 token 和不完整的尾块。 记录缺失或请求关联不完整时返回不可用结果；整轮证据不会自动分配给各负载阶段。 相同负载与探针条件保证指标覆盖和语义一致，不保证测量数值完全相同。

用户操作步骤见选择评测方案。

针对一个就绪的部署执行记录，或一个已发布的模型服务，启动一次基准测试运行。

POST /api/v1/evaluate/runs

请求体：EvaluateRunRequest

  • deployment_execution_id（string，可选）——直接对某个特定的部署执行记录进行基准测试。
  • model_service_group_id（string，可选）——改为对一个已发布、持续健康探测的 模型服务进行基准测试；测试工具会将其解析为该组的活跃成员及已发布名称，并且 要求提供 api_key。deployment_execution_id 与 model_service_group_id 二者必须恰好设置其中一个。
  • api_key（string，可选）——模型访问令牌；当设置了 model_service_group_id 时为必填项。
  • cluster_session_id（string，可选；提供时必须与执行记录所属的会话一致）
  • harness、workload、workload_yaml、parallelism、wait_timeout_seconds、harness_memory_gib
  • accelerator_profile、storage_class_name、matrix、concurrency_stages、warmup_requests、shared_prefix
  • sla_targets（BenchmarkSlaTargets）
  • specification_file、http_proxy、https_proxy、no_proxy、benchmark_source、use_baseline_endpoint
{
  "deployment_execution_id": "00000000-0000-4000-8000-000000000001",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "harness": "inference-perf",
  "workload": "sanity_random.yaml",
  "parallelism": 2,
  "wait_timeout_seconds": 1800,
  "harness_memory_gib": 8,
  "matrix": [
    {"isl": 1024, "osl": 128}
  ],
  "concurrency_stages": [
    {"concurrency": 8, "num_requests": 96}
  ],
  "warmup_requests": 2,
  "sla_targets": {
    "ttft_ms": 500,
    "ttft_percentile": "p99",
    "tpot_ms": 50,
    "tpot_percentile": "p99",
    "throughput_min_tps": 1000,
    "success_rate_min_percent": 99
  },
  "specification_file": "guides/optimized-baseline",
  "http_proxy": "",
  "https_proxy": "",
  "no_proxy": "",
  "benchmark_source": {
    "repository": "https://github.com/llm-d/llm-d-benchmark.git",
    "revision": "main"
  },
  "use_baseline_endpoint": false
}

要改为针对一个已发布的模型服务，请将 deployment_execution_id 替换为 model_service_group_id 并提供 api_key：

{
  "model_service_group_id": "01998e6b-4b2e-7c7e-8c6d-8e2a3f0c9b11",
  "api_key": "lens-mk-…",
  "harness": "inference-perf",
  "workload": "sanity_random.yaml"
}

响应：HTTP 202，BenchmarkRunResponse。已排队的基准测试运行记录。

{
  "kind": "benchmark",
  "id": "7d698fda-8e2e-44b2-86aa-14d24098350d",
  "status": "queued",
  "created_at": "2026-09-23T09:18:24.287000Z",
  "deployment_execution_id": "00000000-0000-4000-8000-000000000001",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "cluster_id": "cluster-1",
  "deployment_ownership": "existing-endpoint",
  "owner_user_id": "user-1",
  "harness": "inference-perf",
  "workload": "sanity_random.yaml",
  "parallelism": 2,
  "wait_timeout_seconds": 1800,
  "harness_memory_gib": 8,
  "matrix": [{"isl": 1024, "osl": 128}],
  "concurrency_stages": [{"concurrency": 8, "num_requests": 96}],
  "warmup_requests": 2,
  "sla_targets": {
    "ttft_ms": 500,
    "ttft_percentile": "p99",
    "tpot_ms": 50,
    "tpot_percentile": "p99",
    "throughput_min_tps": 1000,
    "success_rate_min_percent": 99
  },
  "benchmark_source": {
    "repository": "https://github.com/llm-d/llm-d-benchmark.git",
    "revision": "main"
  },
  "benchmark_runtime": {},
  "benchmark": {
    "harness": "inference-perf",
    "workload": "sanity_random.yaml",
    "parallelism": 2,
    "wait_timeout_seconds": 1800,
    "harness_memory_gib": 8,
    "matrix": [{"isl": 1024, "osl": 128}],
    "concurrency_stages": [{"concurrency": 8, "num_requests": 96}],
    "warmup_requests": 2,
    "shared_prefix": null,
    "workload_yaml": null,
    "accelerator_profile": null,
    "storage_class_name": null
  },
  "specification_file": "guides/optimized-baseline"
}

列出基准测试运行记录​

列出 Lens Evaluate 已记录的一次性基准测试运行记录。

GET /api/v1/evaluate/runs

请求：无请求体。

响应：HTTP 200，BenchmarkRunListResponse。

{
  "items": [
    {
      "kind": "benchmark",
      "id": "7d698fda-8e2e-44b2-86aa-14d24098350d",
      "status": "succeeded",
      "created_at": "2026-09-23T09:18:24.287000Z",
      "deployment_execution_id": "00000000-0000-4000-8000-000000000001",
      "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
      "model": "Qwen/Qwen3-8B",
      "endpoint": "http://127.0.0.1:38000/v1",
      "deployment": {
        "name": "00000000-0000-4000-8000-000000000001",
        "namespace": "prism-eval-1",
        "endpoint": "http://127.0.0.1:38000/v1"
      }
    }
  ]
}

获取一次基准测试运行记录​

GET /api/v1/evaluate/runs/{run_id} 保留压测记录顶层字段，并补充已知的 configuration、deployment_configuration、模型／命名空间／端点信息及只读 deployment_cases。 这些关联不会授予部署生命周期控制权。完成处理及终态详情读取与工作流共用客户端指标、 请求证据、容量／SLO、机制和诊断补全逻辑。保存的客户端报告可用于历史回填， 但缺失的历史 Prometheus 样本、KV capture 和资源快照不能由当前实时数据重建； 需要重新评测才能收集新启用的证据。

获取基准测试结果以及只读的部署证据信息。

GET /api/v1/evaluate/runs/{run_id}

请求：无请求体。

  • 路径参数：run_id（string，必填）

响应：HTTP 200，BenchmarkRunResponse。

{
  "kind": "benchmark",
  "id": "7d698fda-8e2e-44b2-86aa-14d24098350d",
  "status": "succeeded",
  "created_at": "2026-09-23T09:18:24.287000Z",
  "finished_at": "2026-09-23T09:24:10.000000Z",
  "deployment_execution_id": "00000000-0000-4000-8000-000000000001",
  "deployment_run_id": "deployment-1",
  "deployment_case_id": "case-1",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "model": "Qwen/Qwen3-8B",
  "namespace": "prism-eval-1",
  "endpoint": "http://127.0.0.1:38000/v1",
  "endpoint_used": "http://127.0.0.1:38000/v1",
  "guide": "optimized-baseline",
  "deployment_configuration": {
    "schema_version": "deployable-configuration.v1",
    "type": "baseline",
    "format": "helm",
    "provider_ref": "optimized-baseline",
    "checksum": "sha256:9eb2d61d",
    "content": {"model": {"name": "Qwen/Qwen3-8B"}},
    "provenance": {}
  },
  "configuration": {
    "configuration_artifact_id": "00000000-0000-4000-8000-000000000101",
    "model": "Qwen/Qwen3-8B",
    "guide": "optimized-baseline",
    "guide_variant": null,
    "runtime": null,
    "accelerator": null,
    "image": null,
    "replicas": null,
    "tensor_parallel_size": null,
    "decode_replicas": null,
    "decode_tensor_parallel_size": null,
    "prefill_replicas": null,
    "prefill_tensor_parallel_size": null,
    "benchmark": {
      "harness": "inference-perf",
      "workload": "sanity_random.yaml",
      "parallelism": 2,
      "wait_timeout_seconds": 1800,
      "harness_memory_gib": 8,
      "matrix": [{"isl": 1024, "osl": 128}],
      "concurrency_stages": [{"concurrency": 8, "num_requests": 96}],
      "warmup_requests": 2,
      "shared_prefix": null,
      "workload_yaml": null,
      "accelerator_profile": null,
      "storage_class_name": null
    }
  },
  "metrics": {
    "throughput_tps": 1210.5,
    "success_rate": 99.6,
    "latency_distributions": {
      "ttft": {"p99_ms": 420.0},
      "tpot": {"p99_ms": 42.0}
    }
  },
  "resource_snapshot": {},
  "deployment_cases": [
    {
      "id": "case-1",
      "execution_id": "00000000-0000-4000-8000-000000000001",
      "guide": "optimized-baseline",
      "namespace": "prism-eval-1",
      "endpoint": "http://127.0.0.1:38000/v1",
      "status": "ready",
      "execution_status": "ready",
      "monitoring_setup": null,
      "diagnostics": null,
      "resource_snapshot": {}
    }
  ]
}

取消一次基准测试运行​

取消一次正在进行中的一次性基准测试运行。

POST /api/v1/evaluate/runs/{run_id}/cancel

请求：无请求体。

  • 路径参数：run_id（string，必填）

响应：HTTP 200，EvaluationCancelResponse。

{
  "id": "7d698fda-8e2e-44b2-86aa-14d24098350d",
  "status": "cancelled"
}

删除一次基准测试运行​

删除一次已处于终态的独立基准测试及其已保存的结果。

DELETE /api/v1/evaluate/runs/{run_id}

请求：无请求体。

  • 路径参数：run_id（string，必填）

响应：HTTP 204 No Content。不返回任何 JSON 响应体。

启动一次“部署+基准测试”工作流​

启动一个简单的“部署+基准测试”工作流：先创建一个经过优化的基线部署，然后对其进行基准测试。

POST /api/v1/evaluate/workflow-runs

请求体：EvaluateWorkflowRequest

  • model（string，必填）
  • cluster_session_id（string，必填）
  • replicas、tensor_parallel_size、image、harness、workload、parallelism
{
  "model": "Qwen/Qwen3-8B",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "replicas": 1,
  "tensor_parallel_size": 1,
  "image": "ghcr.io/llm-d/llm-d-xpu:v0.9.0",
  "harness": "inference-perf",
  "workload": "sanity_random.yaml",
  "parallelism": 1
}

响应：HTTP 202，EvaluationWorkflowResponse。

{
  "kind": "workflow",
  "deployment_ownership": "evaluation",
  "id": "0e8dd5de-8a1c-4f96-9756-fd91c1da4faa",
  "status": "deploying",
  "created_at": "2026-09-23T09:18:24.287000Z",
  "deployment_run_id": "deployment-1",
  "cluster_id": "cluster-1",
  "owner_user_id": "user-1",
  "model": "Qwen/Qwen3-8B",
  "harness": "inference-perf",
  "workload": "sanity_random.yaml",
  "parallelism": 1,
  "benchmark_runtime": {}
}

启动一次多配置评估​

对于 pd-disaggregation 源配置，独立部署的基线默认匹配 Prefill 与 Decode 的总 GPU 预算：

baseline replicas = (prefill replicas * prefill TP + decode replicas * decode TP) / baseline TP

基线 TP 默认继承源配置 Decode／Serving TP，可用 benchmark_plans[].baseline_parameters[baseline_type].tensor_parallel_size 覆盖。 仅覆盖 TP 会重新计算副本数；显式 replicas 优先，允许刻意选择不等预算。 例如 2P2D、两侧 TP=1 默认生成四个 TP=1 的普通 vLLM 副本；基线 TP=2 时为两个副本。 自动匹配必须产生 1..32 范围内的精确整数，否则在保存及排队前返回 409，不会取整或截断。 非 PD 源配置继承 Decode／Serving 副本数；同 Pod 的 kubernetes-service 比较复用候选资源池， 不分配独立副本。此默认规则仅作用于新建评估，不改写已有／运行中快照或重试； 历史上的不等预算比较不会因此变成等预算。覆盖字段范围参见评估参数。

启动一个多配置评估工作流，它可以部署、基准测试、比较多个已发布的配置工件，并可选地建立基线。

POST /api/v1/evaluate/evaluations

请求体：EvaluationCreateRequest

  • name（string，默认值 "Evaluation"）
  • cluster_session_id（string，必填）
  • runtime（RuntimeSpec）
  • benchmark_source（BenchmarkSource，可选）
  • benchmark_plans（BenchmarkPlan[]，必填）
  • compare_configurations（boolean，默认值 true）
{
  "name": "Qwen comparison",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "runtime": {
    "http_proxy": "",
    "https_proxy": "",
    "no_proxy": ""
  },
  "benchmark_source": {
    "repository": "https://github.com/llm-d/llm-d-benchmark.git",
    "revision": "main"
  },
  "benchmark_plans": [
    {
      "id": "candidate",
      "configuration_artifact_id": "00000000-0000-4000-8000-000000000001",
      "deployment_name": "qwen3-guide",
      "deployment_description": "Candidate guide deployment",
      "benchmark": {
        "harness": "inference-perf",
        "workload": "sanity_random.yaml",
        "parallelism": 1,
        "wait_timeout_seconds": 1800,
        "harness_memory_gib": 8,
        "matrix": [{"isl": 128, "osl": 64}],
        "concurrency_stages": [{"concurrency": 4, "num_requests": 32}],
        "warmup_requests": 2,
        "shared_prefix": null,
        "workload_yaml": null,
        "accelerator_profile": null,
        "storage_class_name": null
      },
      "include_configuration": true,
      "include_baseline": true,
      "baseline_type": "direct-vllm",
      "baseline_types": ["direct-vllm", "optimized-baseline"],
      "baseline_parameters": {},
      "preserve_deployment": false,
      "scenarios": []
    }
  ],
  "compare_configurations": true
}

响应：HTTP 202，EvaluationWorkflowResponse。已排队的工作流记录，包含已生成的用例。

{
  "kind": "workflow",
  "deployment_ownership": "evaluation",
  "id": "2d52bd97-a873-4d39-aa17-a182bf456922",
  "name": "Qwen comparison",
  "status": "queued",
  "created_at": "2026-09-23T09:18:24.287000Z",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "cluster_id": "cluster-1",
  "owner_user_id": "user-1",
  "benchmark_runtime": {},
  "deployment_source": {},
  "runtime": {
    "http_proxy": "",
    "https_proxy": "",
    "no_proxy": ""
  },
  "configuration_artifacts": {
    "00000000-0000-4000-8000-000000000001": "sha256:9eb2d61d"
  },
  "benchmark_plans": [
    {
      "id": "candidate",
      "configuration_artifact_id": "00000000-0000-4000-8000-000000000001",
      "deployment_name": "qwen3-guide",
      "deployment_description": "Candidate guide deployment",
      "benchmark": {
        "harness": "inference-perf",
        "workload": "sanity_random.yaml",
        "parallelism": 1,
        "wait_timeout_seconds": 1800,
        "harness_memory_gib": 8,
        "matrix": [{"isl": 128, "osl": 64}],
        "concurrency_stages": [{"concurrency": 4, "num_requests": 32}],
        "warmup_requests": 2,
        "shared_prefix": null,
        "workload_yaml": null,
        "accelerator_profile": null,
        "storage_class_name": null
      },
      "include_configuration": true,
      "include_baseline": true,
      "baseline_type": "direct-vllm",
      "baseline_types": ["direct-vllm", "optimized-baseline"],
      "baseline_parameters": {},
      "preserve_deployment": false,
      "scenarios": []
    }
  ],
  "compare_configurations": true,
  "cases": [
    {
      "id": "guide-1",
      "kind": "guide",
      "status": "queued",
      "workload_ids": ["candidate"],
      "baseline_case_id": "baseline-1",
      "baseline_case_ids": ["baseline-1", "baseline-2"],
      "benchmark": {
        "harness": "inference-perf",
        "workload": "sanity_random.yaml",
        "parallelism": 1,
        "wait_timeout_seconds": 1800,
        "harness_memory_gib": 8,
        "matrix": [{"isl": 128, "osl": 64}],
        "concurrency_stages": [{"concurrency": 4, "num_requests": 32}],
        "warmup_requests": 2,
        "shared_prefix": null,
        "workload_yaml": null,
        "accelerator_profile": null,
        "storage_class_name": null
      },
      "deployment_run_id": null,
      "evaluation_run_id": null,
      "configuration_artifact_id": "00000000-0000-4000-8000-000000000001",
      "benchmark_plan_id": "candidate",
      "deployment_group_id": "candidate:guide",
      "preserve_deployment": false,
      "deployment_configuration": {
        "schema_version": "deployable-configuration.v1",
        "type": "baseline",
        "format": "helm",
        "provider_ref": "optimized-baseline",
        "checksum": "sha256:9eb2d61d",
        "content": {"model": {"name": "Qwen/Qwen3-8B"}},
        "provenance": {}
      }
    },
    {
      "id": "baseline-1",
      "kind": "baseline",
      "status": "queued",
      "workload_ids": ["candidate"],
      "benchmark": {
        "harness": "inference-perf",
        "workload": "sanity_random.yaml",
        "parallelism": 1,
        "wait_timeout_seconds": 1800,
        "harness_memory_gib": 8,
        "matrix": [{"isl": 128, "osl": 64}],
        "concurrency_stages": [{"concurrency": 4, "num_requests": 32}],
        "warmup_requests": 2,
        "shared_prefix": null,
        "workload_yaml": null,
        "accelerator_profile": null,
        "storage_class_name": null,
        "baseline_type": "direct-vllm",
        "baseline_parameters": {}
      },
      "deployment_run_id": null,
      "evaluation_run_id": null,
      "configuration_artifact_id": null,
      "benchmark_plan_id": "candidate",
      "deployment_group_id": "candidate:baseline:direct-vllm",
      "baseline_type": "direct-vllm",
      "baseline_parameters": {},
      "dependent_guide_case_id": null,
      "preserve_deployment": false,
      "deployment_configuration": {
        "schema_version": "deployable-configuration.v1",
        "type": "baseline",
        "format": "helm",
        "provider_ref": "baseline-vllm",
        "checksum": "sha256:12b3ef20",
        "content": {"model": {"name": "Qwen/Qwen3-8B"}},
        "provenance": {"system_generated": true}
      }
    }
  ],
  "report": null
}

列出评估工作流运行记录​

列出 Lens 已记录的评估工作流运行记录，包括多配置评估以及简单的“部署+基准测试”工作流。

GET /api/v1/evaluate/workflow-runs

请求：无请求体。

响应：HTTP 200，EvaluationWorkflowListResponse。

{
  "items": [
    {
      "kind": "workflow",
      "id": "2d52bd97-a873-4d39-aa17-a182bf456922",
      "name": "Qwen comparison",
      "status": "running",
      "created_at": "2026-09-23T09:18:24.287000Z",
      "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
      "cases": [
        {
          "id": "guide-1",
          "kind": "guide",
          "status": "benchmarking",
          "benchmark_plan_id": "candidate",
          "configuration": {
            "configuration_artifact_id": "00000000-0000-4000-8000-000000000001",
            "model": "Qwen/Qwen3-8B",
            "guide": "optimized-baseline",
            "guide_variant": null,
            "runtime": null,
            "accelerator": null,
            "image": null,
            "replicas": null,
            "tensor_parallel_size": null,
            "decode_replicas": null,
            "decode_tensor_parallel_size": null,
            "prefill_replicas": null,
            "prefill_tensor_parallel_size": null,
            "benchmark": {
              "harness": "inference-perf",
              "workload": "sanity_random.yaml",
              "parallelism": 1,
              "wait_timeout_seconds": 1800,
              "harness_memory_gib": 8,
              "matrix": [{"isl": 128, "osl": 64}],
              "concurrency_stages": [{"concurrency": 4, "num_requests": 32}],
              "warmup_requests": 2,
              "shared_prefix": null,
              "workload_yaml": null,
              "accelerator_profile": null,
              "storage_class_name": null
            }
          }
        }
      ]
    }
  ]
}

获取一次评估工作流运行记录​

清理成功时，会同时删除评估所拥有的临时部署记录及其集群资源，取消运行时也适用。 评估报告和已采集的证据仍然保留。借用的已有部署不会被删除；清理失败时保留部署记录，以便重试。

获取一次评估工作流运行记录当前的顶层记录。

GET /api/v1/evaluate/workflow-runs/{workflow_id}

请求：无请求体。

  • 路径参数：workflow_id（string，必填）

响应：HTTP 200，EvaluationWorkflowResponse。

{
  "kind": "workflow",
  "deployment_ownership": "evaluation",
  "id": "2d52bd97-a873-4d39-aa17-a182bf456922",
  "name": "Qwen comparison",
  "status": "running",
  "created_at": "2026-09-23T09:18:24.287000Z",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "cluster_id": "cluster-1",
  "runtime": {
    "http_proxy": "",
    "https_proxy": "",
    "no_proxy": ""
  },
  "configuration_artifacts": {
    "00000000-0000-4000-8000-000000000001": "sha256:9eb2d61d"
  },
  "benchmark_plans": [
    {
      "id": "candidate",
      "configuration_artifact_id": "00000000-0000-4000-8000-000000000001",
      "baseline_types": ["direct-vllm", "optimized-baseline"],
      "preserve_deployment": false,
      "scenarios": []
    }
  ],
  "compare_configurations": true,
  "cases": [
    {
      "id": "guide-1",
      "kind": "guide",
      "status": "benchmarking",
      "workload_ids": ["candidate"],
      "baseline_case_id": "baseline-1",
      "baseline_case_ids": ["baseline-1", "baseline-2"],
      "benchmark": {
        "harness": "inference-perf",
        "workload": "sanity_random.yaml",
        "parallelism": 1,
        "wait_timeout_seconds": 1800,
        "harness_memory_gib": 8,
        "matrix": [{"isl": 128, "osl": 64}],
        "concurrency_stages": [{"concurrency": 4, "num_requests": 32}],
        "warmup_requests": 2,
        "shared_prefix": null,
        "workload_yaml": null,
        "accelerator_profile": null,
        "storage_class_name": null
      },
      "deployment_run_id": "deployment-1",
      "evaluation_run_id": "benchmark-1",
      "configuration_artifact_id": "00000000-0000-4000-8000-000000000001",
      "benchmark_plan_id": "candidate",
      "deployment_group_id": "candidate:guide",
      "preserve_deployment": false
    }
  ],
  "report": null
}

获取详细的评估工作流结果​

关联的压测及用例结果与独立压测详情共用补全逻辑。用例快照是完成时保存的证据。 deployment_cases 还可能包含供查看的当前 Kubernetes 快照；这些实时快照不属于历史压测测量。

获取展开后的评估工作流结果以及部署关联信息。

对于包含用例的工作流，报告使用 comparison_version: 2。在读取详情时，较旧的缓存报告会根据保留下来的证据数据重新生成。矩阵行会匹配唯一的、成功的 (isl, osl) 数据点以及显式的 concurrency；速率阶段会匹配显式的 rate。缺失、非有限、负数或重复的负载键不构成匹配，数组位置也永远不会作为回退依据。没有匹配行时，对应的统计信息为 null。

比值等于 guide 值除以 baseline 值。有效的零分子会被保留，包括在几何平均值的计算中；零分母则不产生比值。按套件归一化的吞吐量仅使用那些两侧输出 token 数均非负、且实测时长均为正数的匹配配对。均值、P90 和 P99 聚合值保持相互独立。

GET /api/v1/evaluate/workflow-runs/{workflow_id}/details

请求：无请求体。

  • 路径参数：workflow_id（string，必填）

响应：HTTP 200，EvaluationDetailsResponse。

{
  "workflow": {
    "id": "2d52bd97-a873-4d39-aa17-a182bf456922",
    "name": "Qwen comparison",
    "status": "succeeded",
    "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
    "cases": [
      {
        "id": "guide-1",
        "kind": "guide",
        "status": "succeeded",
        "deployment_run_id": "deployment-1",
        "evaluation_run_id": "benchmark-1"
      }
    ]
  },
  "cases": [
    {
      "case": {
        "id": "guide-1",
        "kind": "guide",
        "status": "succeeded",
        "deployment_run_id": "deployment-1",
        "evaluation_run_id": "benchmark-1"
      },
      "deployment": {
        "id": "deployment-1",
        "status": "succeeded"
      },
      "deployment_cases": [
        {
          "id": "case-1",
          "status": "ready",
          "guide": "optimized-baseline",
          "attempt": 1,
          "failure": null,
          "execution_id": "00000000-0000-4000-8000-000000000001",
          "namespace": "prism-eval-1",
          "endpoint": "http://127.0.0.1:38000/v1",
          "execution_status": "ready",
          "monitoring_setup": null,
          "diagnostics": null,
          "resource_snapshot": {}
        }
      ],
      "deployment_warning": null,
      "evaluation": {
        "id": "benchmark-1",
        "status": "succeeded",
        "deployment_cases": [
          {
            "id": "case-1",
            "execution_id": "00000000-0000-4000-8000-000000000001",
            "guide": "optimized-baseline",
            "namespace": "prism-eval-1",
            "endpoint": "http://127.0.0.1:38000/v1",
            "status": "ready",
            "execution_status": "ready",
            "monitoring_setup": null,
            "diagnostics": null,
            "resource_snapshot": {}
          }
        ]
      }
    }
  ],
  "report": null
}

重试一次评估工作流​

重试一个已失败或已取消的多工作负载评估工作流。

POST /api/v1/evaluate/workflow-runs/{workflow_id}/retry

请求：无请求体。

  • 路径参数：workflow_id（string，必填）

响应：HTTP 202，EvaluationWorkflowResponse。

{
  "kind": "workflow",
  "id": "2d52bd97-a873-4d39-aa17-a182bf456922",
  "name": "Qwen comparison",
  "status": "queued",
  "cluster_session_id": "123e4567-e89b-12d3-a456-426614174000",
  "cases": [
    {
      "id": "guide-1",
      "kind": "guide",
      "status": "queued",
      "deployment_run_id": null,
      "evaluation_run_id": null,
      "execution_id": null,
      "error": null
    }
  ],
  "report": null
}

取消一次评估工作流​

取消一个评估工作流，并清理其拥有的部署。

POST /api/v1/evaluate/workflow-runs/{workflow_id}/cancel

请求：无请求体。

  • 路径参数：workflow_id（string，必填）

响应：HTTP 200，EvaluationCancelResponse。

{
  "id": "2d52bd97-a873-4d39-aa17-a182bf456922",
  "status": "cancelled"
}

取消一个评估用例​

取消一个评估用例，并清理其拥有的部署。

POST /api/v1/evaluate/workflow-runs/{workflow_id}/cases/{case_id}/cancel

请求：无请求体。

  • 路径参数：workflow_id（string，必填）、case_id（string，必填）

响应：HTTP 200，EvaluationCancelResponse。

{
  "id": "guide-1",
  "status": "cancelled"
}

删除一个评估工作流​

删除一个已处于终态的评估工作流，以及它所拥有的、仍被保留的子基准测试记录。

DELETE /api/v1/evaluate/workflow-runs/{workflow_id}

请求：无请求体。

  • 路径参数：workflow_id（string，必填）

响应：HTTP 204 No Content。不返回任何 JSON 响应体。

完整的请求/响应 schema 请参见 FastAPI 的 OpenAPI schema（使用 npm run docs:api 重新生成）。