Skip to main content

Model Market

Open Model Market to choose a model and deploy it to a cluster. For your first deployment, use Manual mode.

Use Search models above the categories to filter by model name, publisher, repository, model type, category, or precision. Search ignores case and combines with the selected category; multiple words must all match. The model count updates as you type. Use Clear filters to reset both search and category.

Choose a deployment mode​

ModeChoose it when
ManualYou want to choose an llm-d Guide template, configure resources or review generated YAML
Agentic (Experimental)You want Lens to recommend a configuration for a workload and performance target

Create a manual deployment​

  1. Open Model Market, use Model categories to narrow the list, and click a model card.
  2. In One-click deployment, enter Deployment name and choose a ready Cluster.
  3. Choose Model cache storage containing this model. Wait for This model is ready in the selected cache. Model cache storage is optional in Manual mode.
  4. Keep Mode on Manual and click Configure deployment. Review the prefilled model and deployment settings; use Edit setup if needed.
  5. Check the runtime image, replicas, and tensor parallel size.
  6. Under Guide template, choose Guide and Accelerator. Use Choose an evaluation Guide to compare the options.
  7. For a P/D or tiered-cache Guide, choose a variant from the displayed list. Keep the supplied defaults for the first run.
  8. Click Generate YAML. If available cards are insufficient but the physical total is sufficient, review the Pending warning and click Continue & generate. If the physical total is insufficient, reduce replicas / TP.
  9. Review Generated deployable configuration. Correct any highlighted errors, then click Deploy.
  10. On Deployments, wait for Ready. The deployment is retained without running a benchmark; stop or clean it up explicitly when no longer needed.

Use Back to model card to return to the selected model. To measure performance, see Run an evaluation.

No cluster or cache available? Add a cluster, create cache storage, and download the model. If the model is missing from the selected cache, download it there before returning.

Open Manage model cache from a selected model to open Download a model directly in Model cache. The selected cluster, Hugging Face repository ID, and storage volume are prefilled and remain editable. After closing the dialog, use Back to model to return to the selected model with your deployment settings intact. Browser Back also preserves the selection. Storage choices and model download status refresh when you return.

Change the runtime image only when you have an image compatible with the selected hardware. After changing the cluster or Guide source, check the variant selection again. A resource or network error during generation must be resolved before deployment.

Get an AI-assisted recommendation​

  1. Open a model and fill in the deployment name, cluster and cache storage.
  2. Set Mode to Agentic.
  3. Choose Use case and GPU capacity for planning, then review the workload and performance-target fields.
  4. Leave AI provider on Default (no external AI provider), or select a configured provider to help rank the options.
  5. Optional: expand Preference and enter an English preference, such as use fewer GPU resources or prefer prefix cache.
  6. Click Generate recommendation.
  7. Under Validated alternatives, expand Why this option to compare resource use and estimated performance.
  8. Click Deploy recommendation on the option you choose. Track startup on Deployments.

Optionally, expand Pod placement before generating a recommendation to choose where the deployment runs.

To change the recommendation, enter your revised preference below the alternatives and click Recalculate selection. This also works without an external AI provider.

AI score ranks the alternatives; it is not a measured performance result or a probability of success. Treat SLO not met as a reason to revise the target or resources. Run an evaluation to measure the deployed model.

Adjust optional settings​

GoalWhere to change it
Change context length or batchingAdvanced settings in the Manual configuration editor
Enable tool callingEnable automatic tool choice, then choose Tool call parser
Use a deployment-specific proxyDeployment environment
Change replicas or accelerator countReplicas and Tensor parallelism / TP

If startup reports insufficient KV-cache capacity, lower the maximum context length or allocate more resources in a new configuration.

If Your role cannot create deployments. appears, ask an administrator for deployment access to the target cluster. Open Deployments still lets you view deployments you can access.