Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 18 additions & 4 deletions skills/dstack-prototyping/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,19 +101,33 @@ command, resources, cache, or model behavior needs to change, go back to a task.
If the tested serving setup is still right and only the dstack service
configuration is wrong, fix the configuration and submit the service again.

## PD disaggregation
## Router

If the intention is to use PD disaggregation:
If a fleet has `placement: cluster` and a CPU-only instance, it is generally
recommended to use a router configuration, regardless of whether the workers
are aggregated or PD disaggregated, with the router on the CPU-only instance.
Whenever possible, connect the workers over gRPC, not HTTP: with a gRPC
router, request parsing, serialization, and tokenization move from the
serving engine to the router, so latency improves just by introducing it.

When using a router:

- Use node groups for the task and replica groups for the service: tasks' node
groups are the equivalent of services' replica groups.
- In both cases, you run a router and prefill/decode workers separately, and
you need to use a fleet with an interconnect (`placement: cluster`).
- With tasks, still use `sleep infinity` even when using `groups` (set it in
each group's `commands`; top-level `commands` is not allowed with `groups`),
and run the actual commands on each node interactively over SSH.
- When testing inference, call the router endpoint, not the workers directly
(unless you want to test if they are alive).
- Look for "Prototyping services" in `https://dstack.ai/docs/concepts/tasks.md`
and "Router" in `https://dstack.ai/docs/concepts/services.md`.

## PD disaggregation

If the intention is to use PD disaggregation:

- Follow `## Router`: the router and the prefill/decode workers run as
separate groups, and the fleet needs an interconnect (`placement: cluster`).
- Look for "Node groups" and "PD disaggregation" in
`https://dstack.ai/docs/concepts/tasks.md` and "Replica groups" and
"PD disaggregation" in `https://dstack.ai/docs/concepts/services.md`.
Original file line number Diff line number Diff line change
Expand Up @@ -260,9 +260,9 @@ mindful of which specific change was the root cause.
{"resources": {...}, "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...}
```

2. In case the task is using node groups (e.g. for PD disaggregation),
instead of a single `resources` it includes `groups`, and the fields are
these and no others:
2. In case the task is using node groups (a router group plus worker
groups, see `## Cluster Placement`), instead of a single `resources` it
includes `groups`, and the fields are these and no others:

```
{"groups": [[{...}], [{...}, {...}], [{...}]], "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...}
Expand Down Expand Up @@ -321,9 +321,9 @@ serving engine: use <!--?if dataset-->`dataset` and `concurrency`<!--?else-->`co
`shared_prefix_tokens`<!--?end--> from `constraints.json` and measure all trials the same
way so that their results are comparable with each other.

In case the task is using PD disaggregation, run benchmarks via SSH inside
the router node, directly against the router engine. Never benchmark prefill
or decode workers — each handles only part of a request.
In case the task is using a router (see `## Cluster Placement`), run benchmarks
via SSH inside the router node, directly against the router engine, never against a
worker, so that results stay comparable across trials.

<!--?if dataset-->
Before any benchmark, reset the serving engine's prefix cache, or restart the
Expand Down Expand Up @@ -506,6 +506,13 @@ Use these offers when selecting fleet, backend, and hardware.
To classify each backend's capabilities, fetch `https://dstack.ai/docs/concepts/backends.md` and classify from the fetched document, not from memory. VM-based backends are listed under `## VM-based` (they support idle instances and instance volumes). Kubernetes backend is listed under `## Container-based`, but supports instance volumes and thus is preferred over other container-based backends.
SSH fleets can be treated as VM-based backends as they support both idle instances (its equivalent) and instance volumes.

## Cluster Placement

If a fleet has `placement: cluster`, decide before trial 1 whether to use a
router, following `## Router` in the
`<!--?if codex-->$<!--?else-->/<!--?end-->dstack-prototyping` skill, and
report the decision via `progress`.

# Final Service

Once the trials are over, pick the best trial that has not been verified yet
Expand Down Expand Up @@ -565,9 +572,9 @@ trial benchmarks so that the results are comparable with each other. Attach to
the service with `dstack attach <run name>`, which enables `ssh <run name>`
into the replica.

In case the service is using PD disaggregation, run the final benchmark
inside the router replica, directly against the router engine. Attach to it
via `dstack attach <run name> --replica <replica num> --job <job num>`.
In case the service is using a router (see `## Cluster Placement`), run the
final benchmark inside the router replica, directly against the router engine.
Attach to it via `dstack attach <run name> --replica <replica num> --job <job num>`.

If the service or its benchmark cannot be completed, stop that service,
pick the next-best trial, and repeat, until a service is verified or there
Expand Down
Loading