[Presets] Update system_prompt for router usage

[Bihan]

Previously, presets only used a separate router for PD disaggregation. Now, whenever a fleet has placement: cluster and a CPU-only instance, presets host an SMG router on that instance and run every trial and the final service behind it, with workers connected over gRPC.

[peterschmidt85]

Please find my comments!

[peterschmidt85]

against the router -> against the router engine

Also, curious, why change ". Never benchmark prefill or decode workers — each handles only part of a request."

[Bihan]

Also, curious, why change ". Never benchmark prefill or decode workers — each handles only part of a request."

Because the rule now applies to aggregated workers too, where there are no prefill or decode workers.

It can also be re-added without breaking the unification, for example: ...never against a worker, so that results stay comparable across trials; with PD disaggregation each worker also handles only part of a request.

[peterschmidt85]

I do not like to introduce new terms, such as Topology. I assume, we normally say Cluster placement.

Also, are you sure this solely belongs to this system prompt and not dstack-prototyping skill?

I don't mind referencing the prototyping skill here and cluster placement / router specifically, but I somewhat expect https://github.com/dstackai/dstack/blob/master/skills/dstack-prototyping/SKILL.md#pd-disaggregation to be responsible for it, no?

If a fleet has placement: cluster and a CPU-only instance, that instance
must host a router
I would instead say that it's generally recommended to use a router configuration regardless if its PD aggregated or PD disaggregated configuration for both tasks (reference Prototyping services

dstack/mkdocs/docs/concepts/tasks.md

Line 201 in 3c0d048

??? info "Prototyping services"

) and services (

dstack/mkdocs/docs/concepts/services.md

Line 168 in 3c0d048

### Router

)

Also what I consider low-value text:
"and every trial and the final service run behind it" - obvious
"The GPU instances run
the serving engine as workers in one of two forms: aggregated, where each
worker handles both prefill and decode, or PD disaggregated, where the
workers are split into prefill and decode groups. Use router: sglang
(Shepherd Model Gateway, SMG)." - our referenced documentation describes it well.

"First, cache-aware routing tracks each worker's KV-cache state and sends a
request to the worker that already holds its prefix.

Second, load modeling predicts how long a request will take to complete on
each worker and the router's least_load policy sends the request to the
worker predicted to finish it soonest." - imo agent will understand it itself

connect the workers over gRPC, not HTTP
I would add "Whenever possible, "

[Bihan]

@peterschmidt85
After resolving the review comments, hosting the router on the CPU instance is no longer mandatory; it is "generally recommended". With that wording the preset agent now logs:

Router decision: none. There is one GPU worker, so PD disaggregation isn't possible. A router on the CPU node would add a cross-node hop for only small tokenization savings with a 3B model at c=16.

For a deployment this small the reasoning is correct.

However, I suggest we go back to "must use a router when a cluster fleet has a CPU-only instance" rather than "generally recommended".

A CPU-only instance in a cluster fleet is a deliberate choice that signals the intent to serve larger models in a production. The SMG team's own benchmarks (https://pytorch.org/blog/lightseek-smg/) show the router's gains grow with GPU speed, model size and concurrency, with the largest result on Llama-3.3-70B-FP8 on H100, which is minimum size of deployment with presets. Victor's study also measured a latency improvement from introducing the router alone.

With "generally recommended", we leave this to agent, when it is clear that CPU instance is for router.