Skip to content

[Bug]: Custom probes may break worker registration with type: sglang router #4310

Description

@un-def

Steps to reproduce

type: service
name: smg

env:
  - MODEL_ID=Qwen/Qwen2.5-0.5B-Instruct

groups:
  - replicas: 1
    python: "3.12"
    commands:
      - pip install smg
      - |
        smg launch \
          --host 0.0.0.0 \
          --port 8000 \
          --enable-igw \
          --policy cache_aware \
          --model-path $MODEL_ID
    router:
      type: sglang
    resources:
      gpu: 0

  - replicas: 1
    image: lmsysorg/sglang:v0.5.10.post1
    commands:
      - |
        python3 -m sglang.launch_server \
          --model-path $MODEL_ID \
          --host 0.0.0.0 \
          --port 8000 \
          --context-length 4096
    resources:
      gpu:
        memory: 16GB..
        # SGLang's CuTe DSL kernels don't compile for Turing (sm_75, e.g. T4)
        compute_capability: 8.0

port: 8000

model:
  type: chat
  name: Qwen/Qwen2.5-0.5B-Instruct
  format: openai
  prefix: /v1

probes:
  - type: http
    url: /health_generate
    interval: 5s

/health_generate is a correct probe for both Shepherd Model Gateway and SGLang Server – “Check the health [...] by generating one token” – roughly the same as dstack's _openai_model_probe_spec; but adding this probe reintroduces the same kind of deadlock that was fixed in #4270. Replacing the custom probe with probes: [] solves the issue.

Actual behaviour

The worker is ready, the router is never ready (see the PROBES column):

 NAME                               BACKEND         RESOURCES                                    INSTANCE TYPE  PRICE    STATUS   PROBES  SUBMITTED   ERROR
 smg deployment=0                                   -                                                           -        running          7 mins ago
    group=0 replica=0 deployment=0  gcp (us-west1)  cpu=x86:2 mem=2GB disk=20GB                  e2-highcpu-2   $0.0495  running  ×       7 mins ago
    group=1 replica=1 deployment=0  gcp (us-west1)  cpu=x86:4 mem=16GB disk=100GB gpu=L4:24GB:1  g2-standard-4  $0.7068  running  ✓       7 mins ago

Server logs:

# Both are not ready
dstack._internal.server.background.scheduled_tasks.probes:122 probe(fa83a6)smg-0-0-0: probe status code: 503
dstack._internal.server.background.scheduled_tasks.probes:125 probe(ea13f1)smg-0-1-0: probe failed: ReadError('')

# Both are not ready
dstack._internal.server.background.scheduled_tasks.probes:122 probe(fa83a6)smg-0-0-0: probe status code: 503
dstack._internal.server.background.scheduled_tasks.probes:122 probe(ea13f1)smg-0-1-0: probe status code: 503

# The worker is ready; the router returns 503 infinitely as there is no registered worker
dstack._internal.server.background.scheduled_tasks.probes:122 probe(fa83a6)smg-0-0-0: probe status code: 503
dstack._internal.server.background.scheduled_tasks.probes:122 probe(ea13f1)smg-0-1-0: probe status code: 200

/health_generate response from the router:

GET /health_generate HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: localhost:8000
User-Agent: HTTPie/3.2.4



HTTP/1.1 503 Service Unavailable
access-control-allow-origin: *
access-control-expose-headers: *
content-length: 110
content-type: application/json
date: Thu, 24 Sep 2026 09:19:06 GMT
x-request-id: req-N5bKymgO02OuEHl01IoT6pE5
x-smg-error-code: no_workers

{
    "error": {
        "code": "no_workers",
        "message": "No workers are available",
        "param": null,
        "type": "Service Unavailable"
    }
}

/health_generate response from the worker:

GET /health_generate HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: localhost:8000
User-Agent: HTTPie/3.2.4



HTTP/1.1 200 OK
content-length: 0
date: Thu, 24 Sep 2026 09:19:59 GMT
server: uvicorn

Expected behaviour

No response

dstack version

39b28c2

Server logs

DEBUG    dstack._internal.server.services.runs.router_worker_sync:568 run(f7ba4f)smg: no ready router job in group 0, skipping worker sync

Additional information

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions