Service Autoscaling

Service autoscaling is an early-access feature. The syntax described here may evolve in future releases.

A workflow service normally runs as a single, long-lived container. With an autoscaler block you can instead run a service as a pool of replicas whose size grows and shrinks automatically based on metrics the service itself exposes. This is useful for workloads whose demand varies over time — inference servers, queue workers, or any service that benefits from running more (or fewer) copies as load changes.

The autoscaler decides when to add or remove a replica by evaluating PromQL expressions against the service’s own metrics. Replicas of an autoscaled service are reached through a single DNS name that always resolves to the currently-ready replicas, so clients do not need to know how many replicas exist at any moment.

When a service declares an autoscaler, Fuzzball:

1. Pre-allocates capacity for up to replicas.max copies of the service, starting replicas.min of them (which may be 0).
2. Scrapes the service’s metrics endpoint (when metrics.enabled is set) at the configured interval.
3. Evaluates each scale-up trigger and the scale-down policy against those metrics. When a scale-up trigger fires, the next pending replica is started; when scale-down fires, a running replica is stopped.
4. Maintains a DNS record for the service that contains one address per ready replica, adding a replica’s address once it passes its readiness probe and removing it when the replica is scaled down.

Each scaling action is rate-limited by its cooldown-period, and the replica count is always bounded by replicas.min and replicas.max.

The autoscaler evaluates PromQL expressions against metrics collected from your services. Fuzzball supports two types of metrics:

When you enable metrics.enabled: true on a service, Fuzzball scrapes a Prometheus-compatible metrics endpoint from each replica at the configured interval. These are metrics that your service application exposes — for example, vllm:num_requests_waiting from vLLM, or custom business metrics from your application.

Configure the scrape with:

• metrics.path — HTTP path to the metrics endpoint (default /metrics)
• metrics.port — TCP port where metrics are served
• metrics.interval — Scrape interval in seconds (default 15)

All scraped metrics are tagged with service and replica labels, so your PromQL queries can aggregate across the pool (e.g., sum(my_metric) by (service)) or target specific replicas.

For every autoscaled service (whether or not it exposes its own metrics), Fuzzball automatically collects synthetic container-level metrics from the underlying runtime. These metrics are available immediately without requiring your service to expose a /metrics endpoint, making them ideal for simple CPU- or memory-based scaling policies.

The default metrics follow the cAdvisor naming convention:

These metrics are collected from the container runtime at the same interval as service-exposed metrics (or every 15 seconds if no scrape is configured) and are tagged with the same service and replica labels.

Example: Scale up when average CPU usage across all replicas exceeds 80% of one core for one minute:

autoscaler:
  replicas:
    min: 1
    max: 10
  scale-up:
    triggers:
      - metrics-query: rate(container_cpu_usage_seconds_total[1m]) >= bool 0.8
        cooldown-period: 60

Example: Scale down when a replica’s memory usage falls below 50% of its limit:

autoscaler:
  scale-down:
    metrics-query: |
      (container_memory_usage_bytes / container_memory_limit_bytes) <= bool 0.5
    cooldown-period: 300

Container metrics are synthetic — they are derived from container runtime stats rather than scraped from an endpoint. This means they appear in your PromQL queries just like scraped metrics, but your service does not need to export them.

Replicas of a self-scaling service share a single autoscaler DNS name of the form:

<service>.autoscaler.<workflow-id>.fuzzball

Resolving this name returns one address for every ready replica, so a client (or a load balancer) can distribute requests across the pool. Because the replicas are reached through this shared record, a self-scaling service cannot declare per-replica network endpoints.

The most common pattern is a service that scales its own replicas. Define replicas and one or more scale-up triggers (with no services field) plus an optional scale-down policy:

version: v1
services:
  inference:
    image:
      uri: docker://vllm/vllm-openai:latest
    command: ["--model", "facebook/opt-125m", "--port", "8000"]
    resource:
      cpu:
        cores: 8
      memory:
        size: 16GB
    network:
      ports:
        - name: http
          port: 8000
          protocol: tcp
    readiness-probe:
      http-get:
        path: /health
        port: 8000
      period-seconds: 5
    autoscaler:
      replicas:
        min: 1
        max: 8
      metrics:
        enabled: true
        path: /metrics
        port: 8000
        interval: 15
      scale-up:
        triggers:
          - metrics-query: vllm:num_requests_waiting >= bool 2
            cooldown-period: 60
      scale-down:
        metrics-query: vllm:num_requests_running == bool 0
        cooldown-period: 300

This service starts with one replica, scales up (one replica at a time, at most once per minute) whenever two or more requests are waiting, and scales back down — no more often than every five minutes — when a replica has no requests running.

Use the PromQL bool modifier in comparisons (>= bool 2, == bool 0) so the query returns 1 when the condition holds and 0 when it does not. The autoscaler treats any non-zero result as a trigger to act.

Setting replicas.min: 0 lets a service idle with no running replicas and start its first replica only when a trigger fires. This is well suited to bursty or on-demand workloads where you do not want to hold capacity while the service is idle.

Instead of scaling its own replicas, a service may act as a source that scales one or more other services. This is useful for bringing a pool of workers up from zero in response to demand observed by a frontend.

To do this, give the source service’s scale-up triggers a services list naming the target services. The targets define the replicas pool; the source does not:

version: v1
services:
  frontend:
    image:
      uri: docker://mycompany/frontend:latest
    network:
      ports:
        - name: metrics
          port: 9090
          protocol: tcp
    autoscaler:
      metrics:
        enabled: true
        path: /metrics
        port: 9090
        interval: 15
      scale-up:
        triggers:
          - metrics-query: queue_depth >= bool 10
            cooldown-period: 30
            services: [worker]

  worker:
    image:
      uri: docker://mycompany/worker:latest
    autoscaler:
      replicas:
        min: 0
        max: 20
      scale-down:
        metrics-query: worker_idle == bool 1
        cooldown-period: 120

Here the frontend watches its queue depth and, when it grows, scales the worker pool up from zero. Each worker owns its own scale-down policy, so it removes idle replicas independently of the source.

A service in a workflow can render a configuration file from the live addresses of other services and have it injected into its container, then re-rendered automatically as those services’ replicas scale up and down. This lets a service (such as a load balancer or proxy) keep an up-to-date list of its backends without restarting.

Add a dynamic-config block naming the services to watch, the destination path inside the container, and a script that emits the file contents. The addresses of each watched service are exposed to the script as a bash array named SERVICES_<NAME> (the service name uppercased, with any character that is not a valid identifier replaced by _):

services:
  proxy:
    image:
      uri: docker://haproxy:latest
    dynamic-config:
      path: /usr/local/etc/haproxy/backends.cfg
      services: [worker]
      script: |
        for ip in "${SERVICES_WORKER[@]}"; do
          cat <<EOF
        server ${ip} ${ip}:8080 check
        EOF
        done

The script runs in a heavily restricted interpreter for safety: only bash builtins and heredocs are available. External commands (other than a no-argument cat to support the cat <<EOF form) and any filesystem reads or writes are denied.

Fuzzball rejects a workflow at submission time if its autoscaler configuration is inconsistent. The most common rules are:

• A self-scaling service (no scale-up trigger targets other services) must define replicas and must not define network endpoints.
• replicas.max must be at least 1 and not less than replicas.min.
• A cross-service source (any scale-up trigger sets services) must not define its own replicas or a scale-down policy; each target service owns those.
• Every target named in a services list must exist, must define autoscaler.replicas, and must not define its own scale-up triggers (a pool is scaled up by exactly one source).
• All of a service’s scale-up triggers must agree: either every trigger targets other services, or none do. The two modes cannot be mixed within one service.
• A given service may be targeted by at most one scale-up trigger.
• scale-down may never target other services.
• dynamic-config may only reference services that exist in the workflow.

• Workflow syntax reference: services
• Workflow Endpoints
• Connecting to Workflow Services