Skip to main content
Control how your deployment scales based on traffic and load.

Configuration options

Load target options (use as --load-targets <key>=<value>[,<key>=<value>...]):
  • default=<Fraction> - General load target from 0 to 1
  • tokens_generated_per_second=<Integer> - Desired tokens per second per replica
  • prompt_tokens_per_second=<Integer> - Desired prompt tokens per second per replica
  • requests_per_second=<Number> - Desired requests per second per replica
  • concurrent_requests=<Number> - Desired concurrent requests per replica
When multiple targets are specified, the maximum replica count across all is used. When you create a deployment, default=0.8 is added if you don’t set default yourself. firectl deployment update --load-targets replaces all existing load targets, so include default in the list if you want to keep it:
The console sets one load target at a time, and saving a change there replaces all existing load targets. To combine targets, such as default and concurrent_requests, use firectl.
On deployments served by vLLM or SGLang, concurrent_requests counts the requests the engine is actively running. Requests queued inside the engine are not included.

How scaling decisions work

The autoscaler regularly works out, for each load target, how many replicas would bring the load to that target, and uses the largest of those numbers, between your minimum and maximum replica counts. Scale-up window. The autoscaler applies the lowest replica count recommended during the last --scale-up-window. A higher count takes effect only after it has been recommended for the whole window. New replicas then need time to start before they serve traffic, which depends on model size, so extra capacity arrives roughly one window plus the start-up time after load rises. A shorter window reacts faster; a longer one ignores brief spikes. Scale-down window. The autoscaler applies the highest replica count recommended during the last --scale-down-window, so a lower count takes effect only after load has stayed lower for the whole window. How far one step can go. default tracks how busy each replica is, using measures such as KV-cache usage, compute utilization, or the share of requests waiting, depending on the serving engine. These measures top out at 100% of a replica, so with only default set, one scaling step can raise the replica count to at most about 1 / target times the replicas serving traffic: about 25% more (and at least one more replica) at the default 0.8. Under a sudden traffic spike the deployment therefore grows in several steps. Per-replica targets such as concurrent_requests or requests_per_second are not limited to 100%, so a large backlog can add several replicas in one step. To absorb sharp traffic increases, combine these with a --min-replica-count sized for your expected baseline traffic.

Common patterns

Scale to zero when idle to minimize costs:
Best for: Development, testing, or intermittent production workloads.

Scaling from zero behavior

When a deployment is scaled to zero and receives a request, the system immediately returns a 503 error with the DEPLOYMENT_SCALING_UP error code while initiating the scale-up process:
Requests to a scaled-to-zero deployment are not queued. Your application must implement retry logic to handle 503 responses while the deployment scales up.

Handling scale-from-zero responses

Implement retry logic with exponential backoff to gracefully handle scale-up delays:
Cold start times vary depending on model size—larger models may take longer to download and initialize. If you need instant responses without cold starts, set --min-replica-count 1 or higher to keep replicas always running.
Deployments with min replicas = 0 are auto-deleted after 7 days of no traffic. Reserved capacity guarantees availability during scale-up.