Configuration options
Load target options (use as
--load-targets <key>=<value>[,<key>=<value>...]):
default=<Fraction>- General load target from 0 to 1tokens_generated_per_second=<Integer>- Desired tokens per second per replicaprompt_tokens_per_second=<Integer>- Desired prompt tokens per second per replicarequests_per_second=<Number>- Desired requests per second per replicaconcurrent_requests=<Number>- Desired concurrent requests per replica
default=0.8 is added if you don’t set default yourself. firectl deployment update --load-targets replaces all existing load targets, so include default in the list if you want to keep it:
default and concurrent_requests, use firectl.
On deployments served by vLLM or SGLang,
concurrent_requests counts the requests the engine is actively running. Requests queued inside the engine are not included.How scaling decisions work
The autoscaler regularly works out, for each load target, how many replicas would bring the load to that target, and uses the largest of those numbers, between your minimum and maximum replica counts. Scale-up window. The autoscaler applies the lowest replica count recommended during the last--scale-up-window. A higher count takes effect only after it has been recommended for the whole window. New replicas then need time to start before they serve traffic, which depends on model size, so extra capacity arrives roughly one window plus the start-up time after load rises. A shorter window reacts faster; a longer one ignores brief spikes.
Scale-down window. The autoscaler applies the highest replica count recommended during the last --scale-down-window, so a lower count takes effect only after load has stayed lower for the whole window.
How far one step can go. default tracks how busy each replica is, using measures such as KV-cache usage, compute utilization, or the share of requests waiting, depending on the serving engine. These measures top out at 100% of a replica, so with only default set, one scaling step can raise the replica count to at most about 1 / target times the replicas serving traffic: about 25% more (and at least one more replica) at the default 0.8. Under a sudden traffic spike the deployment therefore grows in several steps. Per-replica targets such as concurrent_requests or requests_per_second are not limited to 100%, so a large backlog can add several replicas in one step.
To absorb sharp traffic increases, combine these with a --min-replica-count sized for your expected baseline traffic.
Common patterns
- Cost optimization
- Performance-focused
- Predictable traffic
Scale to zero when idle to minimize costs:Best for: Development, testing, or intermittent production workloads.
Scaling from zero behavior
When a deployment is scaled to zero and receives a request, the system immediately returns a503 error with the DEPLOYMENT_SCALING_UP error code while initiating the scale-up process:
Handling scale-from-zero responses
Implement retry logic with exponential backoff to gracefully handle scale-up delays:- Python
- JavaScript
- curl
Deployments with min replicas = 0 are auto-deleted after 7 days of no traffic. Reserved capacity guarantees availability during scale-up.