​
Autoscaling and costs

Q: How does autoscaling affect my costs?
  • Scaling from 0: No minimum cost when scaled to zero
  • Scaling up: Each new replica adds to your total cost proportionally. For example:
    • Scaling from 1 to 2 replicas doubles your GPU costs
    • If each replica uses multiple GPUs, costs scale accordingly (e.g., scaling from 1 to 2 replicas with 2 GPUs each means paying for 4 GPUs total)
For current pricing details, please visit our pricing page.

​
Rate-limits for on-demand deployment

Q: What are the rate limits for on-demand deployments? Request throughput scales with your GPU allocation. Base allocations include:
  • Up to 8 A100 GPUs
  • Up to 8 H100 GPUs
On-demand deployments offer several advantages:
  • Predictable pricing based on time units, not token I/O
  • Protected latency and performance, independent of traffic on the serverless platform
  • Choice of GPUs, including A100s and H100s
Need more GPUs? Contact us to discuss higher allocations for your specific use case.

​
On-demand billing

Q: How does billing work for on-demand deployments? On-demand deployments come with automatic cost optimization features:
  • Default autoscaling: Automatically scales to 0 replicas when not in use
  • Pay for what you use: Charged only for GPU time when replicas are active
  • Flexible configuration: Customize autoscaling behavior to match your needs
Best practices for cost management:
  1. Leverage default autoscaling: The system automatically scales down deployments when not in use
  2. Customize carefully: While you can modify autoscaling behavior using our configuration options, note that preventing scale-to-zero will result in continuous GPU charges
  3. Consider your use case: For intermittent or low-frequency usage, serverless deployments might be more cost-effective
For detailed configuration options, see our deployment guide.

​
Scaling structure

Q: How does billing and scaling work for on-demand GPU deployments? On-demand GPU deployments have unique billing and scaling characteristics compared to serverless deployments: Billing:
  • Charges start when the server begins accepting requests
  • Billed by GPU-second for each active instance
  • Costs accumulate even if there are no active API calls
Scaling options:
  • Supports autoscaling from 0 to multiple GPUs
  • Each additional GPU adds to the billing rate
  • Can handle unlimited requests within the GPU’s capacity
Management requirements:
  • Not fully serverless; requires some manual management
  • Manually delete deployments when no longer needed
  • Or configure autoscaling to scale down to 0 during inactive periods
Cost control tips:
  • Regularly monitor active deployments
  • Delete unused deployments to avoid unnecessary costs
  • Consider serverless options for intermittent usage
  • Use autoscaling to 0 to optimize costs during low-demand times

​
Additional resources