- Model size and type
- Number of GPUs allocated to the deployment
- GPU type (e.g., A100, H100)
- Prompt size
- Generation token length
- Deployment type (serverless vs. on-demand)
Examples
FAQ
- Custom base models
- Serverless model availability
- Model availability requests
- Llama 3.1 405B quantization
- API batching & load balancing
- Request handling capacity
- Safety filter controls
- Token limit controls
- Streaming performance metrics
- FLUX multiple images
- FLUX image-to-image
- FLUX custom LoRA
- SDXL ControlNet sizing
Models & Inference
What factors affect the number of simultaneous requests that can be handled?
Request handling capacity depends on several factors:
Was this page helpful?
Does the API support batching and load balancing?
Previous
Can safety filters or content restrictions be disabled on text generation models?
Next
⌘I