> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fireworks.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Fireworks Control Plane
> Set up Fireworks on AWS EKS with Fireworks-managed control plane
Fireworks currently only supports Bring-your-own-cloud (BYOC) with AWS. We are working to support
other cloud providers.
This page will guide you through setting up your Fireworks cluster on your own AWS cloud account using [EKS](https://aws.amazon.com/eks/), and then creating a model and deployment on that cluster.
This setup uses the Fireworks-managed control plane, which enables you to manage models and deployments via `firectl` CLI and monitor metrics through the Fireworks web app.
## Set up your cluster
The first step is to create the resources necessary for running a Fireworks cluster
in your own AWS account. Fireworks has made this easy by preparing a [Terraform module](https://registry.terraform.io/modules/fw-ai-external/aws-cluster/fireworksai/latest)
that can be applied. With this module you can set the input variables to suit your existing cloud set up.
For example, the module can use an existing VPC or subnet, create nodes in various availability zones, etc.
The Fireworks team can help with this configuration process.
The following is an example configuration that creates a new VPC and new subnets, and creates a cluster with a single H100 node in one availability zone:
```terraform theme={null}
module "fireworks_cluster" {
source = "fw-ai-external/aws-cluster/fireworksai"
version = "0.1.2"
vpc = {
cidr = "172.19.0.0/16" # An IP range that does not conflict with any existing VPCs
}
availability_zones = {
"us-east-1a" = {
public_cidr = "172.19.0.0/20" # must be within the VPC range
private_cidr = "172.19.16.0/20" # must be within the VPC range
node_count = "1"
instance_type = "p5.48xlarge" # p4d.24xlarge and p4de.24xlarge are also supported
}
"us-east-1b" = {
public_cidr = "172.19.32.0/20" # must be within the VPC range
private_cidr = "172.19.48.0/20" # must be within the VPC range
node_count = "1"
instance_type = "p5.48xlarge" # p4d.24xlarge and p4de.24xlarge are also supported
}
}
cluster_name = "my-cluster"
}
output "fireworks_cluster" {
value = module.fireworks_cluster
}
```
The module will create the following resources:
1. A VPC network (or reuse an existing one in your account)
2. Subnets for the VPC (or reuse existing ones in your account)
3. Various roles for operating the cluster
4. An EKS cluster
5. A storage bucket for model artifacts
6. Container registries to hold images uses by the cluster
Applying this module will output the following data about your resources:
```json theme={null}
fireworks_cluster = {
"cluster_node_role_arn" = "arn:aws:iam:::role/FireworksClusterNodeRole"
"eks_cluster_autoscaler_role_arn" = "arn:aws:iam:::role/FireworksEKSClusterAutoscalerRole"
"eks_cluster_role_arn" = "arn:aws:iam:::role/FireworksEKSClusterRole"
"eks_load_balancer_controller_role_arn" = "arn:aws:iam:::role/FireworksEKSLoadBalancerControllerRole-"
"fireworks_manager_role_arn" = "arn:aws:iam:::role/FireworksManagerRole"
"inference_role_arn" = "arn:aws:iam:::role/FireworksInferenceRole-"
"llm_downloader_ecr_repo_uri" = ".dkr.ecr.us-east-1.amazonaws.com/fireworks/llm-downloader"
"metrics_writer_role_arn" = "arn:aws:iam:::role/FireworksMetricWriterRole-"
"s3_bucket_arn" = "arn:aws:s3:::fireworks--"
"text_completion_ecr_repo_uri" = ".dkr.ecr.us-east-1.amazonaws.com/fireworks/text-completion"
"vpc_id" = "vpc-12345678901234567"
}
```
The Terraform module assumes that you or an administrator will have permission to upload to the created S3 bucket. If this is not accurate, you will need to add an additional piece of Terraform code granting that permission.
Most of the cluster set up can be done automatically by Fireworks, but before we can access your cluster you will need to apply the following Kubernetes manifest. Note that the `rolearn` fields will need to be filled in with values taken from the `fireworks_cluster` output from the previous step.
```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
namespace: kube-system
name: aws-auth
data:
mapRoles: |
- groups:
- system:masters
rolearn:
username: inference-cluster-manager
- groups:
- system:bootstrappers
- system:nodes
rolearn:
username: system:node:{{EC2PrivateDNSName}}
```
```bash theme={null}
kubectl apply -f /path/to/manifest
```
To continue the setup of your cluster, send Fireworks the Terraform output from step one, and also the ID you'd like to give this cluster (or you can use the same name as the created EKS cluster)
After Fireworks finishes setting up your cluster, you should be able to see the cluster in `firectl`.
```bash theme={null}
$ firectl cluster list
NAME REGION STATE STATUS MESSAGE
my-cluster us-east-1 READY
Total size: 1
```
## Upload a model to your cluster
Now you can create a model in your new cluster. These steps are mostly the same as in [Custom base models](/models/uploading-custom-models#customizing-base-model-configuration), with the following differences:
1. Add the `--cluster-id=` argument to your `firectl model create` command.
2. By default `firectl` will attempt to upload the model files to the S3 bucket associated with your cluster. If your default AWS profile has permission to do this, you don't need to do anything else. If you need to use a different AWS profile, you can set the `AWS_PROFILE` environment variable as it will be respected by `firectl`. If neither of those work, you can pass the `--manual-upload` flag to `firectl model create`. This will output a series of `aws s3 cp` commands that can be run to manually upload the model files.
Once your model is in the `READY` state and you've uploaded the model's files to S3, you can continue to the next section.
```bash theme={null}
$ firectl list models
NAME CREATE TIME KIND CHAT PUBLIC STATE STATUS MESSAGE
my-model 2024-01-01 00:00:00 HF_BASE_MODEL false false READY
```
## Create a deployment in your cluster
Creating a deployment in your cluster is similar to [creating an on-demand deployment](/guides/ondemand-deployments):
```bash theme={null}
firectl deployment create my-cluster --cluster-id=my-cluster --min-replica-count=1 --accelerator-type=NVIDIA_A100_40GB --wait
```
After a few minutes your deployment should be `READY`
```bash theme={null}
$ firectl deployment list ✘ INT venv-fireworks 04:12:48 PM
NAME BASE MODEL REPLICAS GPUS/REPLICA TYPE CREATE TIME STATE STATUS MESSAGE
12345678 accounts/my-account/models/my-cluster 1 [1,1] 1 NVIDIA_A100_40GB 2024-01-01 00:00:00 READY
```
Some on-demand deployment features (like autoscaling) are not yet available on
BYOC clusters.
## Query your deployment
The final step is to test your deployment by querying it. To do this you will need to retrieve the hostname of the the load balancer that was automatically created for your cluster
Navigate to the "Load Balancer" section in the EC2 console. The created
load balancer will have a name that starts with `k8s-default-gateway`. Click this
row, and you'll find the host name under "DNS name".
```bash theme={null}
kubectl get gateway gateway -o json | jq ".status.addresses[0].value"
```
Now you can query your deployment by passing the header `Fireworks-Deployment` to the load balancer:
```python theme={null}
from fireworks.client import Fireworks
client = Fireworks(
base_url="http:///v1",
# API key field is not used for BYOC deployments but cannot be blank
api_key="unused",
)
response = client.completions.create(
model="accounts/my-account/models/my-model",
prompt="The sky is",
extra_headers={
"Fireworks-Deployment": "accounts/my-account/deployments/12345678",
},
)
print(response.choices[0].text)
```
```python theme={null}
from openai import OpenAI
client = OpenAI(
base_url="http:///v1",
# API key field is not used for BYOC deployments but cannot be blank
api_key="unused",
)
response = client.completions.create(
model="accounts/my-account/models/my-model",
prompt="The sky is",
extra_headers={
"Fireworks-Deployment": "accounts/my-account/deployments/12345678",
},
)
print(response.choices[0].text)
```
```bash theme={null}
curl http:///v1/completions \
-H 'Fireworks-Deployment: accounts/my-account/deployments/12345678' \
-H 'Content-Type: application/json' \
--data '{
"model": "accounts/my-account/models/my-model",
"prompt": "The sky is"
}'
```
You will need to run this from an environment that has access to the subnet associated with the cluster. An easy way to enter such an environment is to spin up a `curl` pod in the cluster:
```bash theme={null}
kubectl run curl-test --image=radial/busyboxplus:curl -i --tty --rm
```