Direct routing

​

Internet direct routing

Internet direct routing bypasses our global API load balancer and directly routes your request to the machines where your deployment is running. This can save several tens or even hundreds of milliseconds of time-to-first-token (TTFT) latency.

To create a deployment using Internet direct routing:

$ firectl create deployment accounts/fireworks/models/llama-v3p1-8b-instruct \
    --direct-routing-type INTERNET \
    --direct-route-api-keys <API_KEYS>

Name: accounts/my-account/deployments/abcd1234
...
Direct Route Handle: my-account-abcd1234.us-arizona-1.direct.fireworks.ai
Region: US_ARIZONA_1

You will need to specify a comma-separated list of API keys that can access the direct route deployment. These keys can be any alpha-numeric string and are a distinct concept from the API keys provisioned via the Fireworks console. A key provisioned in the console but not specified the list here will not be allowed when querying the model via direct routing.

Take note of the Direct Route Handle to get the inference endpoint. This is what you will use access the deployment instead of the global https://api.fireworks.ai/inference/ endpoint. For example:

curl \
    --header 'Authorization: Bearer <FIREWORKS_API_KEY>' \
    --header 'Content-Type: application/json' \
    --data '{
        "model": "accounts/fireworks/models/llama-v3-8b-instruct",
        "prompt": "The sky is"
    }' \
    --url https://my-account-abcd1234.us-arizona-1.direct.fireworks.ai/v1/completions

​

Private Service Connect (PSC)

Contact your Fireworks representative to set up GCP Private Service Connect to your deployment.

​

AWS PrivateLink

Contact your Fireworks representative to set up AWS PrivateLink to your deployment.