Quantization

Quantization reduces the number of bits used to serve a model, improving performance and reducing cost by 30-50%. However, this can change model numerics which may introduce small changes to the output.

Read our blog post for a detailed treatment of how quantization affects model quality.

Checking available precisions

Models may support different numerical precisions like FP16, FP8, BF16, or INT8, which affect memory usage and inference speed. Check default precision:

firectl model get accounts/fireworks/models/llama-v3p1-8b-instruct | grep "Default Precision"

Check supported precisions:

firectl model get accounts/fireworks/models/llama-v3p1-8b-instruct | grep -E "(Supported Precisions|Supported Precisions With Calibration)"

The Precisions field indicates what precisions the model has been prepared for.

Quantizing a model

A model can be quantized to 8-bit floating-point (FP8) precision.

firectl
Python (REST API)

firectl prepare-model <MODEL_ID>

import os
import requests

ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
API_KEY = os.environ.get("FIREWORKS_API_KEY")
MODEL_ID = "<YOUR_MODEL_ID>" # The ID of the model you want to prepare

response = requests.post(
  f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/models/{MODEL_ID}:prepare",
  headers={
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
  },
  json={
    "precision": "FP8"
  }
)

print(response.json())

This is an additive process that enables creating deployments with additional precisions. The original FP16 checkpoint is still available for use.

You can check on the status of preparation by running:

firectl
Python (REST API)

firectl model get <MODEL_ID>

import os
import requests

ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
API_KEY = os.environ.get("FIREWORKS_API_KEY")
MODEL_ID = "<YOUR_MODEL_ID>" # The ID of the model you want to get

response = requests.get(
  f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/models/{MODEL_ID}",
  headers={
    "Authorization": f"Bearer {API_KEY}"
  }
)

print(response.json())

and checking if the state is still in PREPARING. A successfully prepared model will have the desired precision added to the Precisions list.

Creating an FP8 deployment

By default, creating a deployment uses the FP16 checkpoint. To use a quantized FP8 checkpoint, first ensure the model has been prepared for FP8 (see Checking available precisions above). Do not deploy without a deployment shape — a shape that includes FP8 precision is the only validated path. List the model’s shapes with firectl deployment-shape-version list --base-model <MODEL> and look for one with the precision you want. If no FP8 shape exists for your model, start from the closest shape and override the precision with --precision. Do not create the deployment without a shape entirely: it skips validation, is the most common cause of failed deployment creations, and the unshaped path may be deprecated in the future:

firectl
Python (REST API)

firectl deployment create <MODEL> --deployment-shape <shape> --precision FP8

import os
import requests

ACCOUNT_ID = os.environ.get("FIREWORKS_ACCOUNT_ID")
API_KEY = os.environ.get("FIREWORKS_API_KEY")
# The ID of the model you want to deploy.
# The model must be prepared for FP8 precision.
MODEL_ID = "<YOUR_MODEL_ID>"
DEPLOYMENT_NAME = "My FP8 Deployment"

response = requests.post(
  f"https://api.fireworks.ai/v1/accounts/{ACCOUNT_ID}/deployments",
  headers={
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json"
  },
  json={
    "displayName": DEPLOYMENT_NAME,
    "baseModel": MODEL_ID,
    "deploymentShape": "<SHAPE_ID>",
    "precision": "FP8",
  }
)

print(response.json())

Quantized deployments can only be served using H100 GPUs.

Get Started

Serverless Inference

Deployments

Models & Inference

Training

Managed Training

Training API

Fire Pass

Fireworks Nexus

Administration

Security & Compliance

Integrations

Reference

Checking available precisions

Quantizing a model

Creating an FP8 deployment

​Checking available precisions

​Quantizing a model

​Creating an FP8 deployment

Checking available precisions

Quantizing a model

Creating an FP8 deployment