Batch API

Process large volumes of requests asynchronously at 50% lower cost. Batch API is ideal for:

Production-scale inference workloads
Large-scale testing and benchmarking
Training smaller models with larger ones (distillation guide)

Batch jobs automatically use prompt caching for additional 50% cost savings on cached tokens. Maximize cache hits by placing static content first in your prompts.

Getting Started

1. Prepare Your Dataset

Datasets must be in JSONL format (one JSON object per line):Requirements:

File format: JSONL (each line is a valid JSON object)
Size limit: Under 500MB
Required fields: custom_id (unique) and body (request parameters)

Example dataset:

{"custom_id": "request-1", "body": {"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}], "max_tokens": 100}}
{"custom_id": "request-2", "body": {"messages": [{"role": "user", "content": "Explain quantum computing"}], "temperature": 0.7}}
{"custom_id": "request-3", "body": {"messages": [{"role": "user", "content": "Tell me a joke"}]}}

Save as batch_input_data.jsonl locally.

2. Upload Your Dataset

UI
firectl
HTTP API

You can simply navigate to the dataset tab, click Create Dataset and follow the wizard.

firectl create dataset batch-input-dataset ./batch_input_data.jsonl

You need to make two separate HTTP requests. One for creating the dataset entry and one for uploading the dataset. Full reference here: Create dataset.

# Create Dataset Entry
curl -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/datasets" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "datasetId": "batch-input-dataset",
    "dataset": { "userUploaded": {} }
  }'

# Upload JSONL file
curl -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/datasets/batch-input-dataset:upload" \
  -H "Authorization: Bearer ${API_KEY}" \
  -F "file=@./batch_input_data.jsonl"

3. Create a Batch Job

UI
firectl
HTTP API

Navigate to the Batch Inference tab and click “Create Batch Inference Job”. Select your input dataset:

Choose your model:

Configure optional settings:

firectl create batch-inference-job \
  --model accounts/fireworks/models/llama-v3p1-8b-instruct \
  --input-dataset-id batch-input-dataset

With additional parameters:

firectl create batch-inference-job \
  --job-id my-batch-job \
  --model accounts/fireworks/models/llama-v3p1-8b-instruct \
  --input-dataset-id batch-input-dataset \
  --output-dataset-id batch-output-dataset \
  --max-tokens 1024 \
  --temperature 0.7 \
  --top-p 0.9

curl -X POST "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/batchInferenceJobs?batchInferenceJobId=my-batch-job" \
  -H "Authorization: Bearer ${API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "accounts/fireworks/models/llama-v3p1-8b-instruct",
    "inputDatasetId": "accounts/'${ACCOUNT_ID}'/datasets/batch-input-dataset",
    "outputDatasetId": "accounts/'${ACCOUNT_ID}'/datasets/batch-output-dataset",
    "inferenceParameters": {
      "maxTokens": 1024,
      "temperature": 0.7,
      "topP": 0.9
    }
  }'

4. Monitor Your Job

UI
firectl
HTTP API

View all your batch inference jobs in the dashboard:

# Get job status
firectl get batch-inference-job my-batch-job

# List all batch jobs
firectl list batch-inference-jobs

# Get specific job
curl -X GET "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/batchInferenceJobs/my-batch-job" \
  -H "Authorization: Bearer ${API_KEY}"

# List all jobs
curl -X GET "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/batchInferenceJobs" \
  -H "Authorization: Bearer ${API_KEY}"

5. Download Results

UI
firectl
HTTP API

Navigate to the output dataset and download the results:

firectl download dataset batch-output-dataset

# Get download endpoint and save response
curl -s -X GET "https://api.fireworks.ai/v1/accounts/${ACCOUNT_ID}/datasets/batch-output-dataset:getDownloadEndpoint" \
  -H "Authorization: Bearer ${API_KEY}" \
  -d '{}' > download.json

# Extract and download all files
jq -r '.filenameToSignedUrls | to_entries[] | "\(.key) \(.value)"' download.json | \
while read -r object_path signed_url; do
    fname=$(basename "$object_path")
    echo "Downloading → $fname"
    curl -L -o "$fname" "$signed_url"
done

The output dataset contains two files: a results file (successful responses in JSONL format) and an error file (failed requests with debugging info).

Reference

Job states

Batch jobs progress through several states:

State	Description
VALIDATING	Dataset is being validated for format requirements
PENDING	Job is queued and waiting for resources
RUNNING	Actively processing requests
COMPLETED	All requests successfully processed
FAILED	Unrecoverable error occurred (check status message)
EXPIRED	Exceeded 24-hour limit (completed requests are saved)

Supported models

Base Models – Any model in the Model Library
Custom Models – Your uploaded or fine-tuned models

Note: Newly added models may have a delay before being supported. See Quantization for precision info.

Limits and constraints

Per-request limits: Same as Chat Completion API limits
Input dataset: Max 500MB
Output dataset: Max 8GB (job may expire early if reached)
Job timeout: 24 hours maximum

Handling expired jobs

Jobs expire after 24 hours. Completed rows are billed and saved to the output dataset.Resume processing:

firectl create batch-inference-job \
  --continue-from original-job-id \
  --model accounts/fireworks/models/llama-v3p1-8b-instruct \
  --output-dataset-id new-output-dataset

This processes only unfinished/failed requests from the original job.Download complete lineage:

firectl download dataset output-dataset-id --download-lineage

Downloads all datasets in the continuation chain.

Best practices

Validate thoroughly: Check dataset format before uploading
Descriptive IDs: Use meaningful custom_id values for tracking
Optimize tokens: Set reasonable max_tokens limits
Monitor progress: Track long-running jobs regularly
Cache optimization: Place static content first in prompts

Next Steps

Prompt Caching

Maximize cost savings with automatic prompt caching

Fine-Tuning

Create custom models for your batch workloads

API Reference

Full API documentation for Batch API

Get Started

Deployments

Models & Inference

Fine Tuning

Administration

Security & Compliance

Integrations

Getting Started

Reference

Next Steps

Prompt Caching

Fine-Tuning

API Reference

Get Started

Deployments

Models & Inference

Fine Tuning

Administration

Security & Compliance

Integrations

​Getting Started

​Reference

​Next Steps

Prompt Caching

Fine-Tuning

API Reference

Getting Started

Reference

Next Steps