Skip to main content
Speculative decoding reduces generation latency by proposing multiple tokens and letting the target model verify them in parallel. The target model still verifies every accepted token; the drafter does not replace the target model. The benefit depends on both the cost of producing draft tokens and how often the target model accepts them. A poorly matched drafter can make generation slower, so benchmark with representative traffic before overriding Fireworks defaults.
The deployment flags on this page apply to dedicated deployments. Fireworks manages the serving configuration for Serverless models.

​
Start with the default

For most supported models, a default drafter and draft-token count are already configured. A new deployment inherits those settings, so you usually do not need to pass any speculative-decoding flags.Create the deployment normally, then benchmark it before changing the drafter:
If the base model does not define a default drafter, the deployment runs without model-based speculative decoding. To explicitly disable an inherited default when creating a comparison deployment, use:

​
Choose a method

Predicted Outputs can be used in addition to a deployment’s model-based speculative decoding.

​
Configuration options

--draft-model and --ngram-speculation-length are alternative deployment strategies and cannot be used together.

​
Custom draft models

For self-service configuration, use a small base model that is compatible with the target model. In practice, this means using the same model family and tokenizer. A model that is merely smaller is not necessarily a useful drafter; its acceptance rate and execution cost both matter.

​
Fallback draft models

If the target model has no default drafter, the following small base models are reasonable starting points for an experiment. A purpose-built drafter generally performs better. Fireworks also supports compatible EAGLE, DFlash, DSpark, and Medusa draft addons. These formats are architecture-specific and require a checkpoint and configuration prepared for the exact target model; they are not drop-in replacements for a small base-model drafter. Contact Fireworks to validate an existing checkpoint or discuss a drafter adapted to your traffic.

​
Examples

Create a deployment with an explicit small base-model drafter:
You can change the explicit drafter and draft-token count on an existing deployment:

​
Benchmark and tune

Compare at least three configurations on the same target model and deployment shape:
  1. The inherited Fireworks default.
  2. Your candidate drafter or N-gram settings.
  3. A deployment created with --disable-speculative-decoding.
Use production-like prompts, output lengths, sampling parameters, and concurrency. Measure time to first token, inter-token latency, p50/p95 request latency, and maximum sustainable throughput. A high acceptance rate alone does not guarantee a speedup because the drafter also consumes compute. To inspect per-request metrics, set perf_metrics_in_response to true in the completion request. For dedicated deployments, the final response or final streaming chunk includes:
  • speculation-generated-tokens: number of tokens generated through speculation
  • speculation-acceptance: acceptance rate by proposed-token position
Acceptance normally falls at later positions. Increase --draft-token-count only while the additional accepted tokens outweigh the extra drafting and verification work. Re-run the benchmark when the traffic mix, prompt format, model, quantization, or deployment shape changes.