Custom base models

Users may upload custom base models in the Hugging Face model format.

​

Requirements

Fireworks currently supports the following model architectures:

• GPT NeoX
• Gemma
• Phi
• Llama
• Mistral
• Mixtral
• GPTBigCode

The model files you will need to provide depend on the model architecture. In general, you will need the following files:

• Model configuration: config.json.
  Fireworks does not support quantization_config option in config.json.
• Model weights, in one of the following formats:
  • *.safetensors
  • *.bin
• Weights index:*.index.json
• Tokenizer file(s), e.g.
  • tokenizer.model
  • tokenizer.json
  • tokenizer_config.json

If the requisite files are not present, model deployment may fail.

​

Uploading a custom base model

To upload a custom base model, ensure all the required files are in a single directory. Then run:

firectl create model <MODEL_ID> /path/to/files

​

Creating a deployment

Custom base models may only be used with on-demand deployments. Just run:

firectl create deployment <MODEL_ID>

​

Chat completions

To enable the chat completions API for your custom base model, ensure your tokenizer_config.json contains a chat_template field. See the Hugging Face guide on Templates for Chat Models for details.