Using function-calling

The function calling API allows a user to describe the set of tools/functions available to the model and have the model intelligently choose the right set of function calls to invoke given the context. This functionality allows users to build dynamic agents that can get access to real-time information & produce structured outputs. The function calling API doesn’t invoke any function calls. Instead, it generates the tool calls to make in OpenAI-compatible format.

At a high level, function calling works by

1. The user specifies a query along with the list of available tools for the model. The tools are specified using JSON Schema.
2. The model intelligently detects intent and based on intent the model outputs either a normal conversation reply or a list of tools/functions to invoke for the user. Based on the specified schema, the model populates the correct set of arguments to invoke a function call.
3. The user receives a reply from the model. If the reply contains a function call specification - the user can execute that method and return its output to the model along with further queries/conversation.

​

Resources

1. Fireworks Blog Post on FireFunction-v2
2. Open AI Docs on Function Calling
3. Open AI Cookbook on Function Calling
4. Function Calling Best Practices

​

Supported models

• Firefunction-v2 - Latest and most performant model
• Firefunction-v1 - Previous generation, Mixtral-based function calling model. Fast and excels at routing and structured output

​

Example usage

TL;DR This example tutorial is available as a Python notebook [code | Colab].

For this example, let’s consider a user looking for Nike’s financial data. We will provide the model with a tool that the model is allowed to invoke & get access to the financial information of any company.

1. To achieve our goal, we will provide the model with information about the get_financial_data function. We detail its purpose, arguments, etc in JSON Schema. We send this information in through the tools argument. We sent the user query as usual through the messages argument.

import openai
import json

client = openai.OpenAI(
    base_url = "https://api.fireworks.ai/inference/v1",
    api_key = "<YOUR_FIREWORKS_API_KEY>"
)

messages = [
    {"role": "system", "content": f"You are a helpful assistant with access to functions." 
     															"Use them if required."},
    {"role": "user", "content": "What are Nike's net income in 2022?"}
]

tools = [
    {
        "type": "function",
        "function": {
            # name of the function 
            "name": "get_financial_data",
            # a good, detailed description for what the function is supposed to do
            "description": "Get financial data for a company given the metric and year.",
            # a well defined json schema: https://json-schema.org/learn/getting-started-step-by-step#define
            "parameters": {
                # for OpenAI compatibility, we always declare a top level object for the parameters of the function
                "type": "object",
                # the properties for the object would be any arguments you want to provide to the function
                "properties": {
                    "metric": {
                        # JSON Schema supports string, number, integer, object, array, boolean and null
                        # for more information, please check out https://json-schema.org/understanding-json-schema/reference/type
                        "type": "string",
                        # You can restrict the space of possible values in an JSON Schema
                        # you can check out https://json-schema.org/understanding-json-schema/reference/enum for more examples on how enum works
                        "enum": ["net_income", "revenue", "ebdita"],
                    },
                    "financial_year": {
                        "type": "integer", 
                        # If the model does not understand how it is supposed to fill the field, a good description goes a long way 
                        "description": "Year for which we want to get financial data."
                    },
                    "company": {
                        "type": "string",
                        "description": "Name of the company for which we want to get financial data."
                    }
                },
                # You can specify which of the properties from above are required
                # for more info on `required` field, please check https://json-schema.org/understanding-json-schema/reference/object#required
                "required": ["metric", "financial_year", "company"],
            },
        },
    }
]

chat_completion = client.chat.completions.create(
    model="accounts/fireworks/models/firefunction-v2",
    messages=messages,
    tools=tools,
    temperature=0.1
)
print(chat_completion.choices[0].message.model_dump_json(indent=4))

{
  "content": "",
  "role": "assistant",
  "function_call": null,
  "tool_calls": [
    {
      "id": "call_XstygHYlzKrI8hbERr0ybeOQ",
      "function": {
        "arguments": "{\"metric\": \"net_income\", \"financial_year\": 2022, \"company\": \"Nike\"}",
        "name": "get_financial_data"
      },
      "type": "function",
      "index": 0
    }
  ]
}

2. In our case, the model decides to invoke the tool get_financial_data with some specific set of arguments. Note The model itself won’t invoke the tool. It just specifies the argument. When the model issues a function call - the completion reason would be set to tool_calls. The API caller is responsible for parsing the function name and arguments supplied by the model & invoking the appropriate tool.

def get_financial_data(metric: str, financial_year: int, company: str):
    print(f"{metric=} {financial_year=} {company=}")
    if metric == "net_income" and financial_year == 2022 and company == "Nike":
        return {"net_income": 6_046_000_000}
    else:
        raise NotImplementedError()

function_call = chat_completion.choices[0].message.tool_calls[0].function
tool_response = locals()[function_call.name](**json.loads(function_call.arguments))
print(tool_response)

metric='net_income' financial_year=2022 company='Nike'
{'net_income': 6046000000}

3. The API caller obtains the response from the tool invocation & passes its response back to the model for generating a response.

agent_response = chat_completion.choices[0].message

# Append the response from the agent
messages.append(
    {
        "role": agent_response.role, 
        "content": "",
        "tool_calls": [
            tool_call.model_dump()
            for tool_call in chat_completion.choices[0].message.tool_calls
        ]
    }
)

# Append the response from the tool 
messages.append(
    {
        "role": "tool",
        "content": json.dumps(tool_response)
    }
)

next_chat_completion = client.chat.completions.create(
    model="accounts/fireworks/models/firefunction-v2",
    messages=messages,
    tools=tools,
    temperature=0.1
)

print(next_chat_completion.choices[0].message.content)

{
  "content": "Nike's net income for the year 2022 was $6,046,000,000.",
  "role": "assistant",
  "function_call": null,
  "tool_calls": null
}

This results in the following response

Nike's net income for the year 2022 was $6,046,000,000.

​

Tools specification

tools field is an array, and each individual component contains the following two fields

• type: string The type of the tool. Currently, only function is supported.
• function: object
  • description: string A description of what the function does, used by the model to choose when and how to call the function.
  • name: string The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
  • parameters: object The parameters the functions accepts, described as a JSON Schema object. See the JSON Schema reference for documentation about the format.

​

Tool choice

The tool_choice parameter controls whether the model is allowed to call functions or not. Currently, we support values auto, none , any or a specific function name.

• auto mode implies the model can pick between generating a message or calling a function. This is the default tool choice when the field is not specified.
• none mode is akin to no tool specification being passed to the model.
• To force a specific function call, you can set thetool_choice = {"type": "function", "function": {"name":"get_financial_data"}}. In this scenario, the model is forced to use this specific function.
• To specify that any function call is a good function call, and a function call should always be made, you can use the any mode, or you can set the function name to be empty when you specify tool choice to be “function”, like tool_choice = {"type": "function"}

​

OpenAI compatibility

Our function calling API is fully compatible with OpenAI including in streaming scenarios. Compared to the OpenAI API - we don’t support parallel and nested function calling.

​

Best practices

1. Number of Functions - The length of the list of functions specified to the model, directly impacts its performance. For best performance, keep the list of functions below 7. It’s possible to see some degradation in the quality of the model as the tool list length exceeds 10.
2. Function Description - The function specification follows JSON Schema. For best performance, describe in great detail what the function does under the “description” section. An example of a good function description is “Get financial data for a company given the metric and year”. A bad example would be “Get financial data for a company”.
3. System Prompt - In order to ensure optimal performance, we recommend not adding any additional system prompt. User-specified system prompts can interfere with the function detection & calling ability of the model. The auto-injected prompt for our function calling model is designed to ensure optimal performance.
4. Temperature Setting temperature to 0.0 or some low value. This helps the model to only generate confident predictions and avoid hallucinating parameter values.
5. Function descriptions Providing verbose descriptions for functions & its parameters. This is similar to prompt engineering: the more elaborate & accurate the function definition/documentation - the better the model is at deciphering the accurate intent of the function and its parameters.

​

Function calling vs JSON mode

When to use function calling vs JSON mode? Use function calling if the use case involves decision-making or interactivity - which function to call, what parameters to fill, or how to follow up with the user to fill in missing function arguments in a chat form. JSON/Grammar mode is a preferred choice for non-interactive structured data extraction and allows you to explore non-JSON formats too.

​

Example apps

• Fireworks-created demos:
  • UI demo for image generation and stock price retrieval
  • Notebook for information extraction
• Langchain integration notebooks:
  • Function Calling with LangChain JS
  • AgentExecutor notebook
  • RAG + Langchain

​

Data policy

Data from Firefunction is logged and automatically deleted after 30 days to ensure product quality and to prevent abuse ( bulk data on average # functions used, etc). This data will never be used to train models. Please contact raythai@fireworks.ai if you have questions, comments, or use cases where data cannot be logged.