Reinforcement fine-tuning (RFT)

• Domain reasoning: Applying domain-specific logic to solve problems.
• Function calling: Understanding when and how to use external tools based on conversation history and tool descriptions.
• Math with reasoning: Solving math problems with logical steps.
• Code generation/fixes: Modifying or generating code by interpreting context and requirements.

​

1. Design Your Evaluation Strategy

• You want a model that outputs step-by-step solutions.
• Evaluating each reasoning step is hard, but checking the final answer is easy.
• So, if the final answer is correct, you assume the reasoning is likely acceptable.

• Extract the final answer from the output.
• Compare it to the known ground truth.
• If they match → score = 1.0. If not → score = 0.0.

​

2. Prepare your dataset

​

For testing reward functions in the UI

{"messages": [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "What is the capital of France?"},
  {"role": "assistant", "content": "Paris."}
]}

​

For actual RFT training

{"messages": [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "What is the capital of France?"}
]}

1. Use your prompts to generate multiple assistant responses
2. Pass each generated response to your reward function for evaluation
3. Use the reward scores to train the model

​

Output dataset example with optional metadata

{
  "messages": [
    {"role": "system", "content": "Return your solution on the last line, formatted as 'Answer: the answer'"},
    {"role": "user", "content": "What is the result of 3+2?"},
    {"role": "assistant", "content": "2 + 2 = 1 + 1 + 1 + 1 + 1 = 4, I think the answer is 4.\nAnswer: 4"}
  ],
  "ground_truth": 5
}

​

3. Build and iterate on the evaluator

Nagivating to the Evaluations tab in your fireworks dashboard, and click Create Evaluator, you should see the following page

​

Example evaluator (math task)

import numpy as np
import pydantic
import reward_kit
from reward_kit import reward_function
from reward_kit.models import EvaluateResult, MetricResult # https://docs.fireworks.ai/evaluators/developer_guide/core_data_types

@reward_function
def evaluate(messages: list[dict], ground_truth: int, **kwargs) -> EvaluateResult:
    """
    The reward function to evaluates a single entry from the dataset. This function is required in the `main.py` file.
    For more details and examples, please refer to https://docs.fireworks.ai/evaluators/examples/examples_overview
    Args:
        messages: A list of dictionaries representing a single line from the dataset jsonl file. This maps to the `messages` field in the dataset json entry.
        ground_truth: The particular dataset has a ground_truth column, which is an integer for the math question answer
        kwargs: Additional fields in your dataset besides `messages`. Highly recommended to not remove this due to potential more keywords being passed to the function.
    Returns:
        EvaluateResult: Evaluate result that should include score, and optionally is_score_valid, reason and sub metrics.
    """
    answer = messages[-1]['content']
    last_line = answer.split('\n')[-1]
    mg = re.match(r"Answer: (\d+)", last_line)
    score = 0.0
    if mg is not None:
        extracted_answer = int(mg.group(1))
        score = 1.0 if extracted_answer == ground_truth else 0.0
    return EvaluateResult(
        score=score,
        is_score_valid=True,
        reason=f"extracted_answer {extracted_answer}, actual answer {ground_truth}"
    )

​

Evaluator function requirements

• Inputs: The function is called for each dataset row. It receives the messages and any custom fields like ground_truth coming from your dataset.
• Output: A dictionary with:
  • score: Float between 0.0 and 1.0
  • reason: (Optional) A string for logging
  • is_score_valid: (Optional, defaults to True) Flag to skip training on invalid outputs

return EvaluateResult(
        score=0.5 * metric_1_score + 0.5 * metric_2_score, 
        reason="This is the eval result for the rollup",
        metrics={
            "metric_1": MetricResult(
                is_score_valid=True,
                score=metric_1_score,
                reason="This is the eval result for metric_1"
            ),
            "metric_2": MetricResult(
                is_score_valid=True,
                score=metric_2_score,
                reason="This is the eval result for metric_2"
            )
        }
    )

​

4. Create an RFT job

1. Go to the “Fine-Tuning” tab.
2. Click “Fine-tune a Model”.
3. Select “Reinforcement” as the tuning method.
4. Follow the wizard to complete the setup.

​

5. Monitor training progress

• Training progress
• Evaluation metrics
• Model checkpoints

​

6. Deploy and use the model

​

Access

​

Additional RFT job settings

​

Training-time hyperparameters

Copy

Ask AI

firectl create rftj \
  --base-model llama-v3p1-8b-instruct \
  --dataset my-dataset \
  --output-model my-rft-model \
  --epochs 3 \
  --learning-rate 1e-4 \
  --lora-rank 16 \
  --max-context-length 16384

​

Roll-out (sampling) parameters

Copy

Ask AI

firectl create rftj \
  ... \
  --inference-max-tokens 1024 \
  --inference-temperature 0.8 \
  --inference-top-p 0.9 \
  --inference-top-k 40 \
  --inference-n 6 \
  --inference-extra-body '{"stop":["\n\n"]}'

​

Practical tips

1. Keep temperature > 0 – greedy sampling (temperature 0) is deterministic and collapses exploration, often leading to mode-dropping and repetitive text. (reddit.com)
2. Use at least two choices – multi-sample roll-outs are standard in Policy-Optimization and rejection-sampling pipelines. (rlhfbook.com, blog.ml.cmu.edu)
3. Log everything – Fireworks dashboards export Weights & Biases runs, so tag each sweep and compare reward curves side-by-side.
4. Watch VRAM – bigger n, max_tokens, or batch size scale memory roughly linearly. Scale horizontally or enable Turbo mode if needed.
5. Iterate in small steps – change one hyperparameter at a time; RLHF is sensitive and unstable grids waste compute. (arxiv.org)

​

Why these limits?

​

Quick reference