Skip to main content
Managed SFT and DPO no longer accept new jobs. Your trained models stay deployable. To keep training, follow Migrate from Managed Training.
Preference optimization trains models on pairs of preferred and non-preferred responses to the same prompt. Managed jobs used one of two objectives:
  • DPO compares the policy against a reference model.
  • ORPO combines supervised and preference objectives without a separate reference model.
Use either method for:
  • Aligning model outputs with brand voice, tone, or style guidelines
  • Reducing hallucinations or incorrect reasoning patterns
  • Improving response quality where there’s no single “correct” answer
  • Teaching models to follow specific formatting or structural preferences

Existing preference jobs

1

Dataset format

Datasets must adhere strictly to the JSONL format, where each line represents a complete JSON-formatted training example.Minimum Requirements:
  • Minimum examples needed: 3
  • Maximum examples: Up to 3 million examples per dataset
  • File format: JSONL (each line is a valid JSON object)
  • Dataset Schema: Each training sample must include the following fields:
    • An input field containing a messages array, where each message is an object with two fields:
      • role: one of system, user, or assistant
      • content: a string representing the message content
    • A preferred_output field containing an assistant message with an ideal response
    • A non_preferred_output field containing an assistant message with a suboptimal response
Here’s an example conversation dataset (one training example):
einstein_dpo.jsonl
We currently only support one-turn conversations for each example, where the preferred and non-preferred messages need to be the last assistant message.
2

Monitor the preference job

Once the job is complete, the state is JOB_STATE_COMPLETED, and the trained model can be deployed.
3

Deploy the DPO trained model

Once training completes, you can create a deployment to interact with the trained model. Refer to deploying a trained model for more details. Deployments scale to zero when idle, and the first request after that returns 503 while the deployment scales back up; see Scaling from zero behavior.

Next Steps

DPO on the Training API

Train with DPO from the cookbook recipe on Serverless or Dedicated Training.

Reinforcement Learning

Optimize models against a reward function on the Training API.

Migrate from Managed Training

Move a managed DPO job to the Training API.