- DPO compares the policy against a reference model.
- ORPO combines supervised and preference objectives without a separate reference model.
- Aligning model outputs with brand voice, tone, or style guidelines
- Reducing hallucinations or incorrect reasoning patterns
- Improving response quality where there’s no single “correct” answer
- Teaching models to follow specific formatting or structural preferences
Existing preference jobs
1
Dataset format
Datasets must adhere strictly to the JSONL format, where each line represents a complete JSON-formatted training example.Minimum Requirements:
- Minimum examples needed: 3
- Maximum examples: Up to 3 million examples per dataset
- File format: JSONL (each line is a valid JSON object)
- Dataset Schema: Each training sample must include the following fields:
- An
inputfield containing amessagesarray, where each message is an object with two fields:role: one ofsystem,user, orassistantcontent: a string representing the message content
- A
preferred_outputfield containing an assistant message with an ideal response - A
non_preferred_outputfield containing an assistant message with a suboptimal response
- An
einstein_dpo.jsonl
2
Monitor the preference job
- firectl
- REST API
- Python SDK
JOB_STATE_COMPLETED, and the trained model can be deployed.3
Deploy the DPO trained model
Once training completes, you can create a deployment to interact with the trained model. Refer to deploying a trained model for more details. Deployments scale to zero when idle, and the first request after that returns 503 while the deployment scales back up; see Scaling from zero behavior.
Next Steps
DPO on the Training API
Train with DPO from the cookbook recipe on Serverless or Dedicated Training.
Reinforcement Learning
Optimize models against a reward function on the Training API.
Migrate from Managed Training
Move a managed DPO job to the Training API.