Model Training Guide 5: DPO Training Guide

Since you have already completed PT (Knowledge Injection) and SFT (Supervised Fine-Tuning), then **DPO (Direct Preference Optimization)** is your final scalpel. Its core function is not to teach the model "to speak," but to teach the model "to choose." In companion AI scenarios, DPO is specifically used to eliminate those "correct but bland" responses, forcing the model to choose the more "human-like" one between two answers.

## DPO Data Preparation

This is the most critical step. DPO training does not require standard answers; it needs **"either-or" comparison groups**.

**Data Format:** Each piece of data contains three parts:

1.  **Prompt:** The user's input.
2.  **Chosen:** The preferred, warm, empathetic, and conversational response.
3.  **Rejected:** The undesirable, mechanical, cold, customer-service-like, and preachy response.

**JSONL Example:**

```JSON
{
  "prompt": "I'm so tired, I feel like I'm just working overtime meaninglessly every day.",
  "chosen": "I'm sending you a hug. That kind of exhaustion where you can't see the end is the most draining. If you can, don't think about anything tonight and get some rest, okay?",
  "rejected": "Overtime is a common phenomenon in the modern workplace. I suggest you plan your time reasonably or communicate your workload with your supervisor. Maintaining a positive attitude helps relieve stress."
}
```

## Operation Steps for Two Approaches

### Type1. LLaMA-Factory

LLaMA-Factory has very mature support for DPO, and it can be run with just a few lines of configuration.

**Workflow:**

1.  **Register Data:** Configure your preference dataset in `dataset_info.json`.
2.  **Write `dpo.yaml`:**

```YAML
stage: dpo # Specify DPO stage
model_name_or_path: ./sft_output # Must use the path to your SFT model
create_new_adapter: true # Stack another LoRA layer on top of SFT
dataset: my_dpo_data
template: qwen
finetuning_type: lora # DPO-specific hyperparameters
pref_beta: 0.1 # Typical Beta value
dpo_ft_loss: sigmoid # Loss function type
learning_rate: 1e-6 # DPO learning rate should be one order of magnitude lower than SFT
num_train_epochs: 2.0
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
```

3.  **Start Command:**

```Bash
llamafactory-cli train dpo.yaml
```

### Type2. Writing Code Yourself

HuggingFace's `TRL` library provides `DPOTrainer`, which is suitable for deep customization.

**Code Core Snippet:**

```Python
from trl import DPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

# 1. Load the SFT model as the policy model
# 2. You also need a reference model, usually a copy of the same model, to calculate the KL divergence to prevent the model from deviating
model = AutoModelForCausalLM.from_pretrained("./sft_model")
ref_model = AutoModelForCausalLM.from_pretrained("./sft_model")

dpo_trainer = DPOTrainer(
    model,
    ref_model=ref_model, # DPO-specific reference model
    beta=0.1, # Adjust
    train_dataset=train_dataset,
    tokenizer=tokenizer,
    args=DPOConfig(
        output_dir="./dpo_model",
        per_device_train_batch_size=1,
        learning_rate=1e-6,
        remove_unused_columns=False # DPO needs to retain prompt/chosen/rejected fields
    ),
)

dpo_trainer.train()
```

## Pitfalls to Watch Out For

1.  **You must SFT before DPO:** If you directly use the raw model for DPO, the model will generate a lot of garbled text because it hasn't learned the dialogue format.
2.  **The Mysticism of the Beta Parameter:** $\beta$ larger means the model is more "obedient," i.e., more inclined towards the Chosen answer; but if it's too large, the model will lose diversity and even repeat itself. 0.1 is a good starting point.
3.  **Memory Pressure:** DPO training usually requires loading two models simultaneously (the currently optimized model and the reference model), and the memory usage is almost **2 times** that of SFT. If you don't have enough memory, be sure to enable `load_in_4bit=True` or use LoRA.
4.  **Preventing Model Degradation:** DPO may cause the model to degrade in logical reasoning ability (e.g., arithmetic). If your companion AI also needs to help users with calculations, remember to mix some general logical alignment data into the DPO dataset.

## Summary

-   **PT:** Solves the "know or not know" problem (business background, professional knowledge).
-   **SFT:** Solves the "understand or not understand the rules" problem (dialogue format, persona positioning).
-   **DPO:** Solves the "good or not good to hear" problem (emotional perception, removing AI flavor).

**With this, the three major axes of model training have been fully introduced.**