Model Training Guide II: A Comparison of Three Practical Training Approaches

This article delves into the three crucial steps in LLM training – PT (Incremental Pre-training), SFT (Supervised Fine-tuning), and DPO (Preference Alignment) – from the perspective of practical implementation.

## 1. PT

PT (Incremental Pre-training): Think of it as "feeding the model books." Your focus is on **"throughput"** and **"data cleaning."**

**The data you have:** A collection of massive `.txt` or `.jsonl` files containing long passages of text, without any distinction between "questions" and "answers."

**Your core actions:**

- **Chunking:** You need to split tens of GB of text into blocks of 2048 or 4096 tokens each.
- **Lower the learning rate? No, reduce it:** You'll set the learning rate (LR) very low (e.g., `1e-5`) because you only want it to learn new words, without disrupting its existing logic.

**The metrics you monitor:** **Loss (loss value).** As long as the Loss is decreasing smoothly, it means the model is "memorizing the book."

**The sign of completion:** You notice the model starts to recognize and output the professional terms you feed it.

## 2. SFT

SFT (Supervised Fine-tuning): Think of it as "rehearsing a script" with the model. Your focus is on **"dialogue templates"** and **"Masking."**

**The data you have:** Strict question-answer pairs.

- `"instruction": "What should I do after a breakup?"`
- `"output": "I'm here for you. Crying it out can help..."`

**Your core actions:**

- **Select a template:** This is where things can easily go wrong. You must specify the template corresponding to the model in the script (e.g., `qwen` or `llama3`), otherwise the model will learn incorrectly.
- **Set Masking:** In your operation script, you need to ensure the model **only calculates Loss on the Output.** If you don't set this up correctly, the model will memorize the user's questions, leading it to simply repeat them back to you.

**The metrics you monitor:** **Validation set accuracy** and **Loss.** If the Loss drops too quickly (e.g., to below 0.1 instantly), it indicates overfitting – it's simply memorizing the script.

**The sign of completion:** When you input test questions, it can chat with you in the tone you've specified (e.g., gentle, humorous).

## 3. DPO

DPO (Preference Alignment): Think of it as "grading the model's answers." Your focus is on **"memory management"** and the **"Beta parameter."**

**The data you have:** Triplets. Each question is followed by a **"good answer"** and a **"bad answer."**

**Your core actions:**

- **Load dual models:** Your memory will be instantly strained. You need to load both the "model being trained" and the "reference model" simultaneously.
- **Adjust Beta value:** This is the only somewhat mysterious parameter in DPO. You modify `pref_beta` in the script. If the model starts speaking in extremes or repeating itself after training, you need to come back and adjust this value (usually between 0.1 and 0.5).

**The metrics you monitor:** **Rewards/margins.** You'll see two lines, one for Chosen and one for Rejected. If the distance between the two lines widens, it means the model has learned to distinguish between "good" and "bad."

**The sign of completion:** The previously "cold and robotic" responses disappear, and the model becomes more attuned to your preferences.

## Summary of Operational Differences

| **Operation Item** | **PT (Knowledge Ingestion)** | **SFT (Rule Learning)** | **DPO (Good vs. Bad)** |
| ------------------- | ----------------------- | ------------------------------ | -------------------------- |
| **Your Biggest Challenge** | Cleaning and removing garbage text | Writing and proofreading high-quality dialogue scripts | Finding incorrect model answers to create comparison pairs |
| **Script Core Switch** | `stage: pt` | `stage: sft` + `template: xxx` | `stage: dpo` + `beta: 0.1` |
| **Memory Requirements** | Stable (single GPU memory) | Stable (single GPU memory) | **Huge (double GPU memory)** |
| **Training Epochs** | Usually only run 1 epoch (to prevent repetition) | Run 3-5 epochs (learn deeply) | Run 1-2 epochs (to prevent distortion) |