Model Training Guide 1: Introduction to Three Training Methods

The Three Core Evolutionary Stages of Large Language Models (LLMs): From General Foundation to Vertical Domain Applications: PT (Incremental Pre-training), SFT (Supervised Fine-tuning), and DPO (Direct Preference Optimization).

- PT (Knowledge Injection): Focuses on absorbing domain knowledge and language distribution, building the model's "knowledge base" and solving the problem of "knowing or not knowing."
- SFT (Behavior Alignment): Through instruction following training and personality shaping, it standardizes the model's interaction logic, solving the problem of "listening to commands or not."
- DPO (Value Alignment): Optimizes emotional decision-making through preference learning, eliminates the "AI flavor" and enhances emotional intelligence, solving the problem of "being likable or not."

## PT (Incremental Pre-training / Continual Pre-training)

**Positioning:** Building the model's knowledge base, solving the problem of "knowing or not knowing."
**Goal:** Learning "new knowledge" or "language distribution" specific to a domain.
**Business Scenarios:**
- **Knowledge Injection:** Feeding in psychology books, anime encyclopedias, and other corpora to create personas like a "senior consultant" or a "two-dimensional girl."
- **Language Habits:** Becoming familiar with industry jargon, dialects, or specific cultural backgrounds (e.g., social media corpora).
**Technical Implementation:**
- **Data:** Massive amounts of raw text, no question-answer pairs required.
- **Loss Function:** `CrossEntropyLoss` in autoregressive tasks.
- **Key Parameters:** Extremely low learning rate (`le 10^{-5}`), to prevent disruption of the original general logic.
**Core Capability Enhancement:**
- **Depth of Understanding:** Reducing misunderstandings of professional terms.
- **Vocabulary Distribution:** Prediction probabilities closer to the speaking habits of the target audience (e.g., in companionship scenarios).

## SFT (Supervised Fine-Tuning)

**Positioning:** Model behavior standardization and personality shaping, solving the problems of "being human-like" and "listening to commands."
**Goal:** Learning "instruction following" and dialogue format.
**Business Scenarios:**
- **Personality Shaping:** Training the model to speak in a specific tone (gentle, humorous, sarcastic).
- **Interaction Alignment:** Transforming robotic "outline listing" responses into natural, conversational multi-turn communication.
- **Function Triggering:** Learning to perform actions in specific contexts (e.g., providing comfort, triggering specific instructions).
**Technical Implementation:**
- **Data:** `Instruction + Input + Output` triplets of question-answer pairs (JSONL format).
- **Main Methods:** **PEFT (LoRA/QLoRA)**, training only a small number of parameters, freezing the foundation, balancing effectiveness and computational resources.
- **Computational Strategy:** Masking Strategy (calculating Loss only on the $Output$ portion).
**Core Capability Enhancement:**
* **Instruction Following:** Strictly executing persona instructions.
- **Dialogue Coherence:** Maintaining consistency in tone and logical rigor in multi-turn dialogues.

## DPO (Direct Preference Optimization)

**Positioning:** Model emotional decision-making and value alignment, solving the problems of "understanding human feelings" and "being likable."
**Goal:** Solving the "good-bad evaluation" problem that SFT cannot eradicate, eliminating stiffness.
**Business Scenarios:**
- **Eliminating "AI Flavor":** Penalizing correct but disappointing responses, encouraging answers with emotional value.
- **Emotional Intelligence Game:** Choosing a more empathetic response between "being reasonable" and "showing empathy," in line with the persona.
- **Value Calibration:** Choosing a position in line with business tone in sensitive and conflicting topics.
**Technical Implementation:**
- **Data:** Pairwise preference pairs `(x, y_w, y_l)`. `y_w` is the better answer, `y_l` is the worse answer.
**Core Formula:**
$$ L_{DPO}(\pi_\theta; \pi_{ref}) = -\mathbb{E}_{(x,y_w,y_l)\sim D}[\log \sigma(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)})] $$
- **Advantages:** No Reward Model required, complexity is much lower than traditional PPO.
**Core Capability Enhancement:**
* **Emotional Perception (Empathy):** Filtering for answers with the highest emotional value.
- **Discrimination:** Internalizing a set of "good-bad standards," making interactions more intuitive.

## Appendix: Technical Stack and Execution Roadmap

| **Stage** | **Core Task** | **Recommended Tools** | **Deliverables** |
| ------------ | ---------------- | ------------------------ | -------------------- |
| **Data Cleaning** | Processing high-quality corpora | Python, Pandas | High-quality JSONL dataset |
| **Framework Selection** | Implementing core training logic | LLaMA-Factory, DeepSpeed | LoRA weights / Full weights |
| **Distributed Training** | Computing resource scheduling and expansion | NCLL, FSDP | Cross-node training capability |
| **Evaluation & Deployment** | Performance testing and online launch | vLLM, GGUF, GPT-4 scoring | External service API |