Model Training Guide Four: SFT Training Guide

After mastering PT (Incremental Pre-training), **SFT (Supervised Fine-tuning)** is the most important step in practice. Its essence is **"instruction alignment"**, which means teaching the model to: when a user asks a question, the model should respond in what format and tone.

## Core Preparation for SFT

### 1. What kind of data to prepare?

SFT requires **QA question-answer pairs** (also known as Instruction data).

**Format:** Usually `JSONL`.

**Content Structure:**

- `instruction`: The user's instruction/question (e.g., "I'm feeling very sad today").
- `input`: Optional background information (e.g., "The user just lost their job").
- `output`: The model's standard answer (e.g., "I'm hugging you, I know how you feel right now...").

**Data Volume:** Companion-style applications usually require **500 - 5,000** high-quality, multi-turn dialogue data that conforms to a specific persona.

### 2. What software and frameworks are needed?

**Hardware:** A GPU with 24GB or more of memory (e.g., 3090/4090/A100).

**System:** Linux (Ubuntu 20.04+) is best, Windows requires WSL2.

**Core Frameworks:**

- `PyTorch`: Deep learning foundation.
- `Transformers`: Developed by HuggingFace, the core library for model loading and training.
- `PEFT`: A library for lightweight fine-tuning such as LoRA.
- `DeepSpeed`: Handles multi-GPU parallelism and memory optimization.

## Operational Steps for Two Schemes

### 1. LLaMA-Factory

This is currently the most recommended **practical solution**, balancing flexibility and ease of use.

1. **Data Registration:** Place your `sft_data.json` in the `data/` folder. Modify `data/dataset_info.json` to add a description of your dataset.
2. **Start Training (Command Line Mode):** Write an `sft.yaml` configuration file:

```YAML
stage: sft # Set to SFT stage
model_name_or_path: qwen/Qwen2.5-7B
preprocessing_num_workers: 8
finetuning_type: lora # Use LoRA to improve efficiency
template: qwen # Key! Must be consistent with the base model
dataset: my_sft_data # Your registered dataset name
output_dir: ./sft_output # Model weight save location
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5e-5 # SFT learning rate is usually higher than PT
num_train_epochs: 3.0
plot_loss: true # Generate Loss curve
```

3. **Run Command:**

```Bash
llamafactory-cli train sft.yaml
```

4. **Model Merging:** The training results in LoRA weights. Execute `llamafactory-cli export` configuration to merge them with the original model.

### 2. Writing Code Yourself

This is the most hardcore approach, based on the `Transformers` library's `SFTTrainer`.

1. **Load Model and Tokenizer:**

```Python
from trl import SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig

model_id = "qwen/Qwen2.5-7B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token # Set padding token
```

2. **Configure LoRA Parameters:**

```Python
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",
    task_type="CAUSAL_LM"
)
```

3. **Define Trainer:**

```Python
args = TrainingArguments(
    output_dir="./my_sft_model",
    per_device_train_batch_size=4,
    max_steps=1000,
    learning_rate=5e-5,
    fp16=True # Enable half-precision training to save memory
)

trainer = SFTTrainer(
    model=model_id,
    args=args,
    train_dataset=my_dataset, # Previously processed Dataset object
    dataset_text_field="text", # Specify the corresponding field in the dataset
    peft_config=peft_config,
    max_seq_length=1024,
)
```

4. **Start Training:**

```Python
trainer.train()
trainer.save_model()
```

## Summary and Suggestions

- **If you pursue efficiency:** Please use **LLaMA-Factory** directly. It helps you handle the problem of dialogue templates (Chat Template). *Note: During SFT training, if the dialogue template (such as `<|im_start|>` tags) is wrong, the model will become dumb.*
- **If you only have CPU or want zero-code:** Choose **Ali Big Model (阿里百炼)**.
- **If you are writing a paper or doing academic research:** Write code yourself (Scheme 3), which allows you to modify the Loss function.