Model Training Guide Expansion: Introduction to LLaMA-Factory

## LLaMA-Factory: The Most Popular and Easy-to-Use "Full-Process Fine-Tuning Toolbox"

Simply put, **LLaMA-Factory** is currently the most popular and easy-to-use **“full-process fine-tuning toolbox”** in the large model industry. If you compare training a model to cooking, writing code yourself (using the Transformers library) is like starting from forging the pot and growing the vegetables; while LLaMA-Factory is like a **fully automated integrated kitchen**. It encapsulates the complex underlying code, and you only need to throw in the “data”, click and select on the interface, and you can complete everything from **pre-training (PT)** to **instruction fine-tuning (SFT)** to **alignment (DPO)**.

## Core Value (Why are technicians using it?)

1. **Fully Integrated:** Supports almost all mainstream open-source models (Qwen, Llama, Baichuan, Yi, Mistral, Gemma, etc.).
2. **Full Process:** Covers:
    - **PT** (Pre-training)
    - **SFT** (Supervised Fine-tuning)
    - **RLHF/DPO** (Human Preference Alignment)
    - **Evaluation** (Model Evaluation)
    - **Export** (Model Merging and Exporting)
3. **Memory-Friendly:** Integrates various memory-saving technologies such as **LoRA, QLoRA, DeepSpeed, GaLore**, allowing you to run fine-tuning of 7B or even 13B models with a single 24G 4090 GPU.
4. **Visualization:** It provides a web interface called **LLaMA Board** where you can directly adjust parameters and view Loss curves in your browser.

## Training Operation Flow (Taking PT Pre-training as an example)

In LLaMA-Factory, you don't need to write complex Python training scripts. You mainly modify a **YAML configuration file** or operate on the **WebUI**.

### 1. Environment Setup

```Bash
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e .[metrics,bitsandbytes,qwen]
```

### 2. Data Preparation

You need to place your plain text corpus in the `data/` directory and register it in `dataset_info.json`. For **PT (Pre-training)**, the data format is usually very simple:

```JSON
[
  {"file_name": "your_corpus.txt"}
]
```

### 3. Start WebUI (The most intuitive way)

```Bash
llamafactory-cli webui
```

Then open `localhost:7860` in your browser:

- **Training Stage:** Select `Pre-Training`.
- **Model Path:** Enter your base model (e.g., `Qwen/Qwen2.5-7B`).
- **Training Method:** Select `LoRA` or `Full`.
- **Learning Rate:** Set to `1e-5`.
- **Start:** Click “Start Training”, and the driver will automatically run in the background.

### 4. Command Line Startup (Recommended for Production Environment)

Write a `train_pt.yaml` file with the following summarized content:

```YAML
### Core Parameters
stage: pt # Stage is pre-training
model_name_or_path: qwen/Qwen2.5-7B
dataset: your_custom_data # Corresponds to your registered dataset name
cutoff_len: 1024 # Text truncation length

### Training Techniques
finetuning_type: lora # Use LoRA to save memory
lora_target: all # Fine-tune all linear layers

### Hyperparameters
learning_rate: 0.00002
num_train_epochs: 3.0
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
```

Then execute:

```Bash
llamafactory-cli train train_pt.yaml
```

## What problems does LLaMA-Factory solve?

1. **Avoids tedious code:** You don't need to handle `model.backward()`, `optimizer.step()`, or the underlying deadlocks of distributed training (Distributed Data Parallel).
2. **Unifies data format:** It provides a standard. As long as you convert your data to its JSON format, it can adapt to all models.
3. **One-click merging:** It provides an `export_model` function to merge the trained LoRA weights (tens of MB) with the original model into a complete model (several GB) with one click, which can be directly deployed.

## Summary

**LLaMA-Factory is currently the most efficient "model training scaffolding".** Your best approach is:

1. **Use LLaMA-Factory's PT mode** to feed in the raw corpus of your business scenario.
2. **Use LLaMA-Factory's SFT mode** to feed in the companion dialogue data you mentioned earlier.
3. **Finally, deploy it using the GGUF or Safetensors** it exports.