Model Training Guide 3: PT Training Guide

The essence of **Continual Pre-training (PT)** is: building upon a large model that already possesses general capabilities, feeding it a massive amount of industry documents, books, or code (Raw Text), and enabling it to absorb new knowledge through the **Next Token Prediction** task.

As a technical professional, you only need to remember: PT aims to enable the model to learn the **language distribution, specialized terminology, and logical context** of a specific domain on **pure text in a non-question-answering format**.

## PT Data Preparation (The Fundamental Basis)

- **Data Format**: Pure text (Raw Text). No `instruction` or `output` labels are required.
- **File Format**: Typically `.txt` or `.jsonl` (each line is a `{"text": "..."}`).
- **Processing Logic**: The model learns this content by "predicting the next Token."

## Two Practical Implementation Steps

### 1. LLaMA-Factory

**Applicable Scenario**: GPU environment available, pursuing maximum efficiency.

**Steps**:

1. **Data Registration**: Place the text in `data/`, and add it to `data/dataset_info.json`:

```JSON
"my_corpus": {
  "file_name": "your_data.jsonl",
  "columns": {
    "prompt": "text"
  }
}
```

2. **Write `pt_config.yaml`**:

```YAML
stage: pt # Key: Specify as the pre-training stage
model_name_or_path: qwen/Qwen2.5-7B
dataset: my_corpus
finetuning_type: lora # Or full (full-parameter training requires extremely high memory)
lora_target: all # Target all linear layers, PT recommends full coverage
cutoff_len: 1024 # Context truncation length
learning_rate: 1e-5 # PT learning rate must be small
num_train_epochs: 1.0
output_dir: ./pt_output
```

3. **Execute Command**:

```Bash
llamafactory-cli train pt_config.yaml
```

### 2. Writing Code Yourself

Based on the Transformers Library

**Applicable Scenario**: Deep customization of Loss or Tokenizer.

**Steps**:

1. **Load Model**: Use `AutoModelForCausalLM`.
2. **Core Code Logic**:

```Python
from transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling

# 1. Prepare data collator (for PT)
# mlm=False indicates causal language modeling task (Next Token Prediction)
data_collator = DataCollatorForLanguageModeling(tokenizer, mlm=False)

# 2. Training configuration
training_args = TrainingArguments(
    output_dir="./output",
    per_device_train_batch_size=4,
    learning_rate=1e-5,
    weight_decay=0.01,
    num_train_epochs=1,
    save_steps=500
)

# 3. Start training
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    data_collator=data_collator,
)
trainer.train()
```

## PT Training Technical Details & Common Pitfalls

1. **Learning Rate**: The learning rate for PT must be one order of magnitude **lower** than that of SFT (typically $10^{-5}$ or $10^{-6}$). Because the base model is already strong, a large learning rate will destroy its original logical capabilities.
2. **Data Quality**: PT is extremely sensitive to data quality. If your corpus contains a large amount of duplicate, garbled, or low-quality web data, the model will quickly become "less intelligent."
3. **Regarding Ollama**: To reiterate, **Ollama is used to "run" models, not to "train" them**. The weights you train using method 2 or 3 above can be used by Ollama after being exported in GGUF format.

**Summary Recommendation**: Start by running PT with **LLaMA-Factory**, as it integrates data slicing, model loading, and LoRA configuration, making it the least error-prone method.