Hugging Face Transformers 模型微调入门

# Hugging Face Transformers 模型微调入门 ## 环境准备 ```bash pip install transformers datasets accelerate peft bitsandbytes ``` ## 数据加载与预处理 ```python from datasets import load_dataset from transformers import AutoTokenizer dataset = load_dataset("csv", data_files={"train": "train.csv", "test": "test.csv"}) tokenizer = AutoTokenizer.from_pretrained("bert-base-chinese") def preprocess(examples): return tokenizer( examples["text"], truncation=True, padding="max_length", max_length=256 ) tokenized = dataset.map(preprocess, batched=True, remove_columns=["text"]) tokenized = tokenized.rename_column("label", "labels") tokenized.set_format("torch") ``` ## 使用 Trainer API 微调 ```python from transformers import ( AutoModelForSequenceClassification, TrainingArguments, Trainer ) import numpy as np from sklearn.metrics import accuracy_score, f1_score model = AutoModelForSequenceClassification.from_pretrained( "bert-base-chinese", num_labels=5 ) def compute_metrics(eval_pred): logits, labels = eval_pred predictions = np.argmax(logits, axis=-1) return { "accuracy": accuracy_score(labels, predictions), "f1": f1_score(labels, predictions, average="weighted") } training_args = TrainingArguments( output_dir="./results", num_train_epochs=3, per_device_train_batch_size=16, per_device_eval_batch_size=32, learning_rate=2e-5, weight_decay=0.01, eval_strategy="epoch", save_strategy="epoch", load_best_model_at_end=True, metric_for_best_model="f1", fp16=True ) trainer = Trainer( model=model, args=training_args, train_dataset=tokenized["train"], eval_dataset=tokenized["test"], compute_metrics=compute_metrics ) trainer.train() trainer.save_model("./best_model") ``` ## LoRA 高效微调 ```python from peft import LoraConfig, get_peft_model, TaskType lora_config = LoraConfig( task_type=TaskType.SEQ_CLS, r=16, lora_alpha=32, lora_dropout=0.1, target_modules=["query", "value"] ) peft_model = get_peft_model(model, lora_config) peft_model.print_trainable_parameters() # trainable params: 0.6M || all params: 102M || trainable%: 0.59 ``` LoRA 只训练约 0.6% 的参数，显存占用从 8GB 降到 2GB 以内，非常适合在消费级 GPU 上进行领域微调。