TRL Fine-Tuning

• You want to instruction-tune a base LLM on prompt-completion pairs using SFT
• You have chosen/rejected preference pairs and want to run DPO alignment without building a reward model
• You're building a full RLHF pipeline and need to train a reward model, then run PPO
• You're on a memory-constrained GPU and want online RL via GRPO with a custom reward function

• You only need basic fine-tuning without any RL or preference alignment — the standard HuggingFace Trainer is simpler
• You prefer YAML-driven configuration over Python code — consider Axolotl instead
• You need maximum LoRA training speed as your primary goal — Unsloth may be faster

Guides you through instruction-tuning any causal LM using SFTTrainer, including dataset loading, SFTConfig setup, and saving the trained model. Works with standard prompt-completion and chat-format datasets.

Aligns a model directly from chosen/rejected preference pairs using DPOTrainer — no reward model required. Includes beta tuning guidance, CLI usage, and dataset format examples.

Step-by-step checklist workflow covering SFT, reward model training with RewardTrainer, and PPO policy optimization. Each stage includes working code and configuration parameters.

Memory-efficient online RL using GRPOTrainer with custom reward functions. Covers reward function design, GRPOConfig setup, and CLI training. References a companion deep-dive guide for advanced patterns.

Trains a sequence classification model to score generations using RewardTrainer on preference datasets. Covers loading SFT checkpoints as the base, dataset requirements, and common training pitfalls.

Addresses OOM errors, unstable PPO training, and poor alignment quality with concrete fixes. Covers gradient checkpointing, LoRA/QLoRA integration, batch size tuning, and multi-GPU setup via accelerate.

Start from a raw pretrained model and fine-tune it on instruction-following data using SFT before any alignment step. The skill walks through dataset selection, config, and training loop end-to-end.

Take an instruction-tuned model and align it to human preferences using a chosen/rejected dataset. DPO skips reward model training, making this the fastest path to a more aligned model.

Go from base model to fully aligned model using the classic SFT → Reward Model → PPO sequence. The skill provides checklists, code for each stage, and guidance on connecting the outputs.

Use GRPO with a custom reward function when GPU memory is limited or when you want to define alignment objectives programmatically rather than through preference data.