The Supervised Fine-Tuning Pipeline

Topics We'll Cover

• Why Fine-Tune — Capability vs Style; Alternatives
• Datasets — Instruction, Conversation, Format Converters
• Tokenisation & Chat Templates (ChatML, Llama-3, Qwen)
• Loss Masking — Train Only on Assistant Tokens
• Sample Packing — More Tokens Per Step
• Frameworks — Axolotl, TRL, Unsloth, torchtune
• Evaluating SFT Runs — Perplexity, Eval Suites
• Hardware Reality — VRAM Math, Throughput, Multi-GPU
• What to Take Away

Why Fine-Tune — Capability vs Style; Alternatives

Supervised fine-tuning (SFT) updates a pre-trained base model's weights on a labelled dataset of (input, desired-output) pairs. It is the first training stage that turns a next-token predictor into something that behaves like an assistant. Understanding when to fine-tune versus reaching for a cheaper tool is the first decision in any LLM deployment.

Datasets — Instruction, Conversation, Format Converters

Dataset quality dominates SFT outcomes far more than model size or training duration. The field has converged on two structural formats.

Format Converters

Most frameworks accept multiple formats and convert internally. Axolotl's datasets: block takes a type: key (alpaca, sharegpt, completion, context_qa, …) and handles the mapping. TRL's DataCollatorForCompletionOnlyLM works on the tokenised stream after template application.

Tokenisation & Chat Templates (ChatML, Llama-3, Qwen)

A chat template is the byte-level contract that maps structured conversation turns into a flat token sequence the model was trained on. Using the wrong template is silent — training completes, but quality collapses at inference because the model sees token patterns it was never optimised against.

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is gradient checkpointing?<|im_end|>
<|im_start|>assistant
Gradient checkpointing trades compute for memory ...<|im_end|>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful assistant.<|eot_id|>
<|start_header_id|>user<|end_header_id|>
What is gradient checkpointing?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
Gradient checkpointing trades compute for memory ...<|eot_id|>

<|im_start|>system\n
You are Qwen, created by Alibaba Cloud.<|im_end|>\n
<|im_start|>user\n
What is gradient checkpointing?<|im_end|>\n
<|im_start|>assistant\n
Gradient checkpointing trades compute for memory ...<|im_end|>\n

Loss Masking — Train Only on Assistant Tokens

In a standard causal-LM training loop the cross-entropy loss is computed over every token in the sequence. For SFT you almost always want to mask out the prompt tokens so that the model only learns to predict assistant responses. Computing loss on human turns is wasteful at best and actively harmful (the model learns to generate user-side content) at worst.

Implementation in TRL

from trl import SFTTrainer, DataCollatorForCompletionOnlyLM

# ChatML assistant header tokens
response_template = "<|im_start|>assistant\n"
collator = DataCollatorForCompletionOnlyLM(
    response_template=response_template,
    tokenizer=tokenizer,
    mlm=False,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    data_collator=collator,   # ← this is all you need
)

Sample Packing — More Tokens Per Step

Naively, each batch entry is one conversation padded to the maximum sequence length. On typical instruction datasets where most examples are 256–512 tokens, padding a 2 048-token context wastes 75–87 % of the compute budget on <pad> tokens that produce zero gradient signal.

Sample packing (also called bin packing) concatenates multiple short samples end-to-end within a single sequence until the context length is filled, with attention masking to prevent cross-contamination between packed items.

Attention Masking for Packing

Without correct masking, token N in example 2 attends to tokens from example 1, causing cross-contamination. The fix is a block-diagonal attention mask: each packed sub-sequence attends only to itself. Flash Attention 2 (flash_attn≥2.1) supports variable-length sequences natively via cu_seqlens — this is the most compute-efficient path.

Frameworks — Axolotl, TRL, Unsloth, torchtune

Four frameworks dominate open-source SFT. Each makes different tradeoffs between configurability, speed, and ease of use.

Evaluating SFT Runs — Perplexity, Eval Suites

Training loss falling does not mean the model is getting better at the task you care about. A comprehensive evaluation strategy triangulates across three axes: intrinsic metrics, downstream benchmarks, and human / model preference ratings.

Hardware Reality — VRAM Math, Throughput, Multi-GPU

VRAM is the hard constraint. Before writing a single config line, compute the memory budget:

# Per-parameter costs for bf16 + full-precision optimiser state (AdamW)
# 2 bytes (bf16 param) + 4 bytes (fp32 copy) + 8 bytes (Adam m + v) = 14 bytes/param
# Activations = batch_size * seq_len * hidden_dim * num_layers * ~2 bytes

params_7b = 7e9
base_vram  = params_7b * 14   # ≈ 98 GB — needs 2×A100 80 GB minimum

# With gradient checkpointing (store only layer boundaries):
# activation VRAM drops ~8x, adds ~30% compute overhead

# QLoRA (4-bit base + bf16 LoRA adapters):
qlora_vram = params_7b * 0.5 + 0.01 * params_7b * 14  # ≈ 4.5 GB + 1 GB adapters

What to Take Away

• Fine-tune for capability gaps, not for what prompting can already do. RAG, few-shot, and prompt caching are cheaper first steps.
• Dataset quality beats dataset size. Deduplicate, length-filter, and PII-scrub before training. 10 k clean examples often outperforms 500 k noisy ones.
• Use the tokeniser's own chat template. tokenizer.apply_chat_template() from transformers≥4.34. Never hand-format; template bugs are silent and devastating.
• Mask prompt tokens. Set train_on_inputs: false (Axolotl) or use DataCollatorForCompletionOnlyLM (TRL). Loss on user turns wastes compute and teaches the wrong distribution.
• Pack samples when your average example length is <50 % of the context window. Expect 2–4× throughput improvement.
• Know the VRAM math before picking a strategy. QLoRA puts 7B fine-tuning on a single consumer GPU; full SFT of 7B needs ≥2×A100 40 GB.
• Evaluate beyond training loss. Run MT-Bench, IFEval, and MMLU before and after; catch catastrophic forgetting early.