LoRA and PEFT Variants

Topics We'll Cover

• The LoRA Idea — Low-Rank Deltas, Frozen Base
• LoRA Math — A·B with Rank r, Scaling α
• Target Modules — q/k/v/o_proj & MLPs
• Hyperparameters — Rank 8 vs 64 vs 128, α, Dropout
• QLoRA — 4-bit Base, NF4, Double Quant
• DoRA — Magnitude-Direction Decomposition
• IA3 — Even Smaller Perturbations
• Merging Adapters & Multi-LoRA Serving
• What to Take Away

The LoRA Idea — Low-Rank Deltas, Frozen Base

Hu et al. (2021) — LoRA: Low-Rank Adaptation of Large Language Models — started from an empirical observation: when a large model is fine-tuned on a downstream task, the weight update matrix ΔW has intrinsically low rank. The pre-training process has already found a high-dimensional representation; the task-specific delta lives in a much lower-dimensional subspace.

If that's true, instead of adding a full-rank ΔW ∈ R^d_out×d_in (which would be 4–16 GB of additional parameters for a 7B model), you can represent it as a product of two thin matrices whose combined parameter count is tiny.

LoRA Math — A·B with Rank r, Scaling α

The forward pass through a LoRA-patched linear layer is:

Key initialisations: A is initialised with Kaiming uniform so the scale of the random projection is sensible; B is initialised to zeros so that at the start of training the LoRA delta contributes nothing — the model begins training identical to the frozen base, which stabilises early loss curves.

The α / r scaling convention

The scaling factor α/r normalises the magnitude of the LoRA update. If you double the rank without changing α, the effective update doubles in magnitude — the learning rate and α are entangled. Common practice: set α = r (scale = 1, no amplification) or α = 2r (doubles the effective update). The HF PEFT library defaults to alpha=16, rank=8 giving a 2× amplification.

Target Modules — q/k/v/o_proj & MLPs

LoRA can be applied to any linear layer. In practice the choice of which layers to target is the most impactful design decision after rank.

Module Names by Architecture

Hyperparameters — Rank 8 vs 64 vs 128, α, Dropout

LoRA has fewer hyperparameters than full fine-tuning, but the ones it has are sensitive. Here is what the research and community practice have converged on.

from peft import LoraConfig, TaskType

config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=16,           # scale = 1.0
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",              # "all" or "lora_only" are rarely better
    use_rslora=True,           # alpha/sqrt(r) normalisation (PEFT ≥0.6)
)

QLoRA — 4-bit Base, NF4, Double Quant

Dettmers et al. (2023) — QLoRA: Efficient Finetuning of Quantized LLMs — combined LoRA with 4-bit weight quantisation. The three technical innovations are tightly coupled and each is necessary for the approach to work without accuracy collapse.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,  # dequantise to bf16 for compute
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_cfg,
    device_map="auto",
)
# Then wrap with PEFT LoRA as normal; gradients flow through bf16 adapters

DoRA — Magnitude-Direction Decomposition

Liu et al. (2024) — DoRA: Weight-Decomposed Low-Rank Adaptation — argued that LoRA fails to match full fine-tuning because it conflates two independent properties of each weight matrix: its magnitude (the column-wise L2 norm) and its direction (the normalised column vectors).

DoRA decomposes the pretrained weight as W₀ = m · V / ||V|| where m ∈ R^1×k is the magnitude vector (one scalar per column) and V ∈ R^d×k is the direction. Only the direction update is low-rank (LoRA); the magnitude is a separate trainable scalar per column.

IA3 — Even Smaller Perturbations

Liu et al. (2022) — Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning — introduced IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations). Instead of adding a rank-r matrix delta, IA³ multiplies key, value, and FFN activation streams by learned scalar vectors.

from peft import IA3Config, TaskType, get_peft_model

config = IA3Config(
    task_type=TaskType.CAUSAL_LM,
    target_modules=["k_proj", "v_proj", "down_proj"],
    feedforward_modules=["down_proj"],  # which modules are feedforward
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 401,408 || all params: 8,030,000,000 || 0.005 %

Merging Adapters & Multi-LoRA Serving

Once training is complete there are two distinct deployment patterns: merge-and-serve (fold the adapter into the base weights permanently) or hot-swap serving (keep the base frozen and load adapters per request at inference time).

Merge (Linear Combination)

The merged weight is simply Wmerged = W0 + (α/r) · B · A. This produces a single model identical at inference to LoRA but with zero adapter overhead. PEFT handles this with model.merge_and_unload(). The result is a standard HuggingFace model you can push to Hub.

from peft import PeftModel

base   = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
model  = PeftModel.from_pretrained(base, "./lora-adapter-dir")
merged = model.merge_and_unload()   # returns a plain AutoModelForCausalLM
merged.save_pretrained("./merged-model", safe_serialization=True)

Multi-LoRA Serving

Frameworks like vLLM (v0.4+) and LoRAX load one base model into GPU VRAM and hot-swap per-request adapter weights from CPU RAM or disk. Because only the LoRA A/B matrices need to change (<1 % of model size), adapter swap overhead is minimal — typically <5 ms for r=16.

What to Take Away

• LoRA freezes the base and trains B·A. Initialise B to zeros so the delta starts at zero; A with Kaiming uniform. Total trainable params is typically 0.1–0.5 % of base.
• α/r is your effective learning rate multiplier. Setting α = r means scale 1.0; α = 2r means 2.0 amplification. RSLoRA (α/√r) stabilises training at high ranks.
• Target modules first, then rank. Start with q/k/v/o_proj at r=16. Add MLP layers only if you need significant domain knowledge transfer.
• QLoRA = NF4 quantisation + LoRA adapters. Fits a 7B model plus adapters in ~6 GB VRAM. The three innovations (NF4, double quant, paged optimiser) are all necessary.
• DoRA splits magnitude from direction. Same rank as LoRA, ~1 % extra params, +1–3 % quality on challenging tasks. Use when LoRA under-fits.
• IA3 is 40× smaller than LoRA. Scales key, value, and FFN streams by learned vectors. Best for few-shot classification, not open-ended generation.
• Multi-LoRA serving (vLLM / LoRAX) keeps one base model, swaps adapters per request. Merge only when you want a single monolithic artefact.