PEFT stands for Parameter-Efficient Fine-Tuning. At its heart, it does two main things:

1. Freezes the majority of the base model’s parameters: The original weights remain completely untouched, which saves an immense(a lot) amount of VRAM and prevents catastrophic forgetting.

2. Applies a tiny fraction of trainable parameters: Instead of updating billions of weights, you only train a very small set of extra parameters (like adapter layers) or modify how weights are represented (like low-rank matrix approximations).

In this part, We need get know 3 algorithms

LoRA(Low-Rank Adaption)

LoRA introduces a small trainable update matrix that is added to the original frozen weight matrix. During fine-tuning, this update is represented as the product of two low-rank matrices, reducing the number of trainable parameters.

For an original weight matrix W, full fine-tuning learns:
$$W^{\prime }=W+\nabla W$$
LoRA assumes that the update ΔW can be approximated by two low-rank matrices:
$$\nabla W\approx BA$$
Therefore:
$$W^{\prime }=W+BA$$
where:

• W: frozen original weight matrix;
• $A\in {\mathbb{R}}^{r\times d}$;
• $B\in {\mathbb{R}}^{d^{\prime }\times r}$;
• r: small rank , such as 4,8, or 16;
• $BA\in {\mathbb{R}}^{d^{\prime }\times d}$, having the same shape as W.

During training:
W is frozen, A, B are trainable

Technically, LoRA does not first compute a full ΔW and then factorize it. It directly parameterizes the update as BA and trains A and B through backpropagation.

For example, if:
$$W\in {\mathbb{R}}^{4096\times 4096},r=8$$

Full update parameters:

4096×4096≈16.8 million

LoRA parameters:

4096×8+8×4096=65,536

So LoRA is essentially:

low-rank parameterization of the weight update, rather than full matrix factorization of the original LLM weights

QLoRA

Q means Quantized,

It combines:

• Quantization: Convert the base model to low-bit precision, usually 4-bit, to reduce memory usage.
• LoRA: Freeze the quantized base model and train only two small low-rank matrices.

The updated weights can be written as:
$$W^{\prime }=W_{4bit}+BA$$

where $W_{4bit}$​ is the frozen 4-bit model and $BA$ is the trainable LoRA update.

In simple terms:

QLoRA compresses the original model and then trains a small adapter on top of it.

This allows larger LLMs to be fine-tuned with much less GPU memory while maintaining performance close to standard LoRA.

DoRA

The Problem with LoRA

Any vector can be represented by two components: its magnitude and its direction. However, LoRA learns the weight update through a single low-rank matrix。Therefore, the same low-rank update must simultaneously capture changes in both the magnitude and the direction of the weights.
For simple tasks, this approximation may be sufficient. However, for more complex tasks, the required update may contain several independent patterns. Since the update is constrained to a low-rank subspace of dimension r, all parameter changes must be represented as combinations of only r basis directions.
If the task requires more independent changes than the selected rank can express, these changes cannot be represented exactly. The model has to approximate them within the limited low-rank space, which may lead to interference or a compromise between different adaptation requirements.

A Simple Example

Assume the required weight update is:
$$△W=\left[\begin{array}{cc}1& 0\\ 0& -1\end{array}\right]$$
This update means:

• increase the first direction by 1;
• decrease the second direction by 1;
• keep the two directions independent.
  The matrix has rank 2: $rank(△W)=2$
  However, if LoRA uses r=1, its update must have the form:
  $$△W=BA,rank(BA)\le 1$$
  A rank-1 update can change multiple parameters, but all changes must follow one shared pattern. It cannot independently increase one direction while decreasing another. Therefore, LoRA cannot represent the target update exactly and must approximate it:
  $$△w\approx BA$$
  This approximation may force the model to compromise between different requirements. For example, it may learn to increase the first direction correctly but fail to decrease the second one sufficiently.

What Does DoRA Add Beyond LoRA?

DoRA stands for Weight-Decomposed Low-Rank Adaptation.

DoRA separates these two types of changes. It decomposes the pretrained weight matrix into a magnitude component and a direction component:
$$W=m\frac{V}{\Vert \begin{array}{c}V\end{array}\Vert }$$
During fine-tuning:

• the direction is updated using the LoRA matrices B and A;
• the magnitude m is learned as a separate trainable parameter.

The updated weight can be written as:
$$W^{\prime }=m^{\prime }\frac{W+BA}{\Vert \begin{array}{c}W+BA\end{array}\Vert }$$

In this way, DoRA does not force the same low-rank update to learn both magnitude and direction. Instead:

LoRA component BA → learns changes in direction
Trainable magnitude m' → learns changes in scale

This gives the model more flexibility and can make its behavior closer to full fine-tuning, especially when the task requires complex weight updates.

However, DoRA does not remove the low-rank constraint completely. The directional update is still represented by BA, so its effectiveness still depends on the selected rank r. DoRA mainly improves how the available adaptation capacity is used.

Rank
矩阵是模型处理和变换信息的基本工具；rank 表示这个矩阵实际能表达多少个独立的信息方向。
矩阵的 rank 表示矩阵中真正独立的信息方向数量，也表示矩阵变换后保留下来的有效维度。

References:
LoRA-happyLLM
LoRA:Low-Rank Adaptation of Large Language Models
QLORA: Efficient Finetuning of Quantized LLMs
推出高性能微调解决方案 DoRA，LoRA 的替代选择
DoRA-知乎