How LoRA Changes a Model With Two Small Matrices

LoRA adapts a large model without training a full replacement. It freezes the original weight matrix and learns a constrained update through two smaller matrices. The update can teach a task, concept, character, or style while keeping the base checkpoint unchanged.

Consider a linear layer with input x, weight W, and output h. Full fine-tuning learns a dense update ΔW with the same shape as W. LoRA factorizes that update into B and A:

If W has shape d by k, choose a rank r much smaller than d and k. A has shape r by k. B has shape d by r. Their product BA has shape d by k, so it can be added to W.

A dense d by k update has dk trainable values. LoRA has r(k + d). For a 4096 by 4096 layer:

Training memory also falls because gradients and optimizer state are needed only for adapter parameters. Base weights still need to be available for the forward and backward passes, though they can be quantized or offloaded in methods built around LoRA.

Rank r controls capacity. Alpha controls scale. Many implementations apply α/r so changing rank does not automatically multiply the update magnitude.

• Low rank: compact, faster, and sometimes too restrictive for complex adaptation.
• High rank: more capacity, larger files, more optimizer state, and greater overfitting risk.
• Low effective scale: subtle adapter influence.
• High effective scale: stronger influence and stronger artifacts when the adapter conflicts with the prompt.

The inference “LoRA strength” slider often multiplies the learned update again. It is related to training scale, but it does not retroactively change what the adapter learned.

The original LoRA paper applies updates to selected Transformer matrices. In creative models, adapters may target attention projections, feed-forward layers, convolutional blocks, or text-encoder layers depending on the architecture and trainer.

Placement is part of compatibility. A LoRA trained for one base model cannot be assumed to fit another model with different layer names, dimensions, tokenizer, or latent representation. Even related checkpoints can respond differently if their internal weights moved substantially.

During training, the base W is frozen. Backpropagation computes gradients for A and B. A common initialization makes the initial product zero, so the adapter begins as a no-op and gradually learns an update.

Dataset quality still dominates. Repeated backgrounds can become entangled with a subject. Narrow captions can make the trigger word absorb pose or lighting. More steps can memorize training images instead of improving generalization.

At inference, a framework can evaluate the base path and adapter path separately, or merge the update into a copy of the base weight:

Multiple compatible adapters contribute multiple updates. Matrix addition is simple, but semantic behavior is not. Two style adapters may compete for the same attention directions. A character adapter and a clothing adapter may reinforce or overwrite each other at different strengths.

Keep the base-model identity, adapter hash, scale, target modules, and merge state in the workflow manifest. “Same LoRA filename” is not enough when files are replaced in place.

• Wrong base model: shape errors or weak, incoherent behavior.
• Overtraining: repeated poses, backgrounds, facial features, or color casts.
• Trigger entanglement: the concept appears only with training-scene artifacts.
• Excessive scale: harsh textures, collapsed anatomy, or reduced prompt control.
• Adapter conflict: stacked updates pull the same modules in incompatible directions.
• Rank inflation: larger files and more overfit without measurable benefit.

Primary LoRA paper and model documentation used for the matrix definitions and deployment context.

• LoRA: Low-Rank Adaptation of Large Language Models defines frozen base weights, rank-decomposition matrices, and parameter-efficiency results.
• Microsoft LoRA reference implementation provides the authors’ implementation and examples.
• Z-Image base model card identifies the non-distilled model as the intended foundation for LoRA and structural conditioning in that family.