Fine-Tuning

01— The Memory Wall

The hidden arithmetic of model training.#

Most engineers assume that if a 70B model fits into 140 GB of VRAM for inference at FP16, it can be fine-tuned with 160 GB. That assumption crashes the cluster. Full fine-tuning requires tracking optimizer states, gradients, and activation buffers that multiply weight memory by a factor of eight.

1,120 GB

Full FT VRAM (70B AdamW)

48 GB

QLoRA VRAM (70B Single GPU)

99.2%

Trainable Param Reduction

0 ms

LoRA Inference Latency Overhead

Consider a running task carried through this entire article: fine-tuning an open 70-billion parameter foundation model (Llama-3-70B) to act as a clinical emergency triage classifier.[1] The system must intake unstructured emergency room transcripts and output structured urgency tiers (ESI 1 through 5), ICD-10 diagnostic codes, and immediate red-flag contraindications.

If you execute a standard full fine-tuning run using the AdamW optimizer in mixed precision (16-bit float weights), your VRAM budget breaks down into non-negotiable physical components per parameter \(\Phi\):

\text{Memory}_{\text{Full FT}} = \underbrace{2\Phi}_{\text{FP16 Weights}} + \underbrace{2\Phi}_{\text{FP16 Gradients}} + \underbrace{4\Phi}_{\text{FP32 Master Weights}} + \underbrace{4\Phi}_{\text{FP32 Momentum}} + \underbrace{4\Phi}_{\text{FP32 Variance}} = 16\Phi \text{ bytes}

For our 70B clinical model, \(16 \times 70 \times 10^9 \text{ bytes} \approx 1{,}120 \text{ GB}\) of raw state memory before allocating a single megabyte for sequence activations or KV-cache. This demands an expensive cluster of at least two 8-way H100 (80GB) nodes running DeepSpeed ZeRO-3 or FSDP just to begin epoch zero.

Full fine-tuning rewrites all 70 billion weights.Every downstream hospital customer or regional medical dialect requires maintaining another separate 140 GB weight checkpoint on disk and in memory.

VRAM Allocation Breakdown: Full Fine-Tuning vs. PEFT (70B Model, FP16)

How to read this: Bars show required physical VRAM for training a 70B parameter model at sequence length 2048. In full fine-tuning, optimizer momentum and variance consume 75% of the total footprint. Freezing the base weights eliminates gradient and optimizer state requirements for the 70B base, reducing state tracking solely to the tiny adapter parameters.

02— Reparameterization

Factorization & 4-bit quantization.#

Rather than optimizing the full weight matrix \(W \in \mathbb{R}^{d \times k}\), low-rank adaptation decomposes the weight update into two miniature bottleneck matrices whose rank \(r \ll d\). When training ends, these matrices collapse back into the base weights with zero inference latency overhead.

Step 01 Freeze Base

The original pre-trained weight matrix \(W_0\) is frozen completely; no gradients are calculated or stored for it.

Step 02 Inject Low-Rank

Two trainable matrices \(A \in \mathbb{R}^{r \times k}\) and \(B \in \mathbb{R}^{d \times r}\) are added in parallel with rank \(r \in \{8, 16, 64\}\).

Step 03 Train A and B

Forward pass computes \(h = W_0 x + \frac{\alpha}{r} B A x\). Only \(A\) and \(B\) receive gradients and optimizer updates.

Step 04 Merge Adapter

Fold \(\Delta W = \frac{\alpha}{r} B A\) directly into \(W_0\). Yields the exact same forward kernel speed with zero serving penalty.

Hu et al. demonstrated that the change in weights \(\Delta W\) during task adaptation has a low intrinsic rank.[2] For a linear projection in our clinical triage transformer—such as the attention query projection \(W_q \in \mathbb{R}^{4096 \times 4096}\)—a full weight update contains \(16.7 \times 10^6\) parameters. By factorizing into \(B \times A\) with rank \(r = 16\):

W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A, \quad A \sim \mathcal{N}\left(0, \frac{1}{r}\right), \quad B = 0

Matrix \(A\) projects the 4096-dimensional hidden state down to 16, and matrix \(B\) projects it back up to 4096. The parameter count drops from \(16{,}777{,}216\) down to \(2 \times 4096 \times 16 = 131{,}072\)—a 99.2% reduction for that projection. Because \(B\) is initialized to zeros, \(\Delta W = 0\) at initialization; the model starts in the exact state as the pre-trained base without training disruption.

The scaling hyperparameter \(\alpha\) (typically set to \(2 \times r\)) stabilizes optimization when adjusting the rank \(r\), avoiding the need to re-tune the learning rate if the rank is changed from 16 to 32.

While LoRA eliminates optimizer states for the base model, the frozen weights of our 70B clinical triage model still occupy 140 GB of VRAM. Dettmers et al. introduced QLoRA to compress the frozen base into 4-bit precision without degrading downstream accuracy.[3] QLoRA achieves this through three innovations:

NormalFloat4 (NF4) Data Type

An information-theoretically optimal quantile quantization for normally distributed weights. Each bin has an equal number of expected parameters, minimizing quantization error compared to standard INT4.

Double Quantization (DQ)

Quantizes the quantization constants themselves. Compresses FP32 scaling blocks from 0.5 bits per parameter down to 0.127 bits per parameter, saving an additional 3 GB across a 70B model.

During the forward and backward passes, the 4-bit NF4 weights are dequantized to 16-bit BF16 just-in-time for tensor dot products with the incoming activation vectors. Paged Optimizers then utilize CUDA Unified Memory to page memory spikes to CPU RAM during activation peaks, preventing out-of-memory crashes on a single 48GB GPU (such as an NVIDIA RTX 6000 Ada or A6000).

Mechanism of LoRA Matrix Factorization & Zero-Latency Inference Merging

How to read this: Left demonstrates the parallel low-rank bypass during backward and forward propagation during training. Right shows weight folding at deployment: because matrix multiplication is distributive, \(W_0 x + \Delta W x = (W_0 + \Delta W) x\). Merging prior to serving eliminates auxiliary runtime kernels entirely.

03— Attention Steering

Prefix Tuning & continuous prompts.#

Instead of altering weight matrices, attention steering modulates intermediate activations directly. Continuous virtual token embeddings prepend to key-value heads or input embeddings, guiding generation without mutating a single parameter in the transformer body.

Formulated by Li & Liang, Prefix Tuning keeps the language model parameters completely frozen.[4] In standard multi-head self-attention, queries, keys, and values are computed from the sequence activations: \(Q = X W_q\), \(K = X W_k\), \(V = X W_v\). Prefix Tuning prepends learnable prefix parameters \(P_K \in \mathbb{R}^{l \times d}\) and \(P_V \in \mathbb{R}^{l \times d}\) (where \(l\) is the prefix length, typically 10 to 30 virtual tokens):

\text{Attention}(Q, \tilde{K}, \tilde{V}) = \text{softmax}\left(\frac{Q \tilde{K}^T}{\sqrt{d_k}}\right) \tilde{V}, \quad \text{where } \tilde{K} = [P_K; K], \; \tilde{V} = [P_V; V]

Because these prefixes exist at every layer rather than just the input embedding layer, they steer the entire hierarchical representation. For our emergency triage system, the prefix vectors establish an immutable context prior across all 80 transformer layers: instructing attention heads to prioritize diagnostic symptoms (e.g., “diaphoresis”, “substernal chest pressure”) regardless of how chaotic the input nurse notes are formatted.

To stabilize early training, prefixes are initially parameterized through an auxiliary multi-layer perceptron (MLP) over small embedding indices: \(P = \text{MLP}(E_{\text{prefix}})\). Once training converges, the MLP is discarded, retaining only the computed prefix tensors \([P_K; P_V]\) for serving.

Traditional prompt engineering (discrete “hard prompts”) suffers from extreme sensitivity: swapping a synonym or moving punctuation in a clinical prompt can trigger unpredictable drops in diagnostic classification accuracy. P-Tuning (Liu et al.) solves this by replacing manual discrete words with continuous prompt embeddings trained through a BiLSTM or MLP prompt encoder.[5]

Discrete Hard Prompts

Constrained to natural language dictionary tokens \(\mathcal{V}\). Non-differentiable; small prompt permutations cause sharp, non-convex loss spikes in clinical evaluation.

P-Tuning Continuous Prompts

Navigates the continuous embedding manifold \(\mathbb{R}^{d}\) via gradient descent. The prompt encoder conditions dependencies between virtual prompt tokens.

In P-Tuning v2, continuous prompts are extended across all layers (unifying prompt tuning with prefix tuning). While P-Tuning trains only 0.01% of parameters, it suffers from a subtle operational constraint: unlike LoRA, prefix and continuous tokens occupy sequence context positions, consuming active context window length and requiring distinct KV-cache slices during generation.

Attention Steering: Key-Value Prefix Injection vs. Input P-Tuning

How to read this: Prefix Tuning (left) intervenes directly inside the self-attention calculation by injecting virtual key and value vectors at every single layer. P-Tuning (right) maps virtual discrete tokens into a continuous embedding manifold via a lightweight encoder before feeding the sequence into the frozen transformer base.

04— Bottleneck Modules

The architectural latency tax of adapters.#

Before LoRA became dominant, adapter tuning (Houlsby et al., Pfeiffer et al.) proved that inserting small residual bottleneck layers between transformer blocks was sufficient for task specialization. However, physical module insertion introduces a runtime latency penalty that cannot be compiled away.

Houlsby et al. introduced the classic parameter-efficient adapter.[6] The module takes hidden state dimension \(d\), applies a down-projection matrix \(W_{\text{down}} \in \mathbb{R}^{m \times d}\) into a low-dimensional bottleneck \(m \ll d\), evaluates a non-linear activation function \(\sigma(\cdot)\) (such as GeLU), and projects back up via \(W_{\text{up}} \in \mathbb{R}^{d \times m}\), accompanied by an outer skip connection:

\text{Adapter}(h) = \sigma(h W_{\text{down}}) W_{\text{up}} + h

Pfeiffer et al. later optimized this design by reducing insertion to a single adapter placed strictly after the transformer feedforward layer (FFN), cutting parameter count in half with minimal impact on accuracy.[7]

Modular Multi-Task Serving

Adapters are outstanding for multi-tenant serving. A single 70B clinical model can remain in VRAM while swapping 15 MB adapter files per request for pediatric triage, cardiology review, and billing coding.

The Sequential Latency Penalty

Because the adapter sits in serial execution between frozen layers, it introduces sequential GPU kernel launches. In batched inference, this adds 12% to 25% inference latency that cannot be eliminated by weight folding.

Adapters alter the network execution graph.Unlike LoRA (which can mathematically merge into \(W_0\)), adapter layers require running dedicated matrix multiplication and activation kernels during every single token generation step.

Series Bottleneck Adapter Module (Houlsby Structure)

How to read this: Hidden representations pass from a frozen transformer block into the adapter. The dimensionality is compressed to bottleneck rank \(m\), passed through a non-linearity, projected back to original dimension \(d\), and merged with the residual skip. While parameters are low, each adapter layer requires a discrete GPU kernel execution.

05— Tradeoff Frontier

Selecting the optimal PEFT strategy.#

Every parameter-efficient fine-tuning technique balances three competing pressures: parameter storage efficiency, training VRAM requirements, and post-training inference latency. The winning architecture is determined by your deployment hardware and serving topology.

Zero Latency Serving

LoRA

Fold weights directly into the base model. Best for single-tenant production deployments with zero inference latency overhead.

Minimal Hardware Budget

QLoRA

Enables fine-tuning 70B parameter models on a single 48GB workstation GPU through 4-bit NF4 quantization and paged optimizers.

Multi-Tenant Routing

Adapters

Hot-swap lightweight 15MB task modules at runtime for hundreds of specialized client tasks on a shared foundation base.

Attention Guidance

Prefix Tuning

Guarantees strong context prior steering across all attention layers without altering pre-trained linear projection weights.

To deploy our clinical triage model into hospital IT infrastructure:

If dedicated inference serving is planned: Use LoRA (Rank \(r=16\), \(\alpha=32\)) applied across all linear attention and MLP projections (\(W_q, W_k, W_v, W_o, W_{\text{gate}}, W_{\text{up}}, W_{\text{down}}\)). Merge weights back into \(W_0\) for zero runtime penalty.
If GPU training hardware is constrained to a single workstation: Use QLoRA with 4-bit NormalFloat base weights and 16-bit LoRA adapters.
If serving hundreds of personalized clinic adapters simultaneously: Maintain one shared base model in VRAM and use S-LoRA or Pfeiffer Adapters to dynamically batch heterogeneous task adapters without duplicating base weights.[8]

\text{Recommendation} = \begin{cases} \text{QLoRA} & \text{if VRAM} \le 48\text{ GB during training} \\ \text{LoRA (Merged)} & \text{if strict 0 ms latency overhead required} \\ \text{Multi-Adapter Routing} & \text{if serving } > 20\text{ tenant tasks on 1 model} \end{cases}

Interactive Architecture Check

Which PEFT technique allows a 70B parameter model to achieve completely zero added inference latency overhead when deployed in production?

Correct! Because matrix multiplication is distributive across addition, the trained low-rank matrices \(B \times A\) can be algebraically added directly into the pre-trained weight matrix \(W_0\) before serializing to disk or loading into vLLM/TensorRT-LLM. The deployed model runs standard GEMM kernels with zero added runtime latency.

06— Sources

Primary literature & PEFT foundations.#

The parameter-efficient fine-tuning landscape rests on these peer-reviewed foundations. Review the original papers for formal proofs regarding intrinsic dimensionality, NormalFloat optimality, and attention steering.

Llama 3 Herd of Models. Meta AI Research, 2024. Architecture and fine-tuning details for large open weights. Read the paper ↗
LoRA: Low-Rank Adaptation of Large Language Models. Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, & Chen, 2021. Formulation of low-rank matrix decomposition for transformer adaptation. Read the paper ↗
QLoRA: Efficient Finetuning of Quantized LLMs. Dettmers, Pagnoni, Holtzman, & Zettlemoyer, 2023. Introduces NF4, double quantization, and paged optimizers for single-GPU fine-tuning. Read the paper ↗
Prefix-Tuning: Optimizing Continuous Prompts for Generation. Li & Liang, ACL 2021. Layer-wise continuous prefix prepending in self-attention mechanisms. Read the paper ↗
GPT Understands, Too. (P-Tuning). Liu, Zheng, Du, Ding, Yang, & Tang, 2021. Replacing discrete prompt tokens with continuous virtual embeddings trained via neural prompt encoders. Read the paper ↗
Parameter-Efficient Transfer Learning for NLP. Houlsby, Giurgiu, Jastrzebski, Morrone, De Laroussilhe, Gesmundo, Attariyan, & Gelly, ICML 2019. Initial introduction of bottleneck adapter modules inside transformer blocks. Read the paper ↗
AdapterFusion: Non-Destructive Task Composition for Transfer Learning. Pfeiffer, Kamath, Rücklé, Cho, & Gurevych, EMNLP 2021. Reduced single-adapter FFN bottleneck placement and composition. Read the paper ↗
S-LoRA: Serving Thousands of Concurrent LoRA Adapters. Sheng, Chen, Gao, Zheng, & Stoica, 2023. Unified memory management for high-throughput multi-adapter LLM serving. Read the paper ↗

Step by Step LLM Evaluation

Exploring Machine Learning Algorithms

Parameter-Efficient Fine-Tuning

The hidden arithmetic of model training.#