# Model Quantization: Concepts, Methods, and Why It Matters > Learn how model quantization reduces LLM memory footprint, optimizes inference latency, and maintains precision through PTQ, QAT, GPTQ, and GGUF. Source: https://camnangai.com/en/engineering/what-is-model-quantization Published: 2026-10-29 In modern deep learning deployment, **model quantization** is the technique of converting neural network parameters and activations from high-precision numerical formats to lower-bit representations. This precision conversion drastically reduces the VRAM memory footprint and speeds up execution, allowing you to run state-of-the-art models on hardware accelerators with constrained resources. By mapping continuous floating-point representations into lower-bit discrete structures, you trade a small, controllable degree of numeric granularity for substantial performance gains. When executed correctly, quantization preserves the original accuracy of large neural networks while unlocking higher compute throughput and lower inference latency. ![The core of model quantization: compressing weights and activations to alleviate VRAM bottlenecks and optimize inference](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/01-infographic-en.webp) ## What Is Model Quantization and Why Does LLM Memory Matter? ![IEEE-754 numerical representations and the severe VRAM memory bottlenecks of large language models](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/02-en.webp) Large Language Models (LLMs) derive their generative capabilities from massive parameter counts, which primarily consist of static network weights. Native parameter storage relies on 32-bit full precision IEEE-754 floating-point numbers (FP32). To calculate the bare minimum memory required to load a model in FP32, you multiply the parameter count by 4 bytes (since 32 bits equal 4 bytes). A standard 70-billion parameter model initialized in FP32 requires approximately 280 GB of pure VRAM just to load its weights into memory, completely exceeding the hardware capacity of standard enterprise GPUs such as a single NVIDIA A100 (80 GB) or H100. The computer architecture underlying floating-point representations is defined by the IEEE-754 standard, which splits a given binary sequence into three explicit components: a sign bit, exponent bits, and fraction (or mantissa) bits. The number of exponent bits dictates the _dynamic range_—the complete numerical interval of representable numbers between the absolute minimum and maximum values. Conversely, the number of fraction bits determines the _precision_—the absolute numerical distance and density between two neighboring representable values. ``` IEEE-754 32-Bit Float (FP32): [ Sign (1 bit) | Exponent (8 bits) | Fraction / Mantissa (23 bits) ] Truncated 16-Bit Brain Float (BF16): [ Sign (1 bit) | Exponent (8 bits) | Fraction / Mantissa (7 bits) ] Half Precision Float (FP16): [ Sign (1 bit) | Exponent (5 bits) | Fraction / Mantissa (10 bits) ] ``` Moving away from FP32 involves selecting lower-bit numerical structures. Half precision (FP16) uses 16 bits (1 sign, 5 exponent, 10 fraction), halving memory requirements but significantly shrinking the dynamic range compared to FP32. Bfloat16 (BF16) mitigates dynamic range truncation by acting as a truncated FP32: it preserves 8 exponent bits while dropping the fraction field to 7 bits, matching FP32's dynamic range at half the bit-width. Transitioning further down to 8-bit integer formats (INT8) reduces memory to 1/4th of FP32 (1 byte per parameter), transferring computations into discrete integer arithmetic. During real-time autoregressive inference, total VRAM consumption extends beyond static weight allocation. Dynamic memory bottlenecks are introduced by intermediate layer outputs (_activations_) and the key-value cache (_KV cache_) generated across extended context windows. Because activations depend directly on variable batch sizes and input prompt length, high-precision activations quickly exhaust memory bandwidth. Quantizing both weights and dynamic activation states is therefore critical to fitting multi-billion parameter architectures within hardware constraints. --- ## The Core Math: Scale Factors, Zero-Points, and Affine Mapping ![Mathematical affine mapping and the trade-offs between symmetric and asymmetric quantization](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/03-en.webp) Linear quantization functions map continuous floating-point values `x [α, β]` to discrete low-bit integer values `x_q [α_q, β_q]`. The standard mathematical formulation for affine (asymmetric) quantization incorporates a positive floating-point scale factor `s` and an integer zero-point `z`: ```text x_q = \text{clip}\left(\text{round}\left(\frac{1}{s}x + z\right), \alpha_q, \beta_q\right) ``` ```text \text{clip}(v, l, u) = \begin{cases} l & \text{if } v < l \\ v & \text{if } l \le v \le u \\ u & \text{if } v > u \end{cases} ``` To restore the approximate floating-point value from the quantized integer, you apply the corresponding dequantization operation: ```text x = s(x_q - z) ``` ``` Symmetric Quantization (Absmax): Float Space: [-α] -------------- 0 -------------- [+α] | | | INT8 Space: [-127] ------------ 0 ------------ [+127] (z = 0) Asymmetric Quantization (Zero-Point): Float Space: [α] ------- 0 ------------------- [β] | | | INT8 Space: [-128] ------- [z] ----------------- [127] ``` In _Symmetric Mapping_ (such as Absolute Maximum or _absmax_ quantization), the floating-point dynamic range is forced to be symmetric around zero (`α = -β`), which fixes the zero-point `z = 0`. For a signed 8-bit integer space constrained to `[α_q, β_q] = [-127, 127]`, the scale factor is computed directly from the maximum absolute value `α = (|x|)`: ```text s = \frac{\alpha}{127} ``` In _Asymmetric Mapping_ (zero-point quantization), an arbitrary floating-point range `[α, β]` maps directly to the full integer bounds `[α_q, β_q]` (e.g., `[-128, 127]` for INT8). This requires calculating both `s` and an integer offset `z` to align floating-point `0.0` perfectly with an integer value to eliminate numerical shift errors: ```text s = \frac{\beta - \alpha}{\beta_q - \alpha_q}, \quad z = \text{round}\left(\frac{\beta \alpha_q - \alpha \beta_q}{\beta - \alpha}\right) ``` When input tensors contain extreme outlier values, mapping the true global minimum and maximum causes normal, non-outlier values to be mapped into tiny, coarse integer intervals, creating high quantization error (`Δ x = x - {x}`). To preserve precision across the majority of weights, range _clipping_ is applied. By manually setting bounds `[α, β]` via calibration (e.g., targeting specific percentiles or minimizing MSE/KL-divergence loss), outliers beyond the boundary are clipped to `α_q` or `β_q`, significantly reducing quantization error for non-outliers. During quantized matrix multiplication (`Y = XW + b`), the linear mapping decouples integer arithmetic from floating-point rescalings. Expanding `Y_{i,j} = b_j + ∑_{k=1}^p X_{i,k} W_{k,j}` via dequantization yields: ```text Y_{q,i,j} = z_Y + \frac{s_b}{s_Y}(b_{q,j} - z_b) + \frac{s_X s_W}{s_Y} \left[ \left(\sum_{k=1}^p X_{q,i,k} W_{q,k,j}\right) - z_W \sum_{k=1}^p X_{q,i,k} - z_X \sum_{k=1}^p W_{q,k,j} + p z_X z_W \right] ``` Understanding how this equation executes on hardware requires distinguishing between static offline constants and dynamic runtime calculations: - **Static Offline Constants:** Because static model weights `W_q` and bias terms `b_q` are known prior to inference, the terms `z_Y`, `(s_b / s_Y)(b_{q,j} - z_b)`, `z_X ∑_{k=1}^p W_{q,k,j}`, and `p z_X z_W` are pre-computed offline. - **Dynamic Runtime Terms:** The row-sum term `z_W ∑_{k=1}^p X_{q,i,k}` depends directly on incoming dynamic activations `X_q` and must be computed at runtime during inference. - **Integer GEMM Execution:** The primary computational bottleneck simplifies to an integer matrix multiplication (`∑_{k=1}^p X_{q,i,k} W_{q,k,j}`), which executes directly on high-speed NVIDIA Tensor Core integer units using hardware instructions such as IMMA. The following Python script demonstrates affine scale and zero-point generation along with quantization and dequantization functions that handle bit-width parameterization explicitly: ```python import numpy as np def generate_affine_constants(alpha: float, beta: float, bit_width: int = 8): alpha_q = -(2 ** (bit_width - 1)) beta_q = (2 ** (bit_width - 1)) - 1 # Calculate scale (s) and zero-point (z) s = (beta - alpha) / (beta_q - alpha_q) z = int(np.round((beta * alpha_q - alpha * beta_q) / (beta - alpha))) return s, z, alpha_q, beta_q def quantize_affine(x: np.ndarray, s: float, z: int, alpha_q: int, beta_q: int, bit_width: int = 8) -> np.ndarray: x_q = np.round((1.0 / s) * x + z) x_q = np.clip(x_q, alpha_q, beta_q) # Select appropriate numpy integer container based on bit width if bit_width <= 8: return x_q.astype(np.int8) elif bit_width <= 16: return x_q.astype(np.int16) else: return x_q.astype(np.int32) def dequantize_affine(x_q: np.ndarray, s: float, z: int) -> np.ndarray: return (s * (x_q.astype(np.float32) - z)).astype(np.float32) # Verification example x_real = np.array([-15.2, 0.0, 3.4, 42.8], dtype=np.float32) s, z, a_q, b_q = generate_affine_constants(alpha=-20.0, beta=50.0, bit_width=8) x_quant = quantize_affine(x_real, s, z, a_q, b_q, bit_width=8) x_dequant = dequantize_affine(x_quant, s, z) print(f"Quantized INT8: {x_quant}") print(f"Dequantized FP32: {x_dequant}") ``` --- ## Post-Training Quantization (PTQ) vs. Quantization-Aware Training (QAT) ![Workflow comparison between Post-Training Quantization and Quantization-Aware Training with STE](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/04-en.webp) Quantization can be integrated into a model at two primary points in its lifecycle: after training completes or during the training/fine-tuning loop itself. ``` Post-Training Quantization (PTQ): [ Pre-trained FP32 Weights ] ---> [ Offline Calibration ] ---> [ INT8/INT4 Model ] Quantization-Aware Training (QAT): [ Forward Pass: Fake Quantization (FP32 -> INT4 -> FP32) ] | [ Compute Loss & Backward Pass: STE Gradient Approximation ] | [ Weights Update into Wide Loss Minima ] ---> [ Convert to Native Low-Bit Model ] ``` _Post-Training Quantization (PTQ)_ applies conversion _after_ full-precision model training has finished without modifying base network parameters. Static weight parameters are easily converted using absmax or zero-point linear mappings. Activation parameters, however, vary with every forward pass. PTQ addresses activation quantization using two distinct operational strategies: 1. **Dynamic Quantization:** The dynamic range `[α, β]`, scale `s`, and zero-point `z` for activations are calculated on-the-fly from the real activation distribution immediately after data exits each hidden layer. While precise, this introduces compute overhead during inference. 2. **Static Quantization:** Avoids runtime calculation by passing an unlabeled _calibration dataset_ through the model prior to deployment. Dynamic activation ranges across hidden layers are logged statistically to fix pre-calculated global `s` and `z` values for deployment. _Quantization-Aware Training (QAT)_ embeds quantization logic directly into training or fine-tuning. QAT inserts "fake quantization" nodes into the model graph. These nodes convert floating-point values to low-precision formats (e.g., INT4) and immediately dequantize them back to FP32 before execution: ```text \hat{x} = s \cdot \left( \text{clip}\left(\text{round}\left(\frac{x}{s} + z\right), \alpha_q, \beta_q\right) - z \right) ``` This mechanism allows the model to compute loss metrics that reflect low-precision truncation errors. Because the mathematical operations `round()` and `clip()` have derivatives of zero almost everywhere, standard backward propagation gradients would vanish (`{∂ {x}}{∂ x} = 0`). To overcome this non-differentiable bottleneck, QAT relies on the _Straight-Through Estimator (STE)_. STE approximates the gradient during backpropagation by treating the non-differentiable rounding operation as an identity pass-through within the clipping bounds: ```text \frac{\partial \hat{x}}{\partial x} = \begin{cases} 1 & \text{if } \alpha \le x \le \beta \\ 0 & \text{otherwise} \end{cases} ``` This gradient approximation forces weight updates to steer away from "narrow" loss minima—where small quantization steps cause massive error spikes—and converge into "wide" loss minima that maintain stability despite precision loss. Empirical benchmarks demonstrate that QAT substantially recovers accuracy degradation caused by post-training bit reduction. In evaluation benchmarks using `torchao` and `torchtune`, QAT recovered **96% of the accuracy degradation** on the HellaSwag benchmark for Llama-3-8B compared to standard PTQ, while recovering **67% of quantized accuracy degradation** on Gemma3-4B (and 68% perplexity degradation on WikiText). The following Python snippet demonstrates a two-step `torchao` QAT execution pipeline: ```python import torch from torchao.quantization import quantize_, Int8DynamicActivationIntxWeightConfig, PerGroup from torchao.quantization.qat import QATConfig # 1. Define base low-bit quantization target configuration base_config = Int8DynamicActivationIntxWeightConfig( weight_dtype=torch.int4, weight_granularity=PerGroup(32), ) # 2. Step 1: Prepare model by inserting fake quantization nodes quantize_(model, QATConfig(base_config, step="prepare")) # --- Standard PyTorch Fine-Tuning Loop Occurs Here --- # loss = criterion(model(inputs), targets) # loss.backward() # Gradients pass through fake quants via STE # optimizer.step() # --------------------------------------------------- # 3. Step 2: Convert fake quantization nodes to native low-bit representations quantize_(model, QATConfig(base_config, step="convert")) ``` --- ## Weight-Only vs. Weight-and-Activation Quantization ![Arithmetic intensity and system bottlenecks: memory bandwidth-bound weight-only vs compute-bound weight-and-activation](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/05-en.webp) Choosing which parameter tensors to target dictates both runtime speedups and deployment options, structured around the model's **Arithmetic Intensity**—the ratio of floating-point operations performed per byte of memory transferred (`FLOPs/byte`). ``` Weight-Only Quantization (e.g., INT4 Weights, FP16 Activations): [ VRAM Memory ] --(INT4 Weights Loaded)--> [ Compute Core ] --(FP16 GEMM Ops)--> [ Output ] * Arithmetic Intensity ≈ 1 FLOP/byte * Optimized for Autoregressive Token Generation (Memory-Bandwidth Bound) Weight-and-Activation Quantization (e.g., FP8/INT8 Weights & Activations): [ VRAM Memory ] --(FP8/INT8 Loaded)------> [ Compute Core ] --(Fast Tensor GEMM)--> [ Output ] * Arithmetic Intensity > 100 FLOPs/byte * Optimized for Prompt Prefill / Context Processing (Compute Bound) ``` _Weight-Only Quantization_ compresses static model parameters (e.g., into INT4, NF4, or INT8) while maintaining dynamic hidden-state activations in standard half precision (FP16 or BF16). Single-token autoregressive generation has an extremely low arithmetic intensity (`≈ 1 FLOP/byte`). In this regime, the system is memory-bandwidth bound: the GPU spends most of its time transferring weight matrices from VRAM to compute logic. Because weight-only schemes reduce weight volume by 50% to 75%, weight transfer latency drops dramatically, speeding up token generation even though weights must be dequantized back to floating-point before linear matrix calculations. _Weight-and-Activation Quantization_ converts both static weights and dynamic activations into low-bit formats (such as INT8/INT8 or FP8/FP8). This approach shines in compute-bound phases, such as prompt processing (_prefill_), high-batch generation, or extended context processing, where arithmetic intensity scales to `100+ FLOPs/byte`. Because both matrices in `X · W` reside in matched low-precision integer or FP8 types, hardware accelerators bypass floating-point conversion entirely, executing matrix multiplication directly on high-speed integer or FP8 Tensor Cores. For extreme low-bit compression, architectural designs move beyond standard integer conversions: - **BitNet (1-bit):** Replaces standard linear layers with specialized `BitLinear` modules. Weights are quantized down to a single binary bit representing `\{-1, 1\}` using a zero-centered signum function: ```text W_q = \text{Sign}(W - \bar{W}) ``` Activations are quantized to INT8 using absmax scaling. Dequantization rescales output matrices using tracked scale metrics `α` (maximum absolute activation value) and `β` (average absolute weight value). - **BitNet 1.58b (Ternary):** Advances 1-bit architectures by introducing ternary weights taking values of `\{-1, 0, 1\}`. It applies _absmean_ weight quantization, scaling weights by their average absolute magnitude `α = (1 / nm) ∑ |W_{i,j}|` and rounding to the nearest ternary value: ```text W_q = \text{clip}\left(\text{round}\left(\frac{W}{\alpha}\right), -1, 1\right) ``` Adding `0` transforms standard matrix multiplication into pure addition and subtraction: ```text \text{Weight} = 1 \implies \text{Add}, \quad \text{Weight} = -1 \implies \text{Subtract}, \quad \text{Weight} = 0 \implies \text{Bypass Instruction} ``` At the hardware level, when a weight evaluates to `0`, the compute unit bypasses the instruction entirely. This skips memory transfers and multiply-accumulate (MAC) cycles, enabling feature filtering directly in hardware. A 13-billion parameter BitNet 1.58b model achieves lower latency and memory consumption than a 3-billion parameter native FP16 LLM. --- ## Modern LLM Quantization Algorithms: GPTQ, AWQ, and bitsandbytes ![Technical comparison of modern quantization algorithms: GPTQ, AWQ, and bitsandbytes](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/06-en.webp) Selecting an operational algorithm depends on deployment targets, calibration constraints, and hardware architectures. ``` GPTQ (Second-Order Inverse-Hessian Optimization): [ Weight Matrix Row ] ---> [ Quantize Weight ] ---> [ Calculate Error (q) ] | [ Update Remaining Unquantized Weights ] <--- [ Scale via Inverse-Hessian (h) ] AWQ (Activation-Aware Channel Protection): [ Inspect Activations ] ---> [ Identify Top 1% Salient Channels ] ---> [ Protect Salient Channels ] bitsandbytes (Zero-Shot Runtime Quantization): [ Load Model ] ---> [ On-the-Fly Conversion to INT8/NF4 ] ---> [ PEFT / QLoRA Integration ] ``` ### GPTQ GPTQ is an asymmetric 4-bit/2-bit Post-Training Quantization method designed for efficient layer-by-layer weight compression. It measures weight importance by constructing a second-order derivative matrix of the loss function known as the inverse-Hessian matrix (`H^{-1}`). During row-by-row weight conversion within a linear layer, quantizing a specific weight parameter produces a localized quantization error `q`. GPTQ multiplies this scalar error `q` by the corresponding inverse-Hessian element `h` and immediately redistributes this offset to update the remaining unquantized weights in the same row: ```text w_{\text{remaining}} \leftarrow w_{\text{remaining}} - q \cdot h ``` By continuously compensating unquantized parameters for precision loss in earlier weights, GPTQ preserves global layer functionality. This yields significant execution speed advantages during autoregressive generation on modern GPUs. ### AWQ (Activation-Aware Weight Quantization) Unlike GPTQ, which redistributes errors across uniform weight matrices, AWQ relies on the insight that not all weights in a neural network are equally important. By analyzing activation distributions during calibration, AWQ observes that protecting a small fraction (e.g., 1%) of salient weight channels—those corresponding to high activation magnitudes—drastically reduces quantization error. Instead of leaving these salient channels in FP16 (which creates structural memory overhead), AWQ protects them by applying a channel-wise scaling factor before quantization. Protecting parameters per channel based on activation magnitude avoids uniform precision loss, preserving model reasoning capabilities without requiring second-order Hessian calculations. ### bitsandbytes `bitsandbytes` provides zero-shot quantization, converting models on-the-fly during load time without requiring calibration datasets. It provides signed 8-bit quantization (`LLM.int8()`) along with specialized 4-bit data structures: NormalFloat4 (NF4) and Float4 (FP4). NF4 is an information-theoretically optimal quantile quantization scheme that maps non-uniformly distributed, normally distributed weights into equal-frequency 4-bit bins. `bitsandbytes` forms the primary backbone for Parameter-Efficient Fine-Tuning (PEFT) and QLoRA. It allows you to freeze a 4-bit quantized base model, attach trainable full-precision LoRA adapter layers, and execute gradient fine-tuning without memory overflows. Once trained, these adapters can be merged back into the dequantized base parameters with zero performance degradation. ### Performance & Latency Trade-offs Evaluating generation latency across NVIDIA GPUs (such as A100, T4, and Titan RTX) highlights structural speed differences. For raw autoregressive token generation with attention caching enabled (`use_cache=True`), GPTQ execution (using optimized exllama kernels) is significantly faster than `bitsandbytes` 4-bit execution—achieving up to **`2×` token throughput** at small batch sizes (e.g., batch size = 4). However, `bitsandbytes` remains faster during adapter fine-tuning because GPTQ requires disabling exllama kernels during backward pass gradient processing. --- ## Formats and Deployment: GGUF, llama.cpp, and vLLM ![Deployment ecosystem: local CPU/GPU memory offloading with GGUF vs enterprise serving with vLLM](https://camnangai.com/articles/luong-tu-hoa-mo-hinh-la-gi/07-en.webp) Converting model parameters into low-bit representations requires serialized target formats and specialized runtime frameworks. ``` GGUF Hierarchical Block Structuring: [ Super-Block ] -------------------------------------> Scale (s_super) |--> [ Sub-Block 1 ] -> Absmax Quantized Data ----> Scale (s_sub) |--> [ Sub-Block 2 ] -> Absmax Quantized Data ----> Scale (s_sub) vLLM High-Throughput Serving Engine: [ PagedAttention KV Cache ] + [ Tensor Parallelism ] + [ Compiled Quant Backend (FP8/GPTQ) ] ``` ### GGUF & llama.cpp GGUF (GGML Unified Format) is a file serialization format designed for edge execution via `llama.cpp`. GGUF structures model weights using hierarchical _block-wise quantization_. Instead of computing single uniform scale parameters across an entire tensor layer, parameters are partitioned into large _super-blocks_, which are subdivided into smaller _sub-blocks_. ```text s_{\text{sub}} = \text{absmax}(W_{\text{sub}}), \quad s_{\text{super}} = \text{quantize}(s_{\text{sub}}) ``` The sub-block scale factor `s_{sub}` is quantized using the higher-precision scale factor `s_{super}` of its parent super-block. This multi-level hierarchy supports custom bit-widths (from 2-bit up to 6-bit) and enables dynamic CPU/GPU memory offloading, allowing layers to be split across consumer CPU RAM and GPU VRAM when VRAM capacity is insufficient. ### Production Serving Frameworks (vLLM) In enterprise server environments, frameworks like `vLLM` serve quantized models (GPTQ, TorchAO, FP8) using PagedAttention, Tensor Parallelism, and kernel fusion via `torch.compile()`. Rather than focusing on single-user, offloaded execution, `vLLM` optimizes batch generation throughput and memory allocation for simultaneous API queries, pushing hardware compute efficiency to its limit. You can launch a pre-quantized FP8 model using vLLM via the CLI as follows: ```bash # Serve a TorchAO/FP8 quantized model with compiler optimizations VLLM_DISABLE_COMPILE_CACHE=1 vllm serve pytorch/Qwen3-32B-FP8 \ --tokenizer Qwen/Qwen3-32B \ -O3 ``` --- ## The Triple Trade-off: VRAM Footprint, Latency, and Perplexity Quantizing neural networks requires managing trade-offs between VRAM memory footprint, throughput/latency, and task accuracy (perplexity). ``` [ Memory Footprint ] / \ / \ / \ [ Latency / Speed ] --- [ Perplexity / Accuracy ] ``` ### VRAM Footprint Savings Moving from FP16 down to 4-bit representations drastically reduces hardware memory footprints. A Llama-2-13B model in FP16 consumes approximately 29 GB of peak VRAM; applying 4-bit quantization (via GPTQ or `bitsandbytes`) reduces peak memory usage to approximately **10.5 GB to 11 GB**, enabling deployment on consumer-grade hardware. Similarly, quantizing Gemma3-12B to INT4 yields a **65% reduction in total VRAM usage** on NVIDIA H100 hardware, dropping overall memory footprint to under 3.4 GB when combined with INT8 activations. ### Latency & Throughput Dynamics Inference performance differs between prompt processing (_prefill_) and token generation: | Batch Size | Metric | FP16 Baseline | GPTQ (4-bit Exllama) | bitsandbytes (4-bit) | | :------------------ | :---------------- | :--------------- | :------------------- | :------------------- | | **Batch Size = 1** | Latency per token | 36.95 ms | **33.71 ms** | 52.00 ms | | | Throughput | 27.05 tok/s | **29.66 tok/s** | 19.23 tok/s | | | Peak VRAM | 29,152 MB | **10,484 MB** | 11,018 MB | | **Batch Size = 16** | Throughput | **228.76 tok/s** | 167.68 tok/s | 140.38 tok/s | | | Peak VRAM | 53,986 MB | **34,777 MB** | 35,532 MB | For single-sequence queries (batch size = 1), GPTQ 4-bit speeds up generation compared to FP16 by eliminating memory-bandwidth overhead. However, at larger batch sizes (e.g., batch size = 16), quantized models can exhibit lower token throughput than FP16 due to the runtime dequantization overhead required before compute operations. Enabling attention caching (`use_cache=True`) remains critical; disabling the KV cache drops generation throughput dramatically across all precision levels. ### Perplexity & Accuracy Loss Evaluations from the Open-LLM Leaderboard demonstrate that precision reduction causes minor benchmark degradation, with larger base models exhibiting far greater resilience against quantization errors than smaller variants: ``` Llama-2 Benchmark Accuracy Across Quantization Schemes: Llama-2-7B: FP16 Baseline : [====================================] 54.32 bnb 4-bit : [=================================== ] 53.40 GPTQ 4-bit : [=================================== ] 53.23 Llama-2-13B: FP16 Baseline : [========================================] 58.66 GPTQ 4-bit* : [======================================= ] 58.03 (*With ActOrder) GPTQ 4-bit : [======================================= ] 57.56 bnb 4-bit : [====================================== ] 56.90 ``` Larger architectures absorb bit-width reduction with minimal accuracy loss because their increased parameter counts provide redundant representation capacity, minimizing overall loss degradation. --- ## How to Choose the Right Quantization Scheme for Production To select the optimal quantization strategy, map your operational target directly to your available hardware acceleration features: ``` [ Deployment Target ] | +---------------------+---------------------+ | | | [ Fine-Tuning ] [ Server Inference ] [ Local / Edge ] | | | v v v bitsandbytes (NF4) GPTQ / FP8 via GGUF via llama.cpp or TorchAO QAT vLLM (CPU/GPU Offload) ``` 1. **Fine-Tuning Adapters on Workstations:** Use **bitsandbytes (4-bit NF4)** or **TorchAO QAT**. `bitsandbytes` allows zero-shot model loading directly into QLoRA configurations without requiring pre-calibration, keeping memory overhead minimal. If you need maximum downstream accuracy without bit-degradation, integrate `TorchAO` QAT fine-tuning recipes to let the model adapt its parameters to integer bounds during training. 2. **High-Throughput Enterprise Servers:** Use **GPTQ (4-bit)** or native **FP8 (dynamic/static)** served via **vLLM** or **SGLang**. Select GPTQ's exllama kernels when serving Ampere/Ada architectures to maximize single-stream autoregressive token generation. Target FP8 when deploying on NVIDIA Hopper (H100/H200) or Blackwell architectures to run directly on native FP8 Tensor Cores for compute-bound, multi-user batched workloads. 3. **Local Offloading and Edge Devices:** Choose **GGUF executed via llama.cpp** (or ExecuTorch for mobile deployment). GGUF's hierarchical block-wise structure allows you to split layer execution across heterogeneous system memory (CPU RAM) and consumer VRAM, maintaining execution stability on constrained edge hardware. --- ## References - [Quantization Overview — Hugging Face](https://huggingface.co/docs/transformers/main/en/quantization) - [Overview of Natively Supported Quantization Schemes in 🤗 Transformers — Hugging Face](https://huggingface.co/blog/overview-quantization-transformers) - [Quantization for Neural Networks — Lei Mao](https://leimao.github.io/article/Neural-Networks-Quantization/) - [A Visual Guide to Quantization — Maarten Grootendorst](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization) - [TorchAO: Architecture Optimization and Quantization — PyTorch](https://github.com/pytorch/ao) - [A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms — arXiv](https://arxiv.org/abs/2409.16694) - [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration — arXiv](https://arxiv.org/abs/2306.00978) - [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — arXiv](https://arxiv.org/abs/2210.17323)