MODEL FORMATS · EVIDENCE-LED PRACTICE GUIDE

AI Model Quantization Explained: GGUF, GPTQ, AWQ, FP8 and NVFP4

Lower precision can make a model practical—but a bit-width label alone never proves that the model will fit, run quickly or keep its original quality.

DIRECT ANSWER

Which AI model quantization format should you use?

Choose the format your exact serving runtime and hardware support, then verify quality on your own prompts. GGUF is a file format commonly used with llama.cpp; GPTQ and AWQ name weight-quantization approaches; FP8 and NVFP4 describe numerical formats and quantization schemes whose acceleration depends on the chosen kernels and devices. They are not interchangeable labels and they do not predict tokens per second by themselves. Review the upstream runtime’s current supported-method matrix before obtaining a checkpoint.

Rule of thumb: decide the runtime first, verify the actual quantized artifact second, and benchmark accuracy, peak memory and latency third. Never infer compatibility from “4-bit” alone.

Understand what quantization changes

Most open-weight checkpoints publish floating-point tensors, often BF16 or FP16. Quantization stores some values in lower-precision representations to reduce storage and potentially accelerate supported matrix operations. Depending on the scheme it may quantize weights only, weights and activations, or selected layers. Group sizes, scale values, zero-points, outlier handling and mixed-precision exceptions affect both fidelity and physical memory.

Compute is not identical to storage. A method described as 4-bit weight-only may dequantize into a higher-precision compute path on the fly. Hardware with a native low-precision matrix instruction may benefit differently from hardware requiring conversion. Every architecture has constraints, and compatible kernels can change across versions. Quantization can reduce transfer and memory pressure without improving end-to-end latency if context processing, attention or communication dominates your workload.

GGUF, GPTQ, AWQ, FP8 and NVFP4 compared

Terminology and typical deployment path; compatibility must be checked for the pinned version.
LabelWhat it isCommon ecosystemImportant caution
GGUFContainer/file representation used for multiple quantization typesllama.cpp and compatible runtimesNot a single 4-bit algorithm or guarantee of model support
GPTQPost-training weight quantization with calibration-aware optimizationSupported GPU serving stacksKernel, group size and source lineage matter
AWQActivation-aware weight quantizationSupported GPU serving stacksMethod name alone gives no quality or speed guarantee
FP88-bit floating-point formats and scaling schemesCompatible accelerator and framework pathsHardware, scale format and kernel matter
NVFP4NVIDIA-oriented 4-bit floating-point quantization schemeSupported recent NVIDIA serving pipelinesNot universal FP4; device and exact engine support required

GGUF may contain several llama.cpp tensor types, commonly named by patterns such as Q4 or Q5; a file extension does not state that every layer uses the same representation. GPTQ and AWQ models may use specialized serving kernels; look for exact version-specific instructions instead of copying a launch flag from another repository. FP8 and NVFP4 require particularly careful distinction between weight format, activation format and supported execution path.

The official vLLM method guide, Transformers documentation and llama.cpp quantization manual are better compatibility references than an unsourced “fastest format” chart.

Memory math: use it as a lower bound

For a hypothetical dense model with 8 billion parameters, raw BF16 weights need approximately 8B × 2 bytes = 16 GB in decimal units. Raw 8-bit weights suggest 8 GB; raw 4-bit weights suggest 4 GB. These are illustrative arithmetic values, not measurements of a particular downloadable artifact. Actual compressed tensors may have metadata, unquantized modules and conversion overhead. Mixture-of-experts checkpoints require reasoning about total stored weights, not just active parameters per token.

At inference time, allocate additional memory for the KV cache, activations, temporary workspaces, tokenizer/processor components and the serving runtime. Higher concurrency and long contexts can change KV-cache needs dramatically. Disk size, system RAM, GPU VRAM and Apple unified memory are distinct planning dimensions; a 4 GB weight file is not a promise of “4 GB VRAM required.” See the 16 GB GPU guide and hardware records for more evidence-aware sizing.

Do not confuse a quantized derivative with the base model

Quantization produces or modifies a deployable artifact; it does not create a new foundational checkpoint merely because the file has a different author, size or format. Record the upstream base model, exact source revision, quantization producer, published quantization configuration, tokenizer/processor pairing and derivative license. Do not assume a third-party upload inherits the original publisher’s verification record or its exact performance claims.

For an instruct-tuned model, output quality can depend on the chat template. GGUF conversion sometimes requires handling of special tokenizer and architecture metadata; multimodal components can need additional projector or processor files. If those artifacts are absent or incompatible, a weight file may load yet fail to implement the advertised use case. Our canonical profiles distinguish source-pinned original releases from deployment variants where evidence permits; verify re-exports independently.

A reproducible quantization evaluation protocol

Choose the same base model and prompt template, then compare a documented full-precision baseline with each supported quantization. Preserve prompt set, random seed (where honored), output length, context length, runtime version, device, batch size and concurrency. Evaluate numerical reasoning, instruction following, extraction, code correctness and your own domain’s critical tasks. Include cases sensitive to precision: exact identifiers, long tables, small numerical differences and structured JSON generation.

  1. Pin original and derived artifact identifiers, hashes and license terms; never mix base revisions.
  2. Confirm the serving engine explicitly supports the format on the actual hardware.
  3. Record cold load, steady-state memory, first-token latency and sustained throughput separately.
  4. Score a fixed held-out task set using an explicit rubric and review failures manually.
  5. Report the quality/runtime trade-off, plus unknown fields and any test exclusions.

Model quality loss is not uniform: a smaller quantized representation may appear acceptable on broad chat but damage a narrow classification or tool-use workflow. Conversely, a particular high-quality quantization may work well enough for the intended task. Do not claim either outcome until measured.

Common mistakes and our editorial view

The first mistake is treating storage reduction as an equal reduction in GPU requirements. The second is comparing throughput across different hardware and concurrency setups. The third is using “GGUF versus GPTQ” as though one were necessarily a superior algorithm: GGUF describes a container, whereas GPTQ specifies a quantization approach. The fourth is neglecting original license obligations and the derivative’s provenance. The fifth is assuming FP8 or NVFP4 support from a GPU brand rather than the specific device, kernel and backend.

Editorial judgment: “4-bit” is a feature label, not a purchasing or deployment decision. A useful open-weight intelligence platform should connect bit-width claims to exact files, engine compatibility and measured consequences. Until independent tests exist, the responsible conclusion is conditional—not a winner badge.

Frequently asked questions

Is GGUF the same thing as 4-bit quantization?

No. GGUF is a model file/container format that can represent different tensor types and quantization levels; 4-bit is a precision class, not a GGUF-specific algorithm.

Which is faster, AWQ or GPTQ?

Neither has a universal speed advantage. The result depends on the model, kernel, batch size, context, hardware, engine version and quantization configuration. Benchmark the exact pair.

Does a 4-bit 8B model require exactly 4 GB of GPU memory?

No. Four gigabytes is an idealized raw weight-only arithmetic estimate in decimal units. Scaling metadata, unquantized layers, KV cache and runtime allocations can increase the actual requirement.

Can I run NVFP4 on any NVIDIA GPU?

No. NVFP4 support is device-, kernel- and serving-stack-dependent. Consult the current vLLM or vendor compatibility matrix for the precise hardware and model format.

Does quantization change licensing?

It does not erase original weight-license obligations. A quantized derivative may also have separate provenance, redistribution conditions or additional components that must be reviewed.

Is FP8 always more accurate than 4-bit?

Higher nominal precision alone does not rank accuracy across different schemes. Scaling, calibration, layer exclusions, model architecture and task sensitivity matter; compare under controlled conditions.

Primary sources and editorial method

Sources below are original maintainers, original researchers or standards publishers. They document methods and capabilities, not tests independently performed by Open Model Weights. Our verification ledger, methodology and change history explain what we independently inspected in each canonical model record. The publication dates and versioned behavior of external tools can change; confirm the current upstream documentation before deployment.

Editorial boundary: This guide provides a reproducible decision framework, not a hands-on hardware benchmark, legal opinion or universal model ranking.