Open-Weight AI Models for 16 GB GPUs: What Fits?

DIRECT ANSWER

Which models are plausible candidates?

Small text models around 3–4 billion parameters are reasonable first candidates for a 16 GB card when the registry's BF16/FP16 weight-only estimate is below 12 decimal GB. Models around seven billion parameters are a tighter proposition at roughly 14.5 GB of raw weights; an eight-billion-parameter checkpoint can exceed 16 GB before runtime overhead. None of those numbers is measured peak VRAM. Quantized models may change the picture, but only a real compatible artifact and an actual run can establish fit.

We deliberately distinguish weight arithmetic, published formats, software compatibility and tested runtime behaviour. Open Model Weights has not run an independent 16 GB GPU inference campaign for these models.

A practical decision method

1. Start with total parameters. For dense models and mixture-of-experts models alike, estimate resident weight storage from total parameters, not “active experts” alone.
2. Choose a conservative weight budget. We use ≤12 decimal GB as a screening margin, not as a universal memory reserve or pass/fail threshold. The actual headroom differs by runtime, context, batching, device and KV-cache type.
3. Check the actual files and license. A theoretical four-bit size does not prove a publisher supplied an INT4 or GGUF version. Treat missing format evidence as unknown, not unsupported.
4. Test your precise setup. Pin checkpoint revision, runtime/build, quantization, prompt, context length and GPU; record load success, peak allocated/reserved GPU memory, TTFT and tokens/s.

For deeper filtering, use the hardware explorer and constraints-first fit tool, then inspect each canonical model profile.

ORIGINAL OMW DATABASE ANALYSIS

What the current registry actually shows

The aggregate figures below are recomputed by the build from data/models-source.json. Models lacking a numeric total-parameter value or verified available repository weights are excluded from the arithmetic cohorts; their status remains unknown. The table is an explicitly selected set of instructive records, not a performance ranking, and every row links to both our model profile and the original repository.

These are arithmetic screening counts, not confirmed 16 GB GPU support. Every count uses documented total parameters and a weight-only arithmetic estimate. INT4 here means a 4-bit arithmetic scenario, not a verified, downloadable INT4 checkpoint.

Interpretation: Qwen3-4B is a lower weight-only scenario than Qwen3-8B. Mistral-7B-Instruct-v0.3 is near the nominal 16 GB limit on BF16 weight arithmetic alone. A model in the “over 16 GB” row is not proof that it cannot run with quantization or offloading; it means the estimated BF16 weights alone cross that screening line.

How the estimates work — and what they omit

For a model with N total parameters, simple weight-only arithmetic is N × bytes per parameter. We use 2 bytes for BF16/FP16, 1 byte for 8-bit arithmetic and 0.5 byte for 4-bit arithmetic, then express the result in decimal GB (10⁹ bytes). This is a coarse estimate: actual file/storage formats, scales, zero-points, tensor alignment and mixed-precision components add details the arithmetic cannot capture.

VRAM is different. A running model needs a KV cache for retained tokens, inference buffers, kernels, temporary activations and allocator overhead. Batch size, context length and some model architectures change memory substantially. A GPU sold as “16 GB” may report capacity in GiB or another convention; a theoretical threshold must never be equated with a successful deployment.

Implementation details vary by serving stack. Official references include vLLM’s cache configuration and the Transformers quantization overview; neither is an OMW hardware benchmark.

In mixture-of-experts models, “active parameters” helps describe token computation, not the full set of checkpoint weights that must be stored somewhere. CPU offload and multi-GPU sharding are separate deployment configurations, not evidence of a single-GPU fit.

Deployment checklist — a protocol, not a claimed test

Before calling a candidate deployable, record the official checkpoint URL and immutable revision, exact files and precision, license basis, runtime and version, GPU SKU and driver, prompt/context, quantization settings, and observed memory, TTFT and throughput across repeat runs. No inference numbers are supplied here because Open Model Weights has not executed this benchmark.

For actual serving options and their untested source evidence, consult Compatibility, Private local agent systems and Measurements methodology. For license-dependent commercial deployment, see License Explorer.

Limitations and editorial judgment

The ≤12 GB threshold is an editorial screening heuristic, not a scientifically derived minimum buffer. The counts include many modalities and specialized components; they do not imply all listed weights are general-purpose chat models. They can also shift when source records or parameter estimates change. Source timestamps are not independent GPU measurements, and a repository file listing is not proof of inference support.

Our recommendation is to begin with the smaller, source-linked checkpoints, verify the exact quantized format and deployment constraints, and only then test a single 16 GB GPU. Favor complete evidence over superficially attractive “it fits” lists.

Questions developers ask

Sources and audit trail

The live table is derived from our source verification registry, not scraped benchmark listings. Each selected row links to the publisher's original checkpoint page. The verification methodology explains which fields were checked and where uncertainties remain. For source changes and data history see Changes and History.

Editorial revision: 2026-10-13. Dataset checks: per-model dates above. A new source check does not silently rewrite the editorial conclusions or imply a new hardware test.