OPEN MODEL WEIGHTS · INDEPENDENT EVIDENCE

Open-weight AI benchmarks, without the guesswork.

A practical guide to model deployment performance, reproducible measurements, real hardware requirements and the evidence needed to compare open-weight models responsibly.

DIRECT ANSWER

Reliable model benchmarks require the same workload and documented execution conditions across comparable runs. Open Model Weights currently provides independently checked Safetensors artifact observations and a deployment test protocol; it has 0 accepted runtime benchmark results. Weight-memory estimates and publisher scores are not substitutes for measured inference speed or task quality.

THREE DIFFERENT KINDS OF NUMBERS

What is independently measured – and what is not?

Benchmark claims are only useful when their origin and scope are visible. Our data separates published claims, calculated estimates and observations from executed independent protocols. A change in evidence class is not a new publisher fact.

01 · SOURCE CLAIM

Publisher-reported results

Scores, latency and capability statements published by a model creator or platform. Useful as attributed context, but not an Open Model Weights measurement.

Read source policy →
02 · OMW DERIVED

Weight-memory estimates

Approximate bytes needed to store parameters at a specified precision, calculated from documented model data. They exclude KV cache, activations and runtime overhead.

Explore model memory →
03 · OMW MEASURED

Executed artifact observations

Reproducible Safetensors header inspections bound to exact artifact and repository revisions. These confirm file structure, tensor counts and parameters – not inference performance.

Inspect measured evidence →
Publisher source→Source verification→Scoped independent run→Public evidence artifact

REAL OBSERVATIONS · PINNED ARTIFACTS

Our independently recorded measurements.

Not a model intelligence leaderboard.

The chart below displays actual parameter counts read from Safetensors headers for five pinned repository artifacts. These are field-level observations from our existing measurement records. No output-token/s, time-to-first-token, or GPU-speed result is implied.

0Accepted runtime benchmark runs
10Pinned artifact targets
30Independent artifact observations

Bars compare Safetensors-header parameter counts for these five pinned artifacts only. Model modalities differ; parameter count is not a performance, quality, speed or hardware-fit ranking. Inspect measurement details and exact method →

Runtime results are not yet available.

The accepted-results API currently lists zero reproducible inference runs. We will display throughput, latency and peak memory here only after hardware, runtime, input workload, raw commands and public run evidence pass validation. The absence of results is not evidence that any model is slow, fast or unsuitable.

MEASURED PERFORMANCE · EXPLAINED

Four metrics that answer different questions.

TTFT

Time to first token

Unit: milliseconds. How long a request takes to produce its first output token; includes prefill and startup effects. Lower is usually preferable under an identical workload.

TPS

Output tokens per second

Unit: tokens/s. Decode throughput, not intelligence or accuracy. Compare the same concurrency, prompt sizes, decoding setup and output lengths.

VRAM

Peak device memory

Unit: GB. Observed GPU memory during execution, including allocations beyond model weights. It is different from our weight-only storage estimates.

HOST

Peak host memory

Unit: GB. CPU-side memory peak, when recorded. Missing measurements remain missing, not zero.

Other benchmark families: task accuracy, code evaluations, retrieval quality, safety and agent behavior need their own datasets, prompts and scoring methods. They must never be collapsed into a single universal score.

REPRODUCIBLE BENCHMARK PROTOCOL · VERSION 1.0.0

What makes two runs comparable?

Our public deployment protocol specifies two warmup runs, five measured runs, concurrency one, temperature zero and 128 maximum output tokens. Report the median, but retain individual run records and raw configurations. Those defaults do not make unlike workloads equivalent.

01

Pin the exact model

Capture the repository SHA, actual weight-file or quantization variant, model configuration and licensing source. A name alone is insufficient.

02

Document the environment

Record GPU/CPU vendor and model, device count, memory, operating system, driver versions, runtime name and version, and the literal execution command.

03

Control the workload

Declare input/output token counts, request concurrency, decoding parameters, context lengths and cold/warm state. Do not blend runs with different settings.

04

Measure and retain raw runs

Perform warmups before recording TTFT, sustained tokens/s and peak device memory. Keep per-run values and derive the stated median without extrapolation.

05

Publish inspectable provenance

Link raw run evidence and validate it against the result JSON schema. Only then can a result appear in the accepted-results feed.

FAIR COMPARISONS

Why a faster score is not automatically a better model.

Compare like with like

Use the same hardware class, runtime version, batch/concurrency profile, quantization and prompt distribution. Record the model revision and test date, not just a model label.

Differences in tokenization, context length and implementation can dominate observed performance. Multiple task types may also favor different systems.

Separate every evidence category

A model with a small INT4 derived weight estimate has not been proven to run efficiently in 8 GB of VRAM. A publisher-reported benchmark is not a result we independently reproduced.

A community-submitted measurement should be identified by provenance and validation status. An unverified result remains unaccepted.

Related comparison resources

For additional perspectives, Artificial Analysis publishes intelligence, speed and pricing comparisons; OpenRouter documents model-usage rankings for its own platform; and the Open LLM Leaderboard maintains evaluation assets. Their populations, methods and claims are not interchangeable with our source-verification or independent artifact measurements.

MODEL SELECTION · PRACTICAL WORKFLOW

Start from deployment constraints, not a universal leaderboard.

1

Identify available memory

Use our hardware explorer to estimate weight storage; add headroom for KV cache, activations and serving overhead.

2

Inspect weights and licensing

Read license evidence, exact formats and accessible model artifacts before selecting a deployment path.

4

Run your real workload

Measure task outputs, latency and hardware memory in your environment before a production commitment. We do not infer that outcome from metadata alone.

Next: compare 2–4 source-verified model records · view accepted independent artifact observations · read verification methodology.

FREQUENTLY ASKED QUESTIONS

Benchmark questions, answered directly.

What does a model benchmark actually measure?

A benchmark measures a specified workload under a known evaluation or deployment setup. Deployment benchmarks may measure time to first token, output throughput and memory; task-quality evaluations require different datasets and scoring procedures.

Are Open Model Weights memory estimates benchmarks?

No. Hardware explorer numbers are derived weight-storage estimates based on documented parameters and precision assumptions. They exclude KV cache, activations, runtime allocations and hardware-specific inference behavior.

Does Open Model Weights publish independent results today?

Yes, artifact-level Safetensors header observations exist for pinned revisions. No accepted inference-speed or task-quality runtime benchmark results are currently present. These categories are explicitly separated.

How can I compare tokens per second fairly?

Keep model revision, quantization, runtime version, device, batch or concurrency, input tokens and output tokens controlled. Compare per-run distributions and report setup changes rather than blending results from different hardware.

Why can time to first token differ from throughput?

Time to first token includes request preparation and prefill before the first generated token. Sustained tokens per second describes decoding throughput after generation starts. A system can excel at one and not the other.

Which benchmark should I trust for commercial deployment?

Inspect exact workload relevance, model and weight revision, runtime command, hardware, measured-run distribution and raw evidence. Do not treat a vendor-reported score or a weight-only estimate as an independently measured deployment result.

How are new results accepted?

A candidate requires an exact source revision, quantization or artifact, runtime and version, hardware and driver details, reproducible commands, measured metrics and public run evidence. Only validated results are eligible for the accepted-results dataset.

Evidence first. Performance claims only after actual runs.

Our benchmark protocol is public, accepted runtime results are transparently marked as unavailable, and existing independently observed artifact data is inspectable. As reproducible deployment results become available, the same source-linked record will show exact hardware and workload scope.