Publisher-reported results
Scores, latency and capability statements published by a model creator or platform. Useful as attributed context, but not an Open Model Weights measurement.
Read source policy →OPEN MODEL WEIGHTS · INDEPENDENT EVIDENCE
A practical guide to model deployment performance, reproducible measurements, real hardware requirements and the evidence needed to compare open-weight models responsibly.
Reliable model benchmarks require the same workload and documented execution conditions across comparable runs. Open Model Weights currently provides independently checked Safetensors artifact observations and a deployment test protocol; it has 0 accepted runtime benchmark results. Weight-memory estimates and publisher scores are not substitutes for measured inference speed or task quality.
THREE DIFFERENT KINDS OF NUMBERS
Benchmark claims are only useful when their origin and scope are visible. Our data separates published claims, calculated estimates and observations from executed independent protocols. A change in evidence class is not a new publisher fact.
Scores, latency and capability statements published by a model creator or platform. Useful as attributed context, but not an Open Model Weights measurement.
Read source policy →Approximate bytes needed to store parameters at a specified precision, calculated from documented model data. They exclude KV cache, activations and runtime overhead.
Explore model memory →Reproducible Safetensors header inspections bound to exact artifact and repository revisions. These confirm file structure, tensor counts and parameters – not inference performance.
Inspect measured evidence →REAL OBSERVATIONS · PINNED ARTIFACTS
Not a model intelligence leaderboard.
The chart below displays actual parameter counts read from Safetensors headers for five pinned repository artifacts. These are field-level observations from our existing measurement records. No output-token/s, time-to-first-token, or GPU-speed result is implied.
Bars compare Safetensors-header parameter counts for these five pinned artifacts only. Model modalities differ; parameter count is not a performance, quality, speed or hardware-fit ranking. Inspect measurement details and exact method →
The accepted-results API currently lists zero reproducible inference runs. We will display throughput, latency and peak memory here only after hardware, runtime, input workload, raw commands and public run evidence pass validation. The absence of results is not evidence that any model is slow, fast or unsuitable.
MEASURED PERFORMANCE · EXPLAINED
Unit: milliseconds. How long a request takes to produce its first output token; includes prefill and startup effects. Lower is usually preferable under an identical workload.
Unit: tokens/s. Decode throughput, not intelligence or accuracy. Compare the same concurrency, prompt sizes, decoding setup and output lengths.
Unit: GB. Observed GPU memory during execution, including allocations beyond model weights. It is different from our weight-only storage estimates.
Unit: GB. CPU-side memory peak, when recorded. Missing measurements remain missing, not zero.
Other benchmark families: task accuracy, code evaluations, retrieval quality, safety and agent behavior need their own datasets, prompts and scoring methods. They must never be collapsed into a single universal score.
REPRODUCIBLE BENCHMARK PROTOCOL · VERSION 1.0.0
Our public deployment protocol specifies two warmup runs, five measured runs, concurrency one, temperature zero and 128 maximum output tokens. Report the median, but retain individual run records and raw configurations. Those defaults do not make unlike workloads equivalent.
Capture the repository SHA, actual weight-file or quantization variant, model configuration and licensing source. A name alone is insufficient.
Record GPU/CPU vendor and model, device count, memory, operating system, driver versions, runtime name and version, and the literal execution command.
Declare input/output token counts, request concurrency, decoding parameters, context lengths and cold/warm state. Do not blend runs with different settings.
Perform warmups before recording TTFT, sustained tokens/s and peak device memory. Keep per-run values and derive the stated median without extrapolation.
Link raw run evidence and validate it against the result JSON schema. Only then can a result appear in the accepted-results feed.
FAIR COMPARISONS
Use the same hardware class, runtime version, batch/concurrency profile, quantization and prompt distribution. Record the model revision and test date, not just a model label.
Differences in tokenization, context length and implementation can dominate observed performance. Multiple task types may also favor different systems.
A model with a small INT4 derived weight estimate has not been proven to run efficiently in 8 GB of VRAM. A publisher-reported benchmark is not a result we independently reproduced.
A community-submitted measurement should be identified by provenance and validation status. An unverified result remains unaccepted.
For additional perspectives, Artificial Analysis publishes intelligence, speed and pricing comparisons; OpenRouter documents model-usage rankings for its own platform; and the Open LLM Leaderboard maintains evaluation assets. Their populations, methods and claims are not interchangeable with our source-verification or independent artifact measurements.
MODEL SELECTION · PRACTICAL WORKFLOW
Use our hardware explorer to estimate weight storage; add headroom for KV cache, activations and serving overhead.
Read license evidence, exact formats and accessible model artifacts before selecting a deployment path.
Compare observed revisions and source comparison signals; unknown must not become false.
Measure task outputs, latency and hardware memory in your environment before a production commitment. We do not infer that outcome from metadata alone.
Next: compare 2–4 source-verified model records · view accepted independent artifact observations · read verification methodology.
FREQUENTLY ASKED QUESTIONS
A benchmark measures a specified workload under a known evaluation or deployment setup. Deployment benchmarks may measure time to first token, output throughput and memory; task-quality evaluations require different datasets and scoring procedures.
No. Hardware explorer numbers are derived weight-storage estimates based on documented parameters and precision assumptions. They exclude KV cache, activations, runtime allocations and hardware-specific inference behavior.
Yes, artifact-level Safetensors header observations exist for pinned revisions. No accepted inference-speed or task-quality runtime benchmark results are currently present. These categories are explicitly separated.
Keep model revision, quantization, runtime version, device, batch or concurrency, input tokens and output tokens controlled. Compare per-run distributions and report setup changes rather than blending results from different hardware.
Time to first token includes request preparation and prefill before the first generated token. Sustained tokens per second describes decoding throughput after generation starts. A system can excel at one and not the other.
Inspect exact workload relevance, model and weight revision, runtime command, hardware, measured-run distribution and raw evidence. Do not treat a vendor-reported score or a weight-only estimate as an independently measured deployment result.
A candidate requires an exact source revision, quantization or artifact, runtime and version, hardware and driver details, reproducible commands, measured metrics and public run evidence. Only validated results are eligible for the accepted-results dataset.
Our benchmark protocol is public, accepted runtime results are transparently marked as unavailable, and existing independently observed artifact data is inspectable. As reproducible deployment results become available, the same source-linked record will show exact hardware and workload scope.