How Open Are Open-Weight AI Models, Really? A Data-Driven Transparency Audit

DIRECT ANSWER

Accessible model weights are only one dimension of openness

An open-weight model can make pretrained or instruction-tuned parameters available while disclosing very little about its training corpus, exact training process, full source code, evaluation conditions or downstream rights. In the Open Model Weights registry, all included records have verified-available weights by selection design. That is not evidence that all included models are fully open-source AI or independently reproducible.

To make the distinction actionable, this audit uses our existing model registry to separate observable repository artifacts, licensing, training-method descriptions, data-disclosure signals and actual reproduction evidence. We avoid a single opaque openness score. Each dimension answers a different question and requires different proof.

Method: field evidence, not marketing labels

Our unit of analysis is the exact model record, linked to a publisher repository and retained source observations. The scheduled verifier checks public model metadata, file listings, model cards and configurations where available. Its field statuses are descriptive. A field classified as not disclosed in the checked sections must not be interpreted as a proven absence across all source material.

1. Weights: does the registry contain a verified public repository and an exact weight-file listing? Availability is not identical to redistribution freedom.
2. License evidence: is a license name declared, is a repository license file recorded, and is commercial use classified with a basis? These are separate checks; metadata is not legal advice.
3. Training recipe: did the checked card describe any training method? A description alone is not a complete runnable recipe or enough information to reproduce the model.
4. Data transparency: are specific dataset metadata or a description visible? A dataset link is not proof of complete training data access, provenance, consent or completeness.
5. Reproducibility: are scripts, hashes, environments, training compute, data snapshots and evaluations sufficient for someone else to repeat the result? We do not assign this outcome without an actual audit or reproduction campaign.

For terminology, the Open Source Initiative’s AI definition provides an external standard to consult. Our audit does not automatically classify a model as satisfying that definition merely because its weights are downloadable.

ORIGINAL REGISTRY ANALYSIS · NO COMPOSITE SCORE

What the verified registry documents

The following coverage counters and examples are generated directly from data/models-source.json at build time. We do not keep a second model catalog for articles. The observed-at dates are source-check timestamps, not editorial revision dates or proof of a fresh manual review of every training statement.

Documented does not mean reproducible. The metrics count what the existing verifier observed. They do not grade documentation quality, confirm accessible training datasets, or prove faithful retraining. Other signals: 716 weight-file listings; 572 accessible configuration files; 294 with data described in model cards; 292 without a disclosure signal in checked sections; 193 commercial-use classifications remain unknown.

The coverage signals are deliberately non-exclusive: a record can have both a readable model card and no reported training dataset in the fields we checked. A repository with no separately detected LICENSE file may still present operative terms elsewhere; the audit makes that distinction visible. The table illustrates heterogeneous cases: for example, an Olmo model with dataset metadata, a Qwen model without the same recorded training-data signal, and a Gemma model with license-specific terms that deserve separate review. No model is ranked as better or worse by these proxies.

How to run an openness review before deployment

1. Pin the model: record the canonical OMW profile, upstream repository, immutable commit/revision and the required files. “Latest” is not a reproducible version identifier.
2. Read the terms: inspect the license file, attached usage policy and relevant downstream restrictions. If commercial use is unknown in our classification, do not silently promote it to permitted.
3. Trace training claims: follow named datasets, code repositories and paper links. Note which are available as exact versions, which are summarized and which remain unknown.
4. Inspect evaluation provenance: separate first-party benchmark claims from independently repeated results; document dataset version, leakage caveats, prompts and hardware.
5. Record an evidence decision: keep per-field findings with source URLs and observation dates; avoid collapsing a scientific reproducibility judgment into a license checkbox.

A concrete team checklist should include a reproducible bill of materials (model file hashes, tokenizer, configs, adapters and runtime), the applicable legal review, safety evaluation, data-protection controls and a clear statement of what was not checked. This is a review protocol, not a report that our team independently retrained the models in this table.

Limits of this analysis

This cohort is a curated source-verified open-weight registry, not a statistical census of every model published online. “Training described” does not mean complete recipe; “dataset metadata declared” does not mean the complete corpus is downloadable; “model card accessible” does not mean the card is accurate or comprehensive. Repository snapshots can change. Our runtime evidence remains source-derived and is not automatically a successful deployment test. Licensing classifications are scoped, not legal conclusions.

Because these dimensions are incommensurable, we intentionally publish counts and links instead of a weighted model-openness leaderboard. For compliance and operational decisions, consult the source files and verify the exact revision on which your deployment depends.

Sources, scope and next steps

The numbers come from the existing verification registry and evidence methodology. Every sample record above links to an official Hugging Face repository. For changes, see History, Source policy and License Explorer. The OSI Open Source AI Definition is external definitional context, not an automatically awarded certification.

Editorially updated 13 October 2026. Model-data observation dates are displayed separately in the generated table. No original training or benchmark reruns were performed for this article.

Frequently asked questions