Open Model WeightsAGENT INTELLIGENCE · SOURCE-FIRST
Model-specific source claims and reported task results, separated from independent measurements.
Among 31 Leading and Notable catalog records, 7 unique models have newly reviewed agent-related publisher statements, 7 reported benchmark observations are attributed to model publishers, and 7 runtime-related references are linked. Zero independent agent runtime benchmarks have been run by Open Model Weights. Absence of an entry is unknown, not evidence of failure.
These author-reported numbers come from different tasks, model versions and sometimes incompletely specified harnesses. This table is not a cross-model leaderboard and does not justify normalizing scores or comparing different benchmarks.
ATTRIBUTED NUMBERS
| Model | Benchmark | Reported result | Evaluator and protocol | Primary source |
|---|---|---|---|---|
| Mellum2.1-12B-A2.5B-Thinking | BFCL v4 | 62.3 percent | Publisher-reported by JetBrains JetBrains evaluation pipeline; BFCL macro-average v1/v2/v3/web-search/memory | Original source ↗ |
| Mellum2.1-12B-A2.5B-Thinking | SWE-bench Verified | 47 percent | Publisher-reported by JetBrains Pi v0.73.1 agent, shell/file tools; 114K context; up to 16K tokens per turn | Original source ↗ |
| Mellum2.1-12B-A2.5B-Thinking | Terminal-Bench 2.1 | 17.4 percent | Publisher-reported by JetBrains Pi v0.73.1 agent, shell/file tools; 114K context; up to 16K tokens per turn | Original source ↗ |
| K2-Horizon-375B-A23B | tau3-Banking | 34 reported score | Publisher-reported by IFM Not fully independently reconstructed by OMW | Original source ↗ |
| K2-Horizon-375B-A23B | Toolathlon Verified | 65.3 reported score | Publisher-reported by IFM Not fully independently reconstructed by OMW | Original source ↗ |
| K2-Horizon-375B-A23B | MCPMark | 67.7 reported score | Publisher-reported by IFM Not fully independently reconstructed by OMW | Original source ↗ |
| Holo4-27B | OSWorld 2.0 | 61.7 percent | Publisher-reported by H Company H Company's released Holo4 agent harness and reported OSWorld 2.0 evaluation; model revision, prompt budget and all environmental variants not independently reproduced | Original source ↗ |
The publisher reports this number; OMW has not run BFCL v4. No comparable result is available here for other curated models.
Macro-average of five BFCL subtasks · Model revision: not pinned to a reproducible run.
This is a vendor evaluation in its specified Pi harness, not an OMW run or a directly transferable deployment result.
Resolved rate reported by JetBrains · Model revision: not pinned to a reproducible run.
Same author-provided benchmark family; not comparable to other tasks or systems without aligned protocol.
Success rate reported by JetBrains · Model revision: not pinned to a reproducible run.
Model-card table only. Treat metric protocol and exact model revision as unpinned pending deeper verification.
Publisher-reported tau3-Banking result · Model revision: not pinned to a reproducible run.
Model-card table only. Do not rank against BFCL or other benchmark versions.
Publisher-reported agentic tool-use result · Model revision: not pinned to a reproducible run.
Model-card table only; no OMW MCPMark run. Results cannot be pooled across benchmarks.
Publisher-reported MCP tool-use result · Model revision: not pinned to a reproducible run.
Not independently rerun by OMW; H Company explicitly notes other leaderboards may use different task subsets/harnesses. Do not treat as a cross-model ranking.
Publisher-reported benchmark success · Model revision: not pinned to a reproducible run.
CAPABILITY EVIDENCE
Tool calling · Publisher documentation. JetBrains documents tool use and agent training in sandboxed software repositories, plus an exact vLLM tool parser configuration.
Documents design and supported interface, not a reproduced OMW tool-success rate.
Coding agents · Publisher documentation. JetBrains describes repository, shell and file-editing reinforcement learning and reports software engineering evaluations.
Publisher description and self-reported evaluation; task outcomes depend on the agent harness.
Tool calling · Publisher documentation. OpenAI documents native function calling, structured outputs and the required Harmony response format for the gpt-oss models.
Supported model behavior is documented; no OMW tool-call accuracy or full agent performance is measured.
Tool calling · Publisher documentation. Mistral's first-party model card documents native function calling and JSON output and provides a vLLM command with the Mistral tool parser.
Native interface is publisher-documented; cannot infer function-call success on a specific task.
Agentic tool use · Publisher documentation. The IFM model card publishes its own evaluations on agentic tool-use and long-horizon workflows.
Publisher-reported evaluations; do not convert these scores into verified runtime compatibility or general tool-call guarantees.
Computer use · Publisher documentation. H Company documents screenshot-guided GUI interaction, code execution and API/MCP tool use via its hai-agents harness.
Official publisher assertion only; OMW did not independently evaluate this checkpoint/harness.
Tool calling · Publisher documentation. The Holo4-27B publisher documents function calling and tool-driven desktop/browser workflows; exact test success is not independently measured by OMW.
Official publisher assertion only; OMW did not independently evaluate this checkpoint/harness.
Tool calling · Publisher documentation. IBM Granite 4.2-30B model card documents native tool-calling behavior, reasoning modes and agentic training, with vLLM-compatible API guidance.
Official publisher assertion only; OMW did not independently evaluate this checkpoint/harness.
Document parsing · Publisher documentation. LightOn documents transcription plus grounding output with region labels, normalized bounding boxes, image descriptions and chart data. It is an ingestion component, not evidence of autonomous agent tool planning.
Official publisher assertion only; OMW did not independently evaluate this checkpoint/harness.
SERVING INTERFACE
publisher setup documented
Exact model card command, not executed by OMW. Requires substantial VRAM and version compatibility.
Original source ↗publisher setup documented
Publisher's full serving command; not an OMW runtime test. Hardware and installed version must be validated.
Original source ↗runtime catalog documented
Official Ollama library serves its own MXFP4 build; do not equate its artifact hash or memory requirement with the Hugging Face weight file.
Original source ↗runtime release documented
Runtime release mentions support. Availability of an exact, validated GGUF checkpoint and tool template is not established.
Original source ↗publisher setup documented
Hugging Face displays a model serve example. This does not confirm function-call parsing or complete computer-use harness setup; OMW has not run it.
Original source ↗publisher setup documented
Official Hugging Face serving example; not an OMW-measured end-to-end function-call configuration.
Original source ↗publisher setup documented
Publisher documented with model-specific no-thinking template; do not deploy --trust-remote-code without code review, pinning and sandboxing. No OMW runtime run.
Original source ↗For every numerical agent result, keep the exact model artifact revision, serving runtime and template, benchmark version, evaluation prompts, tool environment, sampling, evaluation code commit and raw outcomes. Without that chain, this site labels results as publisher-reported, never independent.
We provide a bounded localhost-only tool-call smoke-test protocol and a constraint-aware selector. They are not substitutes for a production benchmark.
DIRECT ANSWERS
No. Every numerical result on this page is explicitly attributed to its model publisher; Open Model Weights has not independently reproduced any of them.
No. BFCL, SWE-bench, Terminal-Bench, tau3, Toolathlon and MCPMark differ in task design, scoring and execution environment.
No. Missing model- or version-specific evidence is unknown rather than a negative test.
Documented interfaces and publisher-run benchmarks do not certify autonomous production deployments.
Open the constraint-aware selector → · Read our methodology →