Open-weight model benchmarks, reliability and evidence

SO-Arena finds fast decision-model accuracy can hide unreliable choices

A new public benchmark evaluates 21 decision and reasoning systems across 11 domains and separately tests option-order stability and missing-answer recognition.

Open Model Weights published15 Nov 2026
Primary sourceLi et al. — SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models, arXiv
Source published2026-11-11
FOLLOW OUR COVERAGE

Follow Open Model Weights on Google See more of our open-weight AI reporting in Google Search.

Add as a Preferred Source on Google
Original high-gloss editorial 3D visualization of a chrome decision compass over a glass evaluation grid, with teal and amber pathways illustrating stable and unstable choices.
Original conceptual artwork of decision reliability and option stability; not a SO-Arena leaderboard chart or independent model evaluation. AI-generated editorial illustration; not a documentary photograph.

Researchers from the National University of Singapore have introduced SO-Arena, a benchmark designed for fast System One decision models rather than exclusively for long-form generative assistants. The November 11 preprint and accompanying project material describe 2,919 tasks covering 11 domains and five capability families, with 21 models in the reported comparison. Tasks include classification, evidence assessment, safety judgments, tool interactions and web-based actions. The distinguishing feature is that the evaluation asks not only whether a model produces a correct decision, but also whether that decision remains dependable. Primary source ↗

SO-Arena separates task capability from reliability. Its authors test option-order stability by changing how candidate answers are presented, and missing-answer recognition by removing the valid option. These tests matter for routing, retrieval, policy checks and agent action selection: a fast classifier may look strong on static examples yet fail when its choices are reordered or when every candidate is wrong. The benchmark also uses manually checked reference labels and interactive environments rather than relying solely on broad, text-generation leaderboard scores.

The study reports a striking gap between those dimensions. Among the System One models assessed, the highest capability performer ranks only seventh on the combined reliability measure. The authors also find that model scaling and extra reasoning produce very different improvements across problem types. These conclusions are based on their experimental setup, chosen prompts, model versions and task panels; Open Model Weights has not independently rerun the evaluation. In particular, a model's position in this benchmark should not be mistaken for a universal measure of safety or production suitability.

The research supplies a more useful framework for evidence-first model comparison. Rather than promoting one open-weight decision model based on a single overall score, users can examine whether the model holds its answer under harmless option permutations and declines to choose when none of the offered answers applies. Dataset provenance, applicable usage licenses and exact revisions must also remain visible. SO-Arena is therefore a notable new evaluation resource, not a newly downloadable model family, and it complements our reporting on specialised decision checkpoints.

OMW REGISTRY WATCH

What this changes for the evidence layer

SO-Arena is an evaluation dataset and methodology, not a newly released model checkpoint or independent Open Model Weights leaderboard. The preprint's results are reported by its authors; reproduce the public code and dataset at pinned revisions, examine the model versions and scoring panel, and check dataset-origin license restrictions before using scores in commercial comparisons. Preserve separate capability, option stability and missing-answer diagnostics instead of collapsing them into an unqualified model ranking.

PRIMARY SOURCE

Li et al. — SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models, arXiv

This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.

Open source article ↗