Researchers from the National University of Singapore have introduced SO-Arena, a benchmark designed for fast System One decision models rather than exclusively for long-form generative assistants. The November 11 preprint and accompanying project material describe 2,919 tasks covering 11 domains and five capability families, with 21 models in the reported comparison. Tasks include classification, evidence assessment, safety judgments, tool interactions and web-based actions. The distinguishing feature is that the evaluation asks not only whether a model produces a correct decision, but also whether that decision remains dependable. Primary source ↗
SO-Arena separates task capability from reliability. Its authors test option-order stability by changing how candidate answers are presented, and missing-answer recognition by removing the valid option. These tests matter for routing, retrieval, policy checks and agent action selection: a fast classifier may look strong on static examples yet fail when its choices are reordered or when every candidate is wrong. The benchmark also uses manually checked reference labels and interactive environments rather than relying solely on broad, text-generation leaderboard scores.
The study reports a striking gap between those dimensions. Among the System One models assessed, the highest capability performer ranks only seventh on the combined reliability measure. The authors also find that model scaling and extra reasoning produce very different improvements across problem types. These conclusions are based on their experimental setup, chosen prompts, model versions and task panels; Open Model Weights has not independently rerun the evaluation. In particular, a model's position in this benchmark should not be mistaken for a universal measure of safety or production suitability.
The research supplies a more useful framework for evidence-first model comparison. Rather than promoting one open-weight decision model based on a single overall score, users can examine whether the model holds its answer under harmless option permutations and declines to choose when none of the offered answers applies. Dataset provenance, applicable usage licenses and exact revisions must also remain visible. SO-Arena is therefore a notable new evaluation resource, not a newly downloadable model family, and it complements our reporting on specialised decision checkpoints.
What this changes for the evidence layer
SO-Arena is an evaluation dataset and methodology, not a newly released model checkpoint or independent Open Model Weights leaderboard. The preprint's results are reported by its authors; reproduce the public code and dataset at pinned revisions, examine the model versions and scoring panel, and check dataset-origin license restrictions before using scores in commercial comparisons. Preserve separate capability, option stability and missing-answer diagnostics instead of collapsing them into an unqualified model ranking.
Li et al. — SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models, arXiv
This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.
Open source article ↗