Open-model evaluation and long-context reliability

Ai2's StateBench exposes long-context failures in tracking changing information

A new evaluation asks whether language models can follow evolving entities and assignments, rather than simply retrieving a fact from a long input.

Open Model Weights published5 Nov 2026
Primary sourceAllen Institute for AI — StateBench original research article
Source published2026-11-03
FOLLOW OUR COVERAGE

Follow Open Model Weights on Google See more of our open-weight AI reporting in Google Search.

Add as a Preferred Source on Google
Original wide conceptual graphic titled STATEBENCH showing a four-stage abstract state machine with changing linked values; not a documentary image or independent performance chart.
Original Open Model Weights conceptual editorial artwork; not a documentary image or independent performance measurement. AI-generated editorial illustration; not a documentary photograph.

Allen Institute for AI has published StateBench, an evaluation designed to expose a failure hidden by many long-context benchmarks. A language model may retrieve an old fact correctly yet miss the later update that changes what the fact means. This matters when an agent reads revised instructions, follows an evolving software configuration or tracks entities in a long document. The right question is not merely whether the first mention is still retrievable, but whether the current state of each relevant object remains correct. Primary source ↗

StateBench introduces four levels of difficulty. R-trivial tasks use values that never change. Aperiodic problems allow later assignments to replace earlier ones. Soluble problems require updates based on a value's own previous state, such as incrementing a running total. General finite problems add assignments by reference, so changing one variable can affect later operations on another. These increasingly connected dependencies distinguish simple search from state tracking, and they let researchers diagnose why different model architectures succeed or fail.

Ai2 evaluates open pretrained checkpoints including Olmo 3 and Olmo Hybrid, as well as smaller transformer, recurrent and hybrid architectures trained specifically on these tasks. The authors find that controlled Gated DeltaNet and hybrid models extrapolate much farther on several update classes than their transformer counterparts. But favorable task-specific training does not automatically establish the behavior of a general pretrained assistant. In the study's frontier-model tests, accuracy on the longest general-finite sequences falls to 13 percent or below. These numbers belong to the paper's specific construction and have not been independently reproduced by Open Model Weights.

The practical lesson is to build evaluations where the world changes. A useful agent test should include edited tables, revised tool permissions, object handoffs, changing balances or revised deployment settings, then require the final state rather than a quoted source passage. StateBench announces a research benchmark, not a new downloadable foundation-model checkpoint. Its value is a sharper way to compare reliability while avoiding unsupported claims that one architecture or model is universally best.

OMW REGISTRY WATCH

What this changes for the evidence layer

StateBench is an evaluation framework, not a new downloadable checkpoint. Open Model Weights already tracks Olmo 3 model profiles; controlled state-tracking experiments must not be conflated with general model rankings.

PRIMARY SOURCE

Allen Institute for AI — StateBench original research article

This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.

Open source article ↗