Open evaluation, reasoning and trustworthy agents

Allen AI's XYBench tests whether assistants solve the real problem

The openly documented 8,115-question benchmark tests whether AI helpers recognize a mistaken proposed solution and redirect toward the user's actual goal.

Open Model Weights published15 Nov 2026
Primary sourceAllen Institute for AI — XYBench: Helping with the problem behind the question
Source published2026-11-13
FOLLOW OUR COVERAGE

Follow Open Model Weights on Google See more of our open-weight AI reporting in Google Search.

Add as a Preferred Source on Google
Original high-gloss illustration of two reflective glass paths separating a misleading question from the actual user goal, with convergent teal solution lights.
Original high-gloss editorial visualization of the XY problem and agent decision paths; not a dataset plot or a benchmark measurement. AI-generated editorial illustration; not a documentary photograph.

The Allen Institute for AI has introduced XYBench, a benchmark for a subtle failure in assistants: answering the question someone asks even when that question assumes the wrong solution. The authors frame this as the XY problem. A user may request advice on improving one attempted fix, while the useful answer would first diagnose why that approach is unsuitable and suggest a better route to the underlying goal. That distinction matters for agents and local assistants intended to provide dependable practical help rather than simply follow literal instructions. Primary source ↗

The November 13 release describes 8,115 questions drawn from four sources: 1,557 Stack Overflow questions, 184 Stack Exchange items, 102 manually curated examples and 6,272 WikiHow-derived scenarios. The evaluation separates three behaviors: whether an answer offers a solution to the stated or actual problem, which of those receives emphasis, and whether the answer explicitly discourages a misconception. The public article links to the paper, dataset and code, providing a stronger audit trail than a headline score alone.

Researchers tested five systems, including the open-weight Qwen3.5-9B alongside proprietary alternatives, in a single-turn setting. According to the authors, models frequently mentioned a relevant remedy but still concentrated on the original mistaken approach. On Stack Overflow questions, their tested models focused on the true goal in only 35–41 percent of responses, compared with 47–57 percent focused on the stated request. These findings are the authors' experimental results, not a comparison independently rerun by Open Model Weights.

One methodological limit is especially important: Claude Sonnet 4.5 served as an automated judge, although two human reviewers also assessed 500 answers to check agreement. The task set covers English and is concentrated on procedural problems, so scores should not be generalized to all languages or conversational agent use. For our evidence layer, XYBench is a promising open evaluation input: model cards can link to version-pinned evaluations without confusing benchmark results with open checkpoint availability.

OMW REGISTRY WATCH

What this changes for the evidence layer

XYBench is a new evaluation resource, not a new open-weight model checkpoint. The public questions and evaluation procedure are useful for comparison, but published performance is author-reported and partly evaluated by another language model. Preserve exact dataset and judge revisions, model configuration and human-review methodology before treating comparative scores as independently reproduced.

PRIMARY SOURCE

Allen Institute for AI — XYBench: Helping with the problem behind the question

This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.

Open source article ↗