The Allen Institute for AI has introduced XYBench, a benchmark for a subtle failure in assistants: answering the question someone asks even when that question assumes the wrong solution. The authors frame this as the XY problem. A user may request advice on improving one attempted fix, while the useful answer would first diagnose why that approach is unsuitable and suggest a better route to the underlying goal. That distinction matters for agents and local assistants intended to provide dependable practical help rather than simply follow literal instructions. Primary source ↗
The November 13 release describes 8,115 questions drawn from four sources: 1,557 Stack Overflow questions, 184 Stack Exchange items, 102 manually curated examples and 6,272 WikiHow-derived scenarios. The evaluation separates three behaviors: whether an answer offers a solution to the stated or actual problem, which of those receives emphasis, and whether the answer explicitly discourages a misconception. The public article links to the paper, dataset and code, providing a stronger audit trail than a headline score alone.
Researchers tested five systems, including the open-weight Qwen3.5-9B alongside proprietary alternatives, in a single-turn setting. According to the authors, models frequently mentioned a relevant remedy but still concentrated on the original mistaken approach. On Stack Overflow questions, their tested models focused on the true goal in only 35–41 percent of responses, compared with 47–57 percent focused on the stated request. These findings are the authors' experimental results, not a comparison independently rerun by Open Model Weights.
One methodological limit is especially important: Claude Sonnet 4.5 served as an automated judge, although two human reviewers also assessed 500 answers to check agreement. The task set covers English and is concentrated on procedural problems, so scores should not be generalized to all languages or conversational agent use. For our evidence layer, XYBench is a promising open evaluation input: model cards can link to version-pinned evaluations without confusing benchmark results with open checkpoint availability.
What this changes for the evidence layer
XYBench is a new evaluation resource, not a new open-weight model checkpoint. The public questions and evaluation procedure are useful for comparison, but published performance is author-reported and partly evaluated by another language model. Preserve exact dataset and judge revisions, model configuration and human-review methodology before treating comparative scores as independently reproduced.
Allen Institute for AI — XYBench: Helping with the problem behind the question
This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.
Open source article ↗