Open-weight agent safety, tool authorization and evaluation reliability

ViKernel Scanner reports unsafe tool calls across 20 open-weight models

A new free scanner subjects open models to mock file, shell and payment tools. Its 20 failure grades raise a deployment question, not a universal verdict on every open checkpoint.

Open Model Weights published18 Nov 2026
Primary sourceFINAL-Bench / SeaWolf-AI — original ViKernel Scanner study and free tool
Source published2026-11-18
FOLLOW OUR COVERAGE

Follow Open Model Weights on Google See more of our open-weight AI reporting in Google Search.

Add as a Preferred Source on Google
Original minimalist editorial illustration representing benchmark in AI research; conceptual, not actual model output.
AI-generated editorial illustration · Not official model imagery.

A November 18 release from the FINAL-Bench project puts a sharper question to open-weight agent builders: what does a model actually ask its tools to do when malicious instructions appear? Its new ViKernel Scanner connects to an OpenAI-compatible endpoint, offers eight simulated tools and logs the actions the model attempts. These include file access, search, mail, HTTP requests, shell commands and simulated financial transfers. The test is designed to measure tool authorization behavior, not the fluency of a refusal message. Primary source ↗

The free version sends 100 attack checks and 20 ordinary-work controls to each model. According to the authors, the adversarial cases include 10 prompts from an AI model-hacking event and 90 disguised shell-command cases selected from a public development split. The attacks span destructive commands, reverse shells, downloaded executable commands, privilege escalation, credential reads, data exfiltration, persistence and evasion of logging. Encodings and string manipulation attempt to hide the original instruction before it reaches a shell tool.

The research team reports that all 20 open-weight models it scanned through the Hugging Face router on November 18 received an overall F. That striking result needs context. The scanner's grade is determined by the most severe domain in which a model produces a finding; it is not an average success rate across 100 attacks. The evaluated sample is not every open-weight model, and an attempted call in a simulated environment is not evidence that a real production system deleted files, stole data or made a payment.

The publisher also controls the attack set delivered to scans and can update it centrally. That means future tests need explicit protocol versioning, model and endpoint identifiers, prompt settings and a preserved action trace before scores are comparable. Some prompts can be unsafe while others resemble legitimate administrator tasks; the normal-work controls are important for measuring whether a defense blocks useful authorized work as well as malicious requests.

The practical takeaway is to enforce permissions where commands execute. Models should not receive unrestricted shells, secrets or payment tools simply because system instructions tell them to behave. Credential scoping, network isolation, explicit approvals and execution-layer policy are more reliable boundaries than trusting a natural-language refusal alone.

Open Model Weights has not repeated the 20-model test, reviewed every payload or independently validated claims about the project's separate firewall. The scanner is a useful security-evaluation candidate, but its headline grades should remain attributed to FINAL-Bench rather than be presented as a universal verdict on open-weight model safety.

OMW REGISTRY WATCH

What this changes for the evidence layer

This is a third-party self-reported security scanner and test of model tool-call behavior, not a released foundation checkpoint or an independent OMW benchmark. The free mock-tool protocol covers 100 adversarial checks and 20 normal-work controls; detailed payloads are served centrally. No model should be demoted or ranked based on this one protocol.

PRIMARY SOURCE

FINAL-Bench / SeaWolf-AI — original ViKernel Scanner study and free tool

This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.

Open source article ↗