A November 18 release from the FINAL-Bench project puts a sharper question to open-weight agent builders: what does a model actually ask its tools to do when malicious instructions appear? Its new ViKernel Scanner connects to an OpenAI-compatible endpoint, offers eight simulated tools and logs the actions the model attempts. These include file access, search, mail, HTTP requests, shell commands and simulated financial transfers. The test is designed to measure tool authorization behavior, not the fluency of a refusal message. Primary source ↗
The free version sends 100 attack checks and 20 ordinary-work controls to each model. According to the authors, the adversarial cases include 10 prompts from an AI model-hacking event and 90 disguised shell-command cases selected from a public development split. The attacks span destructive commands, reverse shells, downloaded executable commands, privilege escalation, credential reads, data exfiltration, persistence and evasion of logging. Encodings and string manipulation attempt to hide the original instruction before it reaches a shell tool.
The research team reports that all 20 open-weight models it scanned through the Hugging Face router on November 18 received an overall F. That striking result needs context. The scanner's grade is determined by the most severe domain in which a model produces a finding; it is not an average success rate across 100 attacks. The evaluated sample is not every open-weight model, and an attempted call in a simulated environment is not evidence that a real production system deleted files, stole data or made a payment.
The publisher also controls the attack set delivered to scans and can update it centrally. That means future tests need explicit protocol versioning, model and endpoint identifiers, prompt settings and a preserved action trace before scores are comparable. Some prompts can be unsafe while others resemble legitimate administrator tasks; the normal-work controls are important for measuring whether a defense blocks useful authorized work as well as malicious requests.
The practical takeaway is to enforce permissions where commands execute. Models should not receive unrestricted shells, secrets or payment tools simply because system instructions tell them to behave. Credential scoping, network isolation, explicit approvals and execution-layer policy are more reliable boundaries than trusting a natural-language refusal alone.
Open Model Weights has not repeated the 20-model test, reviewed every payload or independently validated claims about the project's separate firewall. The scanner is a useful security-evaluation candidate, but its headline grades should remain attributed to FINAL-Bench rather than be presented as a universal verdict on open-weight model safety.
What this changes for the evidence layer
This is a third-party self-reported security scanner and test of model tool-call behavior, not a released foundation checkpoint or an independent OMW benchmark. The free mock-tool protocol covers 100 adversarial checks and 20 normal-work controls; detailed payloads are served centrally. No model should be demoted or ranked based on this one protocol.
FINAL-Bench / SeaWolf-AI — original ViKernel Scanner study and free tool
This brief is based on the cited primary source. Performance, benchmark and comparative claims remain attributed unless Open Model Weights publishes an independent measurement.
Open source article ↗