Deploy Agent Models with vLLM

vLLM can serve compatible open-weight checkpoints through an OpenAI-style endpoint, but auto tool calling needs a model-specific parser and matching tool-aware chat template.

Four steps to a reproducible deployment

Verify model support
Check the exact checkpoint architecture, deployment requirements and vLLM release; repository availability alone does not prove compatibility.
Configure tool selection
When suitable, enable --enable-auto-tool-choice and choose --tool-call-parser for that specific model. A compatible chat template may also be required.
Constrain and evaluate
Test strict tool arguments, concurrency, failure recovery and authorization. Tool extraction and model fidelity are separate layers.
Plan production controls
Use isolated execution workers, observability, authentication, resource caps and version-pinned deployment manifests.

Minimal acceptance tests

Record the repository revision, artifact checksum, runtime version and command line in a deployment manifest. Verify ordinary chat, a negative permission test, malformed tool arguments and model recovery. Measure actual memory and latency on the target hardware rather than extrapolating from a weight-only estimate.

No specific one of the 28 curated checkpoints is certified to run in this stack by this page. See the agent model explorer for source-linked candidates.

COMMON QUESTIONS

Frequently asked questions

Can any listed checkpoint be deployed with vLLM?

Not necessarily. Runtime compatibility, weight format and chat templates must be checked for the exact checkpoint.

Does local inference make tool execution safe?

No. Tool permissions, sandboxing, network policy, logging and human approval remain application responsibilities.

READ THE ORIGINALS

Documentation and primary sources

COPYABLE LOCAL RECIPE · NOT YET REPRODUCED BY OMW

Example setup and acceptance check

JetBrains supplies a model-specific vLLM command for Mellum2.1-12B-A2.5B-Thinking with its qwen3 reasoning parser and hermes tool parser. Reproduce this only after checking the installed vLLM release and GPU capacity. Not run by OMW.

1. Start the documented local service

vllm serve JetBrains/Mellum2.1-12B-A2.5B-Thinking \
  --max-model-len 131072 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

2. Request one harmless tool call

This request declares a deterministic integer-sum tool; it does not execute that tool. Review the model response for the correct name and arguments.

curl -fsS -X POST 'http://127.0.0.1:8000/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  --data-binary '{"model":"JetBrains/Mellum2.1-12B-A2.5B-Thinking","stream":false,"messages":[{"role":"user","content":"Please call sum_numbers with a=13 and b=29. Do not calculate directly."}],"tools":[{"type":"function","function":{"name":"sum_numbers","description":"Return the sum of two integers.","parameters":{"type":"object","properties":{"a":{"type":"integer"},"b":{"type":"integer"}},"required":["a","b"],"additionalProperties":false}}}],"tool_choice":"auto"}'

3. Run a bounded, local-only evaluator

From a checkout of our repository, the following CLI verifies tool-call structure and records raw outcomes from a local endpoint. Its self-test validates the parser only. A live run requires the model server above; none was performed during publication.

python3 scripts/agent_tool_call_eval.py --self-test
python3 scripts/agent_tool_call_eval.py --endpoint http://127.0.0.1:8000/v1/chat/completions --model JetBrains/Mellum2.1-12B-A2.5B-Thinking --output /tmp/agent-tool-calls.json

The full 131,072-token context can require substantial serving memory. Do not assume the checked model's mathematical weight-only memory is a live VRAM measurement.

Check the provider's exact runtime instructions ↗

Minimum acceptance checklist

Record model artifact digest and license, runtime release, template/parser, sampled output, rejection of unexpected tool names/arguments, elapsed time, actual peak RAM/VRAM and all failures. Keep tool execution sandboxed and require human approval for side effects.

Evaluate before deployment.

Model metadata is source-linked; agent task success, tool-call reliability and production readiness require separate tests.

Review the methodology → Return to the explorer →