Serve Local Agent Models with llama.cpp

llama.cpp serves compatible local model artifacts, often GGUF. Function calling depends on its chat-template handling and the checkpoint's actual behavior, not merely a server flag.

Four steps to a reproducible deployment

Choose a verified artifact
Check the exact GGUF file, tokenizer, prompt format and source license. Do not infer a GGUF conversion exists for every curated checkpoint.
Start a local server
Use llama-server with a locally inspected compatible model file. The --jinja option activates template-aware tool-call handling where supported.
Test tool arguments
Send a minimal tool specification and validate output on known cases. Generic templates can differ from native handlers.
Secure the host
Restrict listener access, sandbox execution, protect prompt contents and monitor memory rather than equating disk weight size with VRAM.

Minimal acceptance tests

Record the repository revision, artifact checksum, runtime version and command line in a deployment manifest. Verify ordinary chat, a negative permission test, malformed tool arguments and model recovery. Measure actual memory and latency on the target hardware rather than extrapolating from a weight-only estimate.

No specific one of the 28 curated checkpoints is certified to run in this stack by this page. See the agent model explorer for source-linked candidates.

COMMON QUESTIONS

Frequently asked questions

Can any listed checkpoint be deployed with llama.cpp?

Not necessarily. Runtime compatibility, weight format and chat templates must be checked for the exact checkpoint.

Does local inference make tool execution safe?

No. Tool permissions, sandboxing, network policy, logging and human approval remain application responsibilities.

READ THE ORIGINALS

Documentation and primary sources

COPYABLE LOCAL RECIPE · NOT YET REPRODUCED BY OMW

Example setup and acceptance check

The llama.cpp maintainers document a tool-aware llama-server --jinja example with Qwen2.5-7B-Instruct-GGUF. That smoke-test checkpoint is outside the curated Leading/Notable cohort; it demonstrates the runtime protocol without inventing GGUF availability for a Leading/Notable model.

1. Start the documented local service

llama-server --jinja -fa \
  -hf bartowski/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \
  --host 127.0.0.1 --port 8080

curl -fsS http://127.0.0.1:8080/v1/models

2. Request one harmless tool call

This request declares a deterministic integer-sum tool; it does not execute that tool. Review the model response for the correct name and arguments.

curl -fsS -X POST 'http://127.0.0.1:8080/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  --data-binary '{"model":"gpt-3.5-turbo","stream":false,"messages":[{"role":"user","content":"Please call sum_numbers with a=13 and b=29. Do not calculate directly."}],"tools":[{"type":"function","function":{"name":"sum_numbers","description":"Return the sum of two integers.","parameters":{"type":"object","properties":{"a":{"type":"integer"},"b":{"type":"integer"}},"required":["a","b"],"additionalProperties":false}}}],"tool_choice":"auto"}'

3. Run a bounded, local-only evaluator

From a checkout of our repository, the following CLI verifies tool-call structure and records raw outcomes from a local endpoint. Its self-test validates the parser only. A live run requires the model server above; none was performed during publication.

python3 scripts/agent_tool_call_eval.py --self-test
python3 scripts/agent_tool_call_eval.py --endpoint http://127.0.0.1:8080/v1/chat/completions --model gpt-3.5-turbo --output /tmp/agent-tool-calls.json

The server's example model ID may differ by build; inspect /v1/models and replace the sample model value if needed. Check the selected third-party GGUF license and pin its file/revision. The curated GLM-5.3-Flash llama.cpp release reference is not evidence that an exact GGUF tool-calling template has been independently verified.

Check the provider's exact runtime instructions ↗

Minimum acceptance checklist

Record model artifact digest and license, runtime release, template/parser, sampled output, rejection of unexpected tool names/arguments, elapsed time, actual peak RAM/VRAM and all failures. Keep tool execution sandboxed and require human approval for side effects.

Evaluate before deployment.

Model metadata is source-linked; agent task success, tool-call reliability and production readiness require separate tests.

Review the methodology → Return to the explorer →