Complete endpoint documentation for Ollama Herd.
OpenAI-compatible chat completions with streaming and non-streaming support.
Request body:
{
"model": "llama3.3:70b",
"messages": [
{"role": "system", "content": "You are helpful."},
{"role": "user", "content": "Hello!"}
],
"stream": true,
"temperature": 0.7,
"max_tokens": 1024,
"fallback_models": ["qwen2.5:32b", "qwen2.5:7b"],
"metadata": {"tags": ["my-app", "production"]},
"user": "alice"
}| Field | Type | Default | Description |
|---|---|---|---|
model |
string | required | Model name (must exist on at least one node) |
messages |
array | [] |
Chat messages in OpenAI format |
stream |
boolean | false |
Enable streaming (SSE) |
temperature |
float | 0.7 |
Sampling temperature |
max_tokens |
integer | null |
Maximum tokens to generate |
fallback_models |
array | [] |
Backup models to try if primary unavailable |
metadata.tags |
array | [] |
Tags for per-app analytics (e.g., ["my-app", "prod"]) |
user |
string | null |
User identifier — stored as user:<value> tag |
Request headers:
| Header | Description |
|---|---|
X-Herd-Tags |
Comma-separated tags (alternative to metadata.tags in body) |
Tags from all sources (body, header, user field) are merged and deduplicated. See Request Tagging for details.
Streaming response (stream: true):
data: {"id":"chatcmpl-abc123","choices":[{"delta":{"role":"assistant"},"index":0}]}\n\n
data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":"Hello"},"index":0}]}\n\n
data: {"id":"chatcmpl-abc123","choices":[{"delta":{},"finish_reason":"stop","index":0}]}\n\n
data: [DONE]\n\n
Non-streaming response (stream: false):
{
"id": "chatcmpl-abc123def456",
"object": "chat.completion",
"created": 1710000000,
"model": "llama3.3:70b",
"choices": [
{
"index": 0,
"message": {"role": "assistant", "content": "Hello! How can I help?"},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 12,
"completion_tokens": 8,
"total_tokens": 20
}
}Request headers (optional):
| Header | Effect |
|---|---|
X-Fleet-No-Fallback: true |
Strict mode — serve the exact model requested or return an explicit error; never substitute a loaded model. Without it, a model that isn't loaded is loaded on a node with room, and only when every node would have to evict a resident model is it served by a loaded model of the same category (never another). (Also settable as "fallback": false in the body.) |
X-Herd-Tags |
Comma-separated request tags for per-app analytics. |
Response headers — the canonical X-Fleet-* set is emitted on every proxied response (all endpoints, streaming and non-streaming), so a client reads the same three fields anywhere and knows exactly what ran:
| Header | Always? | Description |
|---|---|---|
X-Request-Id |
✅ | The router's id for this request, the same value stored as the trace's request_id. Also on error responses once the router has given the request an id (a rejection, a failed attempt); a request refused before that, such as one with no model, has none. Pass it to GET /dashboard/api/traces?request_id= for the router's side of the request: node, retries, fallback, queue and route time. Image generation and transcription return it too, but write no trace when they complete |
X-Fleet-Node |
✅ | Node ID that served the request |
X-Fleet-Served-Model |
✅ | The model that actually ran (differs from requested on a fallback) |
X-Fleet-Requested-Model |
✅ | What the client asked for |
X-Fleet-Fallback |
✅ | true / false — whether a substitution happened (a real boolean; not a model name) |
X-Fleet-Backend |
✅ | ollama / mlx / native / vision / image / transcription |
X-Fleet-Retries |
✅ | Retry count (0 when none) |
X-Fleet-Score |
scorer-routed | Winning routing score (integer) |
X-Fleet-Context-Overflow |
on overflow | estimated_tokens=N; context_length=M when input exceeds the node's context window. Ollama truncates such input and still answers 200; when the backend's count confirms it, the router also raises the input_truncated health check |
X-Fleet-Overhead-Ms |
non-streaming chat | Herd's own time for the request: its total time minus Ollama's own total_duration (routing, any wait in herd's queue, translation, the network hop to the node). Each trace's timings splits it. |
X-Thinking-Tokens / X-Output-Tokens / X-Budget-Used / X-Done-Reason |
thinking models, non-streaming | Reasoning-budget diagnostics |
X-Fleet-Modelis retired (0.8.1) — useX-Fleet-Served-Model.
Error responses:
| Status | Condition |
|---|---|
| 400 | Missing model field |
| 404 | Model not found on any node (auto-pull attempted first if FLEET_AUTO_PULL=true) |
| 429 | Per-client concurrency cap exceeded (FLEET_CLIENT_MAX_IN_FLIGHT); includes Retry-After |
| 503 | Model exists but no node can serve it right now |
OpenAI-compatible image generation. Wraps /api/generate-image in OpenAI's image API format.
Request body:
{
"model": "z-image-turbo",
"prompt": "a cat sitting on a laptop",
"size": "1024x1024",
"response_format": "b64_json"
}| Field | Type | Default | Description |
|---|---|---|---|
model |
string | required | Image model name |
prompt |
string | required | Text description of the image |
size |
string | 1024x1024 |
Image dimensions (WIDTHxHEIGHT) |
response_format |
string | b64_json |
b64_json returns base64 PNG; url returns raw PNG bytes |
steps |
integer | model default | Inference steps |
guidance |
float | model default | Guidance scale |
seed |
integer | random | Seed for reproducibility |
negative_prompt |
string | "" |
What to avoid in the image |
Response (b64_json):
{
"created": 1710000000,
"data": [
{
"b64_json": "iVBORw0KGgo...",
"revised_prompt": "a cat sitting on a laptop"
}
]
}Response (url): Raw PNG bytes with Content-Type: image/png.
Error responses: 400 (missing fields), 404 (model not available), 502 (generation failed), 503 (disabled).
List all models across the fleet — LLM, image, and embedding models (loaded + available on disk).
Response:
{
"object": "list",
"data": [
{
"id": "llama3.3:70b",
"object": "model",
"created": 1710000000,
"owned_by": "ollama"
},
{
"id": "qwen2.5:7b",
"object": "model",
"created": 1710000000,
"owned_by": "ollama"
}
]
}OpenAI-compatible embeddings, for clients that only speak OpenAI (Chatbox Knowledge Base, LangChain/LlamaIndex OpenAI embedders, generic "OpenAI-compatible" settings). It is a translation layer over the same path as POST /api/embed: nomic-embed-text still goes to the native fastembed server, vision embedding models to the vision server, and everything else is scored, retried and traced exactly as on /api/embed.
Request:
{
"model": "nomic-embed-text",
"input": ["first document", "second document"],
"encoding_format": "float"
}| Field | Type | Default | Description |
|---|---|---|---|
model |
string | required | Embedding model name (any name /api/embed accepts) |
input |
string or array of strings | required | Text(s) to embed. Empty strings and token-id arrays are rejected with 400. |
encoding_format |
"float" / "base64" |
"float" |
base64 is little-endian float32, as OpenAI encodes it. Anything else is 400. |
dimensions |
integer | — | Forwarded to the backend. If the backend returns a different width (it ignored the parameter), the request fails with 400 rather than returning vectors of the wrong size. |
Response:
{
"object": "list",
"data": [
{"object": "embedding", "index": 0, "embedding": [0.0101, -0.0018, ...]},
{"object": "embedding", "index": 1, "embedding": [0.0207, 0.0114, ...]}
],
"model": "nomic-embed-text",
"usage": {"prompt_tokens": 6, "total_tokens": 6}
}usage is the backend's prompt_eval_count (the fastembed server reports a word-count approximation; 0 if the backend reports nothing). The X-Fleet-* headers from the embed path are preserved.
Errors use OpenAI's shape, {"error": {"message", "type", "param", "code"}}: 400 invalid_request_error, 404 with code: "model_not_found", 503/504 model_overloaded, 502 api_error (including a backend that returned a different number of vectors than inputs).
The OpenAI Responses API — what OpenAI Codex speaks. Codex removed
Chat Completions support in Feb 2026 (wire_api = "chat" is gone), so this is
the only endpoint current Codex can use. Setup: guides/codex-integration.md.
Request:
{
"model": "gpt-5-codex",
"input": [
{"role": "user", "content": [{"type": "input_text", "text": "fix the bug"}]}
],
"tools": [{"type": "function", "name": "bash", "parameters": {}}],
"stream": true
}input accepts a plain string or a list of items: message, function_call,
function_call_output, custom_tool_call, custom_tool_call_output,
reasoning, and additional_tools. Unrecognised item types are dropped and
logged once per type.
Model selection — model may be any id. Resolution order: an explicit
FLEET_ANTHROPIC_MODEL_MAP entry, then a real local model name, then
best-loaded auto-routing, then 404. Requests carrying an image auto-route to a
vision-capable model.
Tools — two shapes are accepted and both are translated:
- a top-level
toolsarray of plain functions, and - Codex's
additional_toolsinput item, whose primary tool is acustom-typedexectaking raw JavaScript ("code mode"). Sincecodex-cli 0.164its tools arrive insidenamespaceentries; members of the defaultfunctionsnamespace are exposed as ordinary tools, other namespaces are not exposed.
Herd also rewrites calls models reliably get wrong, because Codex's tool
descriptions document an API its tool schema doesn't expose. A direct call to
a tool nested inside exec (exec_command, apply_patch, …) becomes a
custom_tool_call on exec of the form text(await tools.<name>(…)) — code mode
returns only what a script passes to text(), and apply_patch takes the patch
as a plain string. The nested tools are declared to the model as stubs so that
Ollama can parse such a call at all; they are never returned to the client as
tools. When exec_command is offered as a top-level tool (older Codex), an
apply_patch tool call becomes an exec_command heredoc instead (apply_patch
was a binary on the sandbox PATH there). Successful rewrites are logged at INFO (WARNING through 0.12.0); a call that cannot be rewritten logs WARNING.
Response (non-streaming) is a Responses object whose output is a list of
items — an assistant message plus one function_call / custom_tool_call per
tool call. Streaming emits the Responses SSE event sequence
(response.output_item.added → …arguments.delta → response.completed).
Not supported: previous_response_id (server-side conversation state) —
rejected with 400. Codex's default stateless mode resends the full input
each turn, which is what Herd implements. mlx: models are skipped for this
endpoint.
Response headers: same X-Fleet-* set as every other route, including
X-Fleet-Served-Model.
Reranks documents against a query, for the rerank step of RAG pipelines (Open
WebUI, Dify, LangChain, LlamaIndex). Ollama has no rerank endpoint. Herd
serves this from the native fastembed server that nodes with the embedding
extra already run, so there's no new dependency. Uses the Jina/Cohere request
and response shape.
curl http://localhost:11435/v1/rerank -H 'Content-Type: application/json' -d '{
"query": "how do I stop herd now that launchd is installed?",
"documents": ["launchctl bootout gui/$UID/...", "pkill is no longer a stop", "MLX setup"],
"top_n": 2,
"return_documents": true
}'Request: query (required) and documents (strings, or {"text": ...} objects).
Optional: model, top_n, and return_documents (default false).
Response: {model, results: [{index, relevance_score, document?}], usage: {total_tokens}}.
Results are sorted best first, and index points into the original documents.
relevance_score is the cross-encoder logit passed through a sigmoid. It's in
0–1 like Cohere's and Jina's, so clients that apply a relevance threshold
behave the same way. The serving node is in X-Fleet-Node, not the body.
Models. Weights download on first use, then stay cached.
model |
Size | Notes |
|---|---|---|
ms-marco-minilm-l-6-v2 (default) |
80 MB | Fastest; English |
ms-marco-minilm-l-12-v2 |
120 MB | English |
jina-reranker-v1-tiny-en |
130 MB | English, long inputs |
jina-reranker-v1-turbo-en |
150 MB | English, long inputs |
bge-reranker-base |
1.04 GB | Quality pick; English/Chinese |
jina-reranker-v2-base-multilingual |
1.11 GB | Quality pick; multilingual |
Limits: at most 1,000 documents of 32,000 characters each → 400. A
cross-encoder runs one forward pass per document.
Errors:
400: invalid request, from the node.404: unknownmodel; the error lists the available ones.503: no online node can serve it right now. With aRetry-Afterheader, the node that runs the reranker is restarting or offline, or the router has just restarted and has not heard from its nodes yet: retry. Without one, no node runs a rerank-capable text server at all (uv sync --extra embedding).504: timed out, usually because a first request is still downloading weights.
Traces carry the rerank tag.
Ollama's decision models (Ollama ≥ 0.35: nimble, tev1), which return
choices with probabilities instead of text. Use them for triage,
classification, and model routing. Herd passes the request through unchanged
to a node whose Ollama reports the decision capability for the model. Nodes on
older Ollama can't report that capability, so they're skipped.
curl http://localhost:11435/v1/systemone -H 'Content-Type: application/json' -d '{
"model": "nimble",
"state": "Our checkout has returned 500 errors since 9am.",
"questions": {"label": {"type": "choice", "instructions": "Which label fits?",
"criteria": {"billing": "Payments", "bug": "Software errors", "account": "Login"}}}
}'Response: Ollama's, unchanged. Example: answers.label.choice == "bug", with
probabilities and confidence, plus usage.
Limits: these are Ollama's, enforced at the router before any network hop:
the body may not exceed 64 KiB unless images is set, and never 32 MiB → 413.
Errors:
404: no node serves that decision model. Needs Ollama ≥ 0.35 and a local GGUF decision model (ollama pull nimble). Recorded as arejectedtrace.400: passed through from Ollama, for example a:cloudor MLX model, since System One only accepts local GGUF models.
Traces carry the decision tag. Node selection and failover are the same as
/api/show: loaded nodes first, and the winning node is in X-Fleet-Node.
Every endpoint below works unchanged from desktop apps, CLIs and server-side code.
Browser-based clients are the exception: the browser sends a preflight OPTIONS
and refuses to hand the page any response without Access-Control-* headers. The
router emits none by default, so a page-based client gets a CORS error even though
the request itself would have succeeded.
Set FLEET_CORS_ORIGINS to an allow-list to enable it — same syntax as
OLLAMA_ORIGINS. All X-Fleet-* response headers are exposed to the page, since
CORS otherwise hides them from exactly the clients this is for. Full detail,
including why nothing is allowed implicitly, is in
Configuration Reference § Browser clients and CORS.
Liveness probe, returns 200 with an empty body, like Ollama. OllamaKit (Ollamac, Enchanted) checks reachability with exactly this request and requires a 2xx. GET / is unchanged: it redirects browsers to /dashboard.
Ollama-compatible chat endpoint. Streaming is enabled by default (matches Ollama behavior).
Request body:
{
"model": "llama3.3:70b",
"messages": [
{"role": "user", "content": "Hello!"}
],
"stream": true,
"options": {
"temperature": 0.7,
"num_predict": 1024
},
"fallback_models": ["qwen2.5:32b"],
"metadata": {"tags": ["my-app"]}
}| Field | Type | Default | Description |
|---|---|---|---|
model |
string | required | Model name |
messages |
array | [] |
Chat messages |
stream |
boolean | true |
Streaming (NDJSON) |
options.temperature |
float | 0.7 |
Sampling temperature |
options.num_predict |
integer | null |
Max tokens |
fallback_models |
array | [] |
Backup models |
metadata.tags |
array | [] |
Tags for per-app analytics |
user |
string | null |
User identifier — stored as user:<value> tag |
The metadata and fallback_models fields are stripped before proxying to Ollama. The X-Herd-Tags header is also supported (same as OpenAI endpoint).
Streaming response (NDJSON, one JSON object per line):
{"message":{"role":"assistant","content":"Hello"},"done":false}
{"message":{"role":"assistant","content":"!"},"done":false}
{"message":{"role":"assistant","content":""},"done":true,"total_duration":1234567890,"prompt_eval_count":12,"eval_count":8}Non-streaming response:
{
"message": {"role": "assistant", "content": "Hello! How can I help?"},
"done": true,
"total_duration": 1234567890,
"prompt_eval_count": 12,
"eval_count": 8
}Ollama-compatible generate endpoint. Uses prompt instead of messages.
Request body:
{
"model": "llama3.3:70b",
"prompt": "Why is the sky blue?",
"stream": true,
"options": {
"temperature": 0.7
}
}Supports the same metadata.tags, user, and X-Herd-Tags header as /api/chat.
Response format is the same as /api/chat.
List all models across the fleet with node information. Includes LLM models, mlx: models, image models (mflux + DiffusionKit) and vision embedding models.
Each entry has Ollama's full field set, plus Herd's details.fleet_nodes. Strict clients depend on this: OllamaKit (the library behind Enchanted and Ollamac) and Reins decode modified_at, digest and every details string as required and non-null, so one missing key fails the whole listing and the model picker comes up empty.
Response:
{
"models": [
{
"name": "gpt-oss:120b",
"model": "gpt-oss:120b",
"modified_at": "2026-04-24T05:29:24.844674018-03:00",
"size": 65369818941,
"digest": "a951a23b46a1f6093dafee2ea481d634b4e31ac720a8a16f3f91e04f5a40ecd9",
"details": {
"parent_model": "",
"format": "gguf",
"family": "gptoss",
"families": ["gptoss"],
"parameter_size": "116.8B",
"quantization_level": "MXFP4",
"fleet_nodes": ["mac-studio-ultra", "macbook-pro-m4"]
}
},
{
"name": "mlx:Qwen3-Coder-Next-4bit",
"model": "mlx:Qwen3-Coder-Next-4bit",
"modified_at": "2026-09-29T17:04:11.482113Z",
"size": 44800000000,
"digest": "3f0c…(sha256 of the name, see below)",
"details": {
"parent_model": "",
"format": "mlx",
"family": "",
"families": [],
"parameter_size": "",
"quantization_level": "",
"fleet_nodes": ["mac-studio-ultra"]
}
},
{
"name": "z-image-turbo",
"model": "z-image-turbo",
"modified_at": "2026-09-29T17:04:11.482200Z",
"size": 0,
"digest": "…",
"details": {
"parent_model": "", "format": "mflux", "family": "", "families": [],
"parameter_size": "", "quantization_level": "",
"fleet_nodes": ["mac-studio-ultra"],
"type": "image"
}
}
]
}Where the values come from:
- Ollama models: passed through unchanged from the node's own Ollama
/api/tags. The node agent carries them in its heartbeat (ollama.models_available_meta), but only when they change plus once a minute as a refresh, since for a 50-model node they are about 13.6 KB, 15x the rest of the Ollama section. Ollama's ownnulls (it sends"families": nullfor some models) are normalized to[]/"". - Models with no Ollama metadata (
mlx:models, image and vision-embedding models, or any model on a node running an agent older than this field) get non-null defaults instead ofnull:details.format:"mlx"formlx:models, the runtime ("onnx") for vision embedding models, the generator ("mflux","diffusionkit") for image models, and""when unknown. Herd does not guess"gguf".family,parameter_size,quantization_level,parent_model:"".families:[].modified_at: the time this router process first listed the model (RFC3339 with fractional seconds, UTC). Stable across calls, so a list sorted by date does not reshuffle on every poll; it resets when the router restarts.digest:sha256("ollama-herd:<name>"). It is not a content digest. It is unique and stable so clients that key their lists ondigestdon't collapse every such model into one row.
sizeis the on-disk size in bytes, as in Ollama, when any node reports it; it falls back to the resident size for a loaded model with no disk size, and0otherwise.fleet_nodeslists every node that has the model (loaded or on disk). Image models add"type": "image"; vision embedding models add"type": "vision-embedding",runtimeanddimensions.
List all currently loaded (hot) models across the fleet.
Response:
{
"models": [
{
"name": "llama3.3:70b",
"model": "llama3.3:70b",
"size": 42949672960,
"fleet_node": "mac-studio-ultra"
},
{
"name": "qwen2.5:7b",
"model": "qwen2.5:7b",
"size": 4294967296,
"fleet_node": "macbook-air-m2"
}
]
}Ollama-compatible model details. Ollama's desktop app calls it before every chat, AnythingLLM reads the context window (model_info["<arch>.context_length"]) and tool support (capabilities) from it, and Cherry Studio's "Check" button probes it.
Request body: {"model": "gpt-oss:120b"}. The legacy {"name": ...} field is also accepted, and a bare name resolves to :latest as in Ollama. Other fields (e.g. verbose) are forwarded.
Behavior:
| Model | Response |
|---|---|
| Ollama model | Proxied to a node that has it (nodes with it loaded first, then on disk). That node's Ollama response is returned unchanged, with X-Fleet-Node. If a node is unreachable or errors, the next one is tried. |
mlx: model |
Synthesized: details.format: "mlx", capabilities: ["completion"], empty modelfile / parameters / template, and an empty model_info (Herd doesn't know the context length, so it doesn't invent one). |
| Vision embedding model | Synthesized, capabilities: ["embedding"]. |
| Image model | Synthesized, capabilities: ["image"]. |
| Unknown | 404 {"error": "model '<name>' not found"}, matching Ollama. |
Error responses: 400 (missing model / invalid JSON), 404 (unknown model, or every node that should have it answered 404), 502 (no node holding the model was reachable).
Ollama-compatible embeddings endpoint. The router dispatches to the right backend based on model name — clients need no changes:
| Model type | Backend | Port |
|---|---|---|
dinov2-vit-s14, clip, siglip |
Native vision embedding (ONNX) | :11438 |
nomic-embed-text, nomic-embed-text:latest |
Native text embedding (fastembed) | :11439 |
| All other models | Ollama on the selected node | :11434 |
Vision embedding input is base64 image bytes in images (or input), and must be JPEG, PNG or WebP. Any other format, including GIF, TIFF, BMP and EPS, gets 400 {"error": "Image N is not one of: JPEG, PNG, WEBP", "node": "<node>"} before any decoder runs: those decoders carry most of Pillow's security advisories, and the server takes unauthenticated bytes from the LAN.
The native text embedding backend (fastembed / ONNX Runtime) runs entirely outside Ollama's inference queue — nomic-embed-text requests never consume OLLAMA_NUM_PARALLEL slots and can never be stalled by concurrent LLM inference.
Request body:
{
"model": "nomic-embed-text",
"input": "The quick brown fox jumps over the lazy dog"
}| Field | Type | Default | Description |
|---|---|---|---|
model |
string | required | Embedding model name |
input |
string/array | — | Text(s) to embed (string or ["text1", "text2"]) |
prompt |
string | — | Alias for input (legacy compat) |
metadata.tags |
array | [] |
Tags for per-app analytics |
Task prefixes for nomic-embed-text: the model supports "search_query: ..." and "search_document: ..." prefixes for asymmetric retrieval quality. These are the caller's responsibility — neither herd nor Ollama adds them automatically.
Response (Ollama-compatible for all backends):
{
"model": "nomic-embed-text",
"embeddings": [[0.0101, -0.0018, ...]],
"total_duration": 4500000,
"load_duration": 0,
"prompt_eval_count": 9
}Error responses:
| Status | Condition |
|---|---|
| 400 | Missing model or input field |
| 404 | Model not found on any node |
| 503 | No online node can serve it. With Retry-After: the node running the native server is restarting or offline, or the router has just restarted. It is transient, so retry. Without it: no node runs the native server; install fastembed (uv sync --extra embedding). /v1/embeddings passes Retry-After through. |
| 504 | Text embedding timed out — first request may trigger 130 MB model download; retry after 30s |
Pull a model onto the fleet. Ollama-compatible — streams NDJSON progress matching Ollama's wire format. Auto-selects the node with the most available memory, or accepts an explicit node_id.
Request body:
| Field | Type | Default | Description |
|---|---|---|---|
name |
string | — | Model to pull (Ollama standard field) |
model |
string | — | Model to pull (alias for name — many agents send this) |
stream |
bool | true |
Stream NDJSON progress or return on completion |
node_id |
string | auto | Target node (auto-selects best node if omitted) |
Streaming response (stream: true, default):
{"status":"pulling manifest"}
{"status":"pulling abc123...","digest":"sha256:abc123","total":5000000,"completed":2500000}
{"status":"verifying sha256:abc123"}
{"status":"writing manifest"}
{"status":"success"}
Non-streaming response (stream: false):
{"status": "success"}Response headers:
| Header | Description |
|---|---|
X-Fleet-Node |
Node that received the pull |
Error responses:
| Status | Condition |
|---|---|
| 400 | Missing model name, or model is a non-Ollama model (mflux, DiffusionKit, MLX) — returns install instructions |
| 404 | Specified node_id not found or offline |
| 409 | Model is already being pulled |
| 503 | No node has enough available memory |
Non-Ollama models: Image generation models (z-image-turbo, flux-dev, sd3-medium, sd3.5-large) and speech-to-text models (qwen3-asr) are not Ollama models and cannot be pulled via this endpoint. The error response includes the correct install command for each.
Examples:
# Pull a model (streams NDJSON progress)
curl -N http://localhost:11435/api/pull -d '{"name": "codestral"}'
# Pull to a specific node
curl -N http://localhost:11435/api/pull -d '{"name": "llama3.3:70b", "node_id": "mac-studio"}'
# Non-streaming (blocks until complete)
curl http://localhost:11435/api/pull -d '{"name": "phi4", "stream": false}'List all image models across the fleet (mflux, DiffusionKit, and Ollama native).
Response:
{
"models": [
{
"name": "z-image-turbo",
"type": "image",
"backend": "mflux",
"fleet_nodes": ["mac-studio-ultra"]
},
{
"name": "sd3-medium",
"type": "image",
"backend": "mflux",
"fleet_nodes": ["macbook-pro-m4"]
},
{
"name": "x/z-image-turbo:latest",
"type": "image",
"backend": "ollama",
"fleet_nodes": ["mac-studio-ultra"]
}
]
}The backend field indicates which system handles the model: mflux (mflux/DiffusionKit via port 11436) or ollama (Ollama native image models with x/ prefix).
Native Anthropic Messages API surface for Claude Code and other anthropic-SDK clients. See docs/guides/claude-code-integration.md for the full setup walkthrough.
Anthropic Messages endpoint — streaming + non-streaming, full tool use, system prompts, stop sequences.
Request body:
{
"model": "claude-sonnet-4-5",
"max_tokens": 4096,
"messages": [
{"role": "user", "content": "What's the weather in Paris?"}
],
"system": "You are a helpful assistant.",
"tools": [{
"name": "get_weather",
"description": "Get current weather for a city",
"input_schema": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}],
"tool_choice": {"type": "auto"},
"stream": false,
"temperature": 0.7,
"stop_sequences": ["\n\nHuman:"],
"metadata": {"user_id": "alice"}
}Headers (recommended):
x-api-key: <key>— required whenFLEET_ANTHROPIC_REQUIRE_KEY=trueanthropic-version: 2023-06-01— reflected back in response headers
Model resolution: by default claude-* ids auto-route to the best local model you have loaded for that tier — no configuration required. FLEET_ANTHROPIC_MODEL_MAP is optional and pins specific aliases to specific models (its entries win over auto-routing). Real local model names (e.g. qwen3-coder:30b) pass through unchanged. The response's X-Fleet-Served-Model header reports what actually ran. See anthropic-auto-routing.md and configuration-reference.md.
Non-streaming response:
{
"id": "msg_abc123...",
"type": "message",
"role": "assistant",
"model": "claude-sonnet-4-5",
"content": [
{"type": "text", "text": "I'll check that for you."},
{"type": "tool_use", "id": "toolu_xyz...", "name": "get_weather", "input": {"city": "Paris"}}
],
"stop_reason": "tool_use",
"stop_sequence": null,
"usage": {"input_tokens": 25, "output_tokens": 18}
}Stop reasons: end_turn, max_tokens, stop_sequence, tool_use.
Streaming SSE events: message_start → content_block_start/delta/stop (one set per text or tool_use block) → message_delta → message_stop. Tool calls open a new content_block_start of type tool_use mid-stream and emit args via input_json_delta.
Tool result round-trip: Send a follow-up request with the assistant's tool_use block in messages and a user message containing a tool_result block referencing the same tool_use_id:
{
"role": "user",
"content": [{"type": "tool_result", "tool_use_id": "toolu_xyz...", "content": "18C, sunny"}]
}Token estimate for a Messages payload — used by Claude Code for budget gating before each turn. Best-effort: tiktoken cl100k if installed, otherwise len(text)/4. Not for billing.
Request body: same shape as /v1/messages.
Response:
{"input_tokens": 142}Friendly probe endpoint — Claude Code may GET this during connection setup.
Response:
{"ok": true, "service": "ollama-herd", "endpoint": "/v1/messages", "ts": 1776857000}X-Request-Id— the router's id for this request (the trace'srequest_id)X-Fleet-Node— node id that served the requestX-Fleet-Score— routing score (higher = better fit)X-Fleet-Fallback— present when a fallback model was usedX-Fleet-Retries— present when the request was retriedanthropic-version— echoed from the request
The route emits the following non-2xx status codes in addition to the usual 400/401/5xx from upstream. All error bodies follow the Anthropic error shape: {"type": "error", "error": {"type": "...", "message": "..."}}.
-
413 Payload Too Large—error.type: "prompt_too_large". Returned in two situations:- Pre-inference hard cap. After server-side context management (tool-result clearing + LLM compaction with
force_all), the prompt still exceedsFLEET_ANTHROPIC_MAX_PROMPT_TOKENS(default 180K). The message tells the caller to run/compactin Claude Code and resubmit. - MLX wall-clock timeout. An MLX request exceeded
FLEET_MLX_WALL_CLOCK_TIMEOUT_S(default 300s) from admission → last byte. The slot is released; the caller gets the sametry /compacthint. Catches wedged-request syndrome wheremlx_lm.serveremits tokens slowly but never hits a stop condition.
The server never silently retries either case — correctness of agentic tool-use workflows depends on the client owning the decision to resubmit with altered context.
- Pre-inference hard cap. After server-side context management (tool-result clearing + LLM compaction with
-
503 Service UnavailablewithRetry-After— MLX admission control rejected the request (queue full).error.type: "overloaded_error". Clients should back off and retry after the header value.
Full fleet state — nodes, queues, hardware metrics, and health summary.
Response:
{
"fleet": {
"nodes_total": 3,
"nodes_online": 2,
"nodes_degraded": 0,
"models_loaded": 4,
"requests_active": 1
},
"nodes": [
{
"node_id": "mac-studio-ultra",
"status": "online",
"hardware": {
"memory_total_gb": 192,
"cores_physical": 24,
"chip": "Apple M3 Ultra",
"memory_bandwidth_gbps": 819.0,
"arch": "apple_silicon"
},
"ollama_url": "http://10.0.0.100:11434",
"cpu": {
"cores_physical": 24,
"utilization_pct": 15.2
},
"memory": {
"total_gb": 192.0,
"used_gb": 45.3,
"available_gb": 146.7,
"pressure": "normal",
"wired_gb": 12.1,
"compressed_gb": 1.4,
"swap_used_gb": 0.0,
"swap_total_gb": 0.0
},
"ollama": {
"models_loaded": [
{"name": "llama3.3:70b", "size_gb": 40.0}
],
"models_available": ["llama3.3:70b", "qwen2.5:32b"],
"requests_active": 1
},
"mlx_servers": [
{
"port": 11440,
"model": "mlx-community/Qwen3-Coder-Next-4bit",
"status": "healthy",
"status_reason": "",
"kv_bits": 8,
"model_size_gb": 41.8,
"last_ok_ts": 1714000000.0
},
{
"port": 11441,
"model": "mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit",
"status": "healthy",
"status_reason": "",
"kv_bits": 8,
"model_size_gb": 16.0,
"last_ok_ts": 1714000000.0
}
],
"mlx_bind_host": "0.0.0.0",
"vision_embedding_status": {
"backend_available": true,
"cached_model_count": 3
},
"text_embedding": {
"models_available": [
{"name": "nomic-embed-text", "dimensions": 768, "cached": true}
],
"processing": false
},
"text_embedding_port": 11439,
"text_embedding_status": {
"backend_available": true,
"cached_model_count": 1
}
}
],
"queues": {
"mac-studio-ultra:llama3.3:70b": {
"node_id": "mac-studio-ultra",
"model": "llama3.3:70b",
"pending": 0,
"in_flight": 1,
"completed": 42,
"failed": 0,
"concurrency": 8
}
},
"timestamp": 1710000000.0
}fleet.nodes_online counts every node requests are routed to: online plus degraded (a late heartbeat, not yet offline). fleet.nodes_degraded is the part of that which is late.
memory field:
| Field | Type | Description |
|---|---|---|
total_gb / used_gb / available_gb |
float | System memory (psutil) |
pressure |
normal | warn | critical |
On macOS the kernel's own level (kern.memorystatus_vm_pressure_level). Critical withholds cold loads; resident models keep serving |
wired_gb |
float | Wired (non-pageable) memory |
compressed_gb |
float | RAM the macOS compressor occupies (vm.compressor_bytes_used), not the larger amount it holds logically. 0 off macOS |
swap_used_gb / swap_total_gb |
float | Swap in use / allocated. On macOS swap grows on demand, so used/total is ~100% whenever any exists; compare swap_used_gb to total_gb instead |
The swap and compressed fields are 0 from agents at 0.10.0 or earlier, meaning "not
reported", not "healthy". Pressure drops back to normal once paging settles, even with
tens of GB still swapped out, so read both. Related health checks: memory_pressure
(WARNING/CRITICAL from pressure), and swap_usage (WARNING when swap_used_gb reaches
half of total_gb: memory committed well beyond RAM, so models or other apps are paged
out).
mlx_servers field (optional — present only on nodes with MLX configured).
Note context_length is null, not 0: mlx_lm.server is OpenAI-native, so
herd can neither set a window per request nor ask which one the server launched
with. null means "cannot be known"; a 0 would read as a real zero-length
window. It is populated only if a future mlx_lm reports one:
One entry per mlx_lm.server subprocess on that node. Lets operators and
downstream tooling see per-URL health without polling each port individually.
| Field | Type | Description |
|---|---|---|
port |
int | Port the server listens on (unique per-node) |
model |
string | HF repo id or local path (without mlx: prefix) |
status |
string | healthy / starting / unhealthy / memory_blocked / quarantined / stopped |
status_reason |
string | Human-readable detail (empty when healthy) — e.g. "memory gate: 42.0 GB needed, 16 GB available", "subprocess exited rc=1", or "5 crashes within 300s; next restart in 600s" |
kv_bits |
int | 0 / 4 / 8 — KV quantization level (requires setup-mlx.sh patch) |
model_size_gb |
float | Estimated weight size from HF cache on disk |
last_ok_ts |
float | Epoch seconds of last successful /v1/models poll |
mlx_bind_host at the node top level is what the servers bind to (127.0.0.1
local-only; 0.0.0.0 LAN-reachable). Nodes running older agents that predate
multi-MLX return an empty mlx_servers list and omit mlx_bind_host.
vision_embedding_status field (optional — present on node agents 0.6.1+):
Carries the asymmetry between "weights cached on disk" and "backend can serve them"
so the dashboard can stop falsely advertising chips when onnxruntime isn't
installed in the herd-node venv.
| Field | Type | Description |
|---|---|---|
backend_available |
bool | True iff onnxruntime is importable in the herd-node venv at the moment of the last heartbeat |
cached_model_count |
int | Number of vision embedding models (DINOv2 / SigLIP / CLIP) with weights present in ~/.cache/huggingface/hub/ |
The vision_backend_missing health check fires WARNING when
backend_available=false AND cached_model_count>0 — i.e., the operator
intended to use vision embedding (downloaded the weights) but the runtime
isn't installed. Fix: uv sync --extra embedding (or uv sync --all-extras)
on the node, then restart herd-node.
text_embedding / text_embedding_port / text_embedding_status fields (optional — present on node agents 0.7.0+):
The native fastembed text embedding server. When running, nomic-embed-text requests are routed to port text_embedding_port (default 11439) instead of Ollama, so embed requests never consume OLLAMA_NUM_PARALLEL inference slots.
text_embedding.models_available lists registered models with name, dimensions, and cached (whether the ONNX weights are on disk). cached=false means the server is running but the model hasn't been requested yet — it will auto-download (130 MB) on the first request.
text_embedding_status carries the same asymmetry signal as vision_embedding_status:
| Field | Type | Description |
|---|---|---|
backend_available |
bool | True iff fastembed is importable in the herd-node venv |
cached_model_count |
int | Number of text embedding models with weights on disk in ~/.fleet-manager/models/text-embedding/ |
Health checks related to text embedding:
text_embedding_backend_missing— WARNING: weights cached but fastembed not installedtext_embedding_ollama_bypass— WARNING: nomic-embed-text in Ollama but native server not running (embed requests will compete with LLM inference)nomic_loaded_in_ollama— INFO: native server is running but nomic-embed-text still occupies an Ollama slot (cosmetic; self-resolves on KEEP_ALIVE expiry)
Without the native server, nomic-embed-text is served by Ollama's copy until a node in the herd has served it natively; after that it stays native (the two backends' vectors differ, and one index must not mix them).
embed_error_rate— WARNING/CRITICAL: embed failures in the last hour (regardless of backend)
Caching note: /dashboard/api/health responses are cached server-side for
30 seconds (configurable in dashboard.py::_HEALTH_CACHE_TTL_S). At the
dashboard's 15-second poll cadence this gives a guaranteed cache hit on
alternate polls — about a 50% reduction in trace-DB-aggregation work per
active dashboard tab. Recommendations are derived from windows of
minutes-to-hours, so 30-second staleness is invisible to operators. This
endpoint is the only cached one; everything else is computed live.
Receives heartbeats from node agents. Internal endpoint — not intended for external clients.
Regular heartbeat:
{
"node_id": "mac-studio-ultra",
"cpu": {"cores_physical": 24, "utilization_pct": 15.2},
"memory": {"total_gb": 192.0, "used_gb": 45.3, "available_gb": 146.7, "pressure": "normal"},
"ollama": {
"models_loaded": [{"name": "llama3.3:70b", "size_gb": 40.0}],
"models_available": ["llama3.3:70b", "qwen2.5:32b"],
"requests_active": 1
},
"hardware": {
"memory_total_gb": 192,
"cores_physical": 24,
"chip": "Apple M3 Ultra",
"memory_bandwidth_gbps": 819.0,
"arch": "apple_silicon"
}
}ollama.models_available_meta (optional) maps each model to its Ollama /api/tags metadata (modified_at, digest, format, family, families, parameter_size, quantization_level, parent_model), which the router's GET /api/tags returns. The agent sends it only when it changes, plus once every 60 s; null means "unchanged", and the router keeps its previous copy.
Drain signal:
{
"node_id": "mac-studio-ultra",
"draining": true
}Response:
{"status": "ok", "node_status": "online"}Generate an image on the best available node with mflux. Requires FLEET_IMAGE_GENERATION=true.
Request body (JSON):
{
"model": "z-image-turbo",
"prompt": "a cat sitting on a laptop",
"width": 1024,
"height": 1024,
"steps": 4,
"quantize": 8,
"seed": 42,
"negative_prompt": ""
}Response: Raw PNG bytes with Content-Type: image/png.
Response headers:
| Header | Description |
|---|---|
X-Request-Id |
The router's id for this request (in its log; no trace is written for a completed image) |
X-Fleet-Node |
Node that generated the image |
X-Fleet-Served-Model |
Image model used (with the rest of the canonical X-Fleet-* set) |
X-Generation-Time |
Generation time in ms |
Error responses: 400 (missing fields), 404 (model not available), 502 (generation failed), 503 (disabled).
Transcribe an audio file on the best available node with Qwen3-ASR. Requires FLEET_TRANSCRIPTION=true.
Request: multipart/form-data with audio file field.
curl http://localhost:11435/api/transcribe -F "audio=@recording.wav"Supported formats: WAV, MP3, M4A, FLAC, MP4, OGG (any format FFmpeg supports).
Response:
{
"text": "Full transcription text...",
"language": "English",
"chunks": [
{
"text": "Hello, this is a test.",
"start": 0.0,
"end": 2.5,
"chunk_index": 0,
"language": "English"
}
]
}Response headers:
| Header | Description |
|---|---|
X-Request-Id |
The router's id for this request (in its log; no trace is written for a completed transcription) |
X-Fleet-Node |
Node that transcribed the audio |
X-Fleet-Served-Model |
Transcription model used (with the rest of the canonical X-Fleet-* set) |
X-Transcription-Time |
Processing time in ms |
Error responses: 404 (no STT models available), 502 (transcription failed), 503 (disabled).
Lightweight queue status for client-side backoff decisions. Designed for high-frequency polling.
Note: returns Ollama and MLX queue entries only — these are the backends with real queue depth. Native text embedding (fastembed, port 11439) and vision embedding (port 11438) are instant backends with no queue depth; they appear in the dashboard's Node Models section but not here, since their pending=0 / in_flight=0 entries carry no backoff signal.
Response:
{
"queue_depth": 5,
"pending": 2,
"in_flight": 3,
"completed": 1250,
"failed": 3,
"estimated_wait_ms": 15000,
"nodes_online": 2,
"queues": {
"Studio:gpt-oss:120b": {
"pending": 2,
"in_flight": 3,
"concurrency": 2,
"model": "gpt-oss:120b",
"node_id": "Studio"
},
"mlx-local:mlx:mlx-community/Qwen3-Coder-Next-4bit": {
"pending": 0, "in_flight": 1, "concurrency": 1,
"max_queue_depth": 10,
"cache_hit_rate": 0.997, "warm_hit_rate": 1.0, "cold_request_pct": 0.02,
"avg_latency_ms": 5293.6, "avg_prompt_tokens": 201293, "avg_completion_tokens": 95,
"stats_samples": 1244,
"tool_repair": {"attempts": 3, "successes": 2, "failures": 1}
}
},
"timestamp": 1712345678.123
}Use estimated_wait_ms to decide whether to send a request now or back off. queue_depth = pending + in_flight.
MLX-specific fields (present on mlx-local: queue entries):
max_queue_depth— admission-control cap; requests over this get 503 (seeFLEET_MLX_MAX_QUEUE_DEPTH)cache_hit_rate,warm_hit_rate,cold_request_pct— prompt-cache observability (seedocs/plans/mlx-prompt-cache-optimization.md)avg_latency_ms,avg_prompt_tokens,avg_completion_tokens,stats_samples— running averages since process starttool_repair: {attempts, successes, failures}— tool-call JSON repair counters.attemptsfires whenever the model emits malformed JSON;successesmeans repair produced a dict that passed the tool'sinput_schema;failuresmeans repair couldn't produce valid output and the original was passed through. Sustained rate above ~1% on any model is a signal the model is unreliable. Seeserver/tool_call_repair.py.
| Endpoint | Description |
|---|---|
GET /dashboard |
Overview — the fleet's status and health score; live figures with sparklines; an activity chart (tokens/s by model, requests/min, failures); node cards with CPU, memory and tokens/s sparklines; models at work; Fleet Intelligence briefing. Uses the full window width |
GET /dashboard/models |
Models — traffic trends (requests, latency, tokens per hour) and per-model performance, one time range for both |
GET /dashboard/benchmarks |
Benchmarks — fleet throughput under controlled load, run history and charts; linked from Models and Settings (not a nav tab) |
GET /dashboard/tags |
Usage — per-tag analytics with latency, tokens, errors, daily trends |
GET /dashboard/health |
Health — findings by severity (suggestions folded), fleet vitals, model recommendations with one-click pull, briefing history |
GET /dashboard/settings |
Settings — appearance (per browser), runtime toggles, context windows, nodes, configuration |
GET /dashboard/color-states |
Colour reference — every colour token in both appearances, and the utilization bars in each warning state |
Pages merged by the dashboard redesign keep their URLs: /dashboard/trends renders Models, /dashboard/recommendations renders Health, and /dashboard/apps renders Usage. Each scrolls to the section it used to be.
Server-Sent Events stream for real-time fleet updates. Pushes a JSON snapshot every 2 seconds.
Event data:
{
"nodes": [...],
"queues": {...},
"timestamp": 1710000000.0
}Each node includes cpu, memory, ollama, and optionally capacity fields (when adaptive capacity learning is enabled).
Per-bucket request activity for the Overview's live charts, from the trace store.
| Param | Default | Notes |
|---|---|---|
window |
3600 |
Seconds. Snapped to the nearest of 3600, 7200, 10800 (60 s buckets), 21600 (180 s) or 86400 (600 s), so every viewer's buckets align |
Returns window_s, step_s, buckets (bucket start times; the last is the current, partial bucket), and arrays aligned to them: requests, failed (any status other than completed or retried, as the error-rate vital counts it), tokens (completion tokens), ttft_p50_ms (null for an empty bucket). Token series are also split as by_model, by_node and by_node_model (keyed node:model, like /fleet/queue), plus requests_by_node_model.
CPU %, memory %, queued and running per node, sampled by the router every 5 s and kept in memory for 3 hours (cleared on restart). window (seconds, max 10800). Returns t plus queued/running fleet totals aligned to it, and nodes[id] with cpu, mem, queued, running arrays; null where a node was not online at that sample.
Hourly aggregated stats for the trends charts.
| Parameter | Type | Default | Description |
|---|---|---|---|
hours |
integer | 72 |
Number of hours of history to return |
Response:
{
"hours": 72,
"data": [
{
"hour_bucket": "2026-03-08 01:00:00",
"request_count": 15,
"avg_latency_ms": 4200.5,
"avg_prompt_tokens": 125.0,
"avg_completion_tokens": 350.0
}
]
}Per-model daily stats and a per-model summary for the Models page. With start_ts and end_ts, the summary covers that range; without them it is all time.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
integer | 7 |
Number of days of history |
Response:
{
"days": 7,
"daily": [
{
"model": "llama3.3:70b",
"day": "2026-03-08",
"request_count": 42,
"avg_latency_ms": 4200.5,
"total_prompt_tokens": 5250,
"total_completion_tokens": 14700
}
],
"summary": [
{
"model": "llama3.3:70b",
"total_requests": 312,
"avg_latency_ms": 4100.2,
"total_prompt_tokens": 39000,
"total_completion_tokens": 109200
}
]
}Summary totals for the dashboard header cards.
Response:
{
"total_requests": 847,
"total_prompt_tokens": 105875,
"total_completion_tokens": 296450,
"total_tokens": 402325,
"models_count": 5
}Per-node, per-model, per-day usage stats from request traces.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
integer | 7 |
Number of days of history |
Response:
{
"days": 7,
"data": [
{
"node_id": "mac-studio-ultra",
"model": "llama3.3:70b",
"day": "2026-03-08",
"request_count": 25,
"completed": 24,
"failed": 1,
"avg_latency_ms": 4200.5,
"total_prompt_tokens": 3125,
"total_completion_tokens": 8750
}
]
}Request traces, newest first, for debugging and observability. Filters combine with AND.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
integer | 50 |
Maximum number of traces to return |
request_id |
string | Only this request: the X-Request-Id response header. A retried request has one row per attempt |
|
tag |
string | Only requests carrying this tag (one whole entry of tags, not a substring) |
|
since |
number | Only traces at or after this unix timestamp, in seconds |
A trace is written as the request finishes (for a stream, after its last chunk), so a lookup made the instant a response ends can come back empty; retry after a few milliseconds.
Response:
{
"traces": [
{
"request_id": "abc12345-def6-7890",
"model": "llama3.3:70b",
"original_model": "llama3.3:70b",
"node_id": "mac-studio-ultra",
"score": 73.0,
"scores_breakdown": {
"model_thermal": 50,
"memory_fit": 20,
"queue_depth": 0,
"wait_time": 0,
"role_affinity": 15
},
"status": "completed",
"latency_ms": 4200.5,
"time_to_first_token_ms": 850.2,
"prompt_tokens": 125,
"completion_tokens": 350,
"retry_count": 0,
"fallback_used": 0,
"excluded_nodes": [],
"client_ip": "10.0.0.50",
"original_format": "openai",
"error_message": null,
"timestamp": 1710000000.0,
"tags": ["my-app", "production"],
"timings": {
"route_ms": 1.2,
"queue_ms": 0.0,
"upstream_ms": 4180.3,
"backend_ms": 4172.0,
"overhead_ms": 8.9
},
"finish_reason": "stop",
"estimated_tokens": 118
}
]
}Per-tag aggregated stats for the Apps page.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
integer | 7 |
Number of days of history |
Response:
{
"days": 7,
"by_tag": [
{
"tag": "my-app",
"request_count": 150,
"avg_latency_ms": 3200.5,
"avg_ttft_ms": 420.1,
"total_prompt_tokens": 18750,
"total_completion_tokens": 52500,
"error_rate": 0.02
}
]
}Per-tag, per-day breakdown for trend charts.
| Parameter | Type | Default | Description |
|---|---|---|---|
days |
integer | 7 |
Number of days of history |
Response:
{
"days": 7,
"daily": [
{
"tag": "my-app",
"day": "2026-03-08",
"request_count": 25,
"avg_latency_ms": 3100.0,
"total_tokens": 8500
}
]
}List stored benchmark runs.
| Parameter | Type | Default | Description |
|---|---|---|---|
limit |
integer | 50 |
Maximum number of runs to return |
Response:
{
"data": [
{
"run_id": "bench-1772966095",
"timestamp": 1710000000.0,
"duration_s": 300.0,
"total_requests": 150,
"total_failures": 0,
"total_prompt_tokens": 18750,
"total_completion_tokens": 52500,
"requests_per_sec": 0.5,
"tokens_per_sec": 175.0,
"latency_p50_ms": 2100.0,
"latency_p95_ms": 4500.0,
"latency_p99_ms": 6200.0,
"ttft_p50_ms": 450.0,
"ttft_p95_ms": 1200.0,
"ttft_p99_ms": 1800.0,
"fleet_snapshot": "{...}",
"per_model_results": "[...]",
"per_node_results": "[...]",
"peak_utilization": "[...]"
}
]
}Save benchmark results from the benchmark script.
Request body:
{
"run_id": "bench-1772966095",
"timestamp": 1710000000.0,
"duration_s": 300.0,
"total_requests": 150,
"total_failures": 0,
"total_prompt_tokens": 18750,
"total_completion_tokens": 52500,
"requests_per_sec": 0.5,
"tokens_per_sec": 175.0,
"latency_p50_ms": 2100.0,
"latency_p95_ms": 4500.0,
"latency_p99_ms": 6200.0,
"ttft_p50_ms": 450.0,
"ttft_p95_ms": 1200.0,
"ttft_p99_ms": 1800.0,
"fleet_snapshot": "{}",
"per_model_results": "[]",
"per_node_results": "[]",
"peak_utilization": "[]"
}Response:
{"status": "saved", "run_id": "bench-1772966095"}Start a benchmark run from the dashboard. Supports default (loaded models only) and smart mode (fill memory with recommended models first).
Request body:
{
"mode": "smart",
"duration": 300,
"model_types": ["llm", "vision", "embed", "image"]
}| Field | Type | Default | Description |
|---|---|---|---|
mode |
string | "default" |
"default" (loaded models) or "smart" (fill memory first) |
duration |
float | 300 |
Benchmark duration in seconds |
model_types |
array | ["llm"] |
Types to benchmark: "llm", "vision", "embed", "image" |
Response: {"status": "started", "run_id": "bench-...", "mode": "smart", "duration": 300, "model_types": ["llm", "vision", "embed", "image"]}
Error: 409 if a benchmark is already running.
Get current benchmark status and progress. Poll every 2 seconds during a run.
Response:
{
"status": "running",
"phase": "Running 300s benchmark (embed, image, llm)...",
"elapsed": 45.2,
"duration": 300,
"requests_completed": 1250,
"requests_failed": 0,
"tok_per_sec": 85.3,
"models": ["gpt-oss:120b", "nomic-embed-text:latest", "z-image-turbo"],
"models_pulled": ["codestral:22b", "llama3.2:1b"],
"pull_progress": {},
"error": null,
"run_id": "bench-..."
}Status values: idle, pulling, warming_up, running, complete, error, cancelled.
During pull phase, pull_progress includes: model, node_id, current, total, ram_gb, on_disk, pct, completed_gb, total_gb.
Cancel a running benchmark.
Response: {"status": "cancelled"} or {"status": "not_running"}.
Per-model context usage analysis — actual vs allocated context sizes.
Query params: days (default: 7) — lookback window for trace analysis.
Response:
{
"days": 7,
"models": [
{
"model": "gpt-oss:120b",
"allocated_ctx": 131072,
"override_ctx": 16384,
"request_count": 67234,
"prompt_tokens": {"avg": 1288, "p50": 833, "p75": 1322, "p95": 3706, "p99": 4797, "max": 10938},
"total_tokens": {"p95": 4120, "p99": 5409, "max": 34721, "max_24h": 8675},
"utilization_pct": 4.1,
"recommended_ctx": 16384,
"savings_pct": 87.5
}
]
}Model mix recommendations for the fleet based on hardware, usage patterns, and curated benchmark data. Results are cached for 5 minutes.
Query parameters:
| Parameter | Type | Default | Description |
|---|---|---|---|
refresh |
int | 0 |
Pass 1 to force re-analysis (bypasses cache) |
Response:
{
"nodes": [
{
"node_id": "mac-studio",
"total_ram_gb": 512.0,
"usable_ram_gb": 506.0,
"current_models": ["gpt-oss:120b", "qwen3:235b-a22b"],
"recommendations": [
{
"model": "gpt-oss:120b",
"display_name": "GPT-OSS 120B",
"category": "reasoning",
"ram_gb": 72.0,
"quality_score": 89.0,
"reason": "Currently your most-used reasoning model (4303 requests in 24h)",
"priority": "high",
"already_available": true
}
],
"total_recommended_ram_gb": 313.0,
"ram_headroom_gb": 193.0
}
],
"usage": {
"total_requests_24h": 4484,
"category_breakdown": {"reasoning": 4390, "general": 52, "coding": 42},
"top_models": [{"model": "gpt-oss:120b", "requests": 4390, "category": "reasoning"}],
"category_coverage": {"general": true, "coding": true, "reasoning": true, "creative": true, "fast-chat": true}
},
"fleet_summary": "1 node(s), 512GB total RAM, 6 model(s) recommended using 313GB",
"uncovered_categories": [],
"generated_at": 1741700000.0
}The recommender uses a curated catalog of 30+ models with benchmark data (MMLU, HumanEval, MT-Bench) and considers: available RAM per node (with 6GB OS overhead), last 24h request patterns, models already downloaded, cross-fleet distribution (avoids redundant large models), and a 50% RAM cap per model to ensure variety.
Pull a model onto a specific node via Ollama's /api/pull API. Blocks until the pull completes (may take minutes for large models).
Request body:
{
"node_id": "mac-studio",
"model": "codestral:22b"
}Response:
{"ok": true, "node_id": "mac-studio", "model": "codestral:22b"}Returns {"ok": false, ...} if the pull fails. The router proxies the pull request to the target node's Ollama instance using the same HTTP client used for inference (600s read timeout).
Delete a model from a specific node via Ollama's DELETE /api/delete API.
Request body:
{
"node_id": "mac-studio",
"model": "qwen3-coder-480b:latest"
}Response:
{"ok": true, "node_id": "mac-studio", "model": "qwen3-coder-480b:latest"}Returns {"ok": false, ...} if the delete fails. This permanently removes the model from the node's disk — it must be re-downloaded to use again.
Return the pin state that feeds the Ollama preloader. Env-level pins (from FLEET_PINNED_MODELS) are fleet-wide and read-only here; per-node pins live in <data_dir>/pinned_models.json and are managed via POST /dashboard/api/pinned-models.
Response:
{
"env_pins": ["gpt-oss:120b"],
"per_node": {
"Neons-Mac-Studio": ["gemma3:27b", "gpt-oss:120b"]
},
"max_count": 3
}max_count is FLEET_MODEL_PRELOAD_MAX_COUNT; if the sum of pins on any node exceeds it, the preloader truncates and logs a warning — raise the env var only if your backend can actually hold more concurrent models (macOS Ollama 0.20.4 caps at 3 regardless).
Toggle a per-node pin. Writes atomically to <data_dir>/pinned_models.json. The model preloader re-reads the file every refresh cycle (10 min), so dashboard toggles take effect within that window.
Request body:
{"node_id": "Neons-Mac-Studio", "model": "gemma3:27b", "pinned": true}Response:
{"ok": true, "per_node": {"Neons-Mac-Studio": ["gemma3:27b", "gpt-oss:120b"]}}Validation:
node_idandmodelrequired — 400 if either is empty.vision-embeddingcategory models (DINOv2, SigLIP, CLIP) are rejected whenpinned=true— those run on the embedding service, not Ollama, so a preloader pin would be meaningless. Unpin (pinned=false) always succeeds so stale entries can be cleaned up.
Per-node model details with disk sizes, usage statistics, and last-used timestamps for the model management UI.
Response:
{
"nodes": [
{
"node_id": "mac-studio",
"models": [
{
"name": "gpt-oss:120b",
"display_name": "GPT-OSS 120B",
"category": "reasoning",
"size_gb": 60.9,
"parameter_size": "120B",
"quantization": "Q4_K_M",
"last_used": 1741700000.0,
"days_unused": 0.1,
"total_requests": 18065,
"loaded_in_vram": true,
"unused": false
}
],
"total_size_gb": 959.0,
"disk_available_gb": 2994.0,
"disk_total_gb": 3722.0
}
]
}Models are sorted by: loaded in VRAM first, then by last-used descending, then alphabetically. The unused flag is true if the model has never been used through the router or hasn't been used in 7+ days.
Current router configuration, toggleable settings, and registered node list with versions.
Response:
{
"router_version": "0.3.0",
"router_hostname": "Neons-Mac-Studio",
"config": {
"toggles": { "auto_pull": true, "image_generation": false, "transcription": false, "dynamic_num_ctx": false },
"server": { "host": "Neons-Mac-Studio", "port": 11435, "data_dir": "~/.fleet-manager", "max_retries": 2 },
"heartbeat": { "heartbeat_interval": 5.0, "heartbeat_timeout": 15.0, "heartbeat_offline": 30.0 },
"scoring": { "score_model_hot": 50.0, "score_model_warm": 30.0, "..." : "..." },
"rebalancer": { "rebalance_interval": 5.0, "rebalance_threshold": 4, "rebalance_max_per_cycle": 3 },
"pre_warm": { "pre_warm_threshold": 3, "pre_warm_min_availability": 0.6 },
"auto_pull_config": { "auto_pull_timeout": 300.0 },
"context_protection": { "context_protection": "strip" }
},
"nodes": [
{
"node_id": "Neons-Mac-Studio",
"status": "online",
"agent_version": "0.3.0",
"ip": "http://localhost:11434",
"models_loaded_count": 2,
"is_router": true
}
]
}Toggle runtime-mutable boolean settings: auto_pull, image_generation, transcription, dynamic_num_ctx, num_ctx_auto_calculate, fleet_intelligence; all other fields are ignored ("status": "no_change"). Changes take effect immediately and are written to ~/.fleet-manager/env, so they survive a restart.
Request:
{"auto_pull": false}Response:
{"status": "updated", "updated": {"auto_pull": false}}HTML page for the Settings dashboard tab. Shows router info, toggle switches, read-only configuration tables grouped by category, and a node list with version tracking.
Read + control endpoints for clients that need to reason about capacity or pin
models (e.g. a benchmark that must hold hardware constant). See
docs/plans/client-ergonomics-from-agent-feedback.md.
Full fleet state — every node's hardware/cpu/memory/ollama/mlx status plus
derived models_loaded_count and free_slots per node, and all queue info.
Lightweight queue depths + estimated wait, for high-frequency client backoff.
Effective serving constraints so a client can auto-serialize instead of
self-DoSing: hot_model_cap, per-node free_slots, router max_retries,
and the MLX in-flight caps. Firing more concurrent requests than the box can
absorb just produces 429s (or Ollama queue-full 503s) — read this first.
Pre-warm a model resident and persist the pin (reloaded if evicted).
Body: {"model": "<name>", "node_id": "<optional>", "wait": <bool>, "timeout_s": <float>}.
One call replaces the manual ollama … keep_alive dance. Returns 404 if the
model isn't on disk on any online node.
wait (default false) — when true, the call blocks until the router's
routing view confirms the model resident before returning, so a caller that
pins-then-scans doesn't race the heartbeat and get fallback-substituted. The
pre-warm itself already blocks through the Ollama load; wait additionally
waits out the heartbeat-reflection lag (~5s, the window where the model is
loaded but routing hasn't seen it yet). timeout_s (default 30) caps that
wait. Response adds "ready": true|false and "ready_after_ms": <int>. On
timeout, ready: false (the model may still become resident shortly — the
client learns the truth instead of racing).
mlx: models are always-resident subprocesses (configured via
FLEET_NODE_MLX_SERVERS), so pinning is a no-op: the call skips warming and
reports ready from the MLX server's health, with an explanatory note.
A pin persists until you delete it. It is not scoped to your session, and nothing expires it. The preloader reloads pinned models unconditionally — no recency check, no budget — because you asked for them.
Pinned models must all stay resident simultaneously. Pin more than the box can hold and they evict each other, and the preloader reloads them in a loop, forever. A benchmark that pinned each model it tested and never unpinned reached 307 GB of pins (including a 290 GB model) on a 512 GB box and produced hours of ~300 GB disk→memory churn (2026-07-17).
If you're benchmarking, the pattern is:
POST /fleet/pin {"model": "X", "wait": true} # pin + wait until resident …run your requests against X… DELETE /fleet/pin/X # ← DO NOT SKIP THISSince 0.8.2 the herd refuses a pin that can't physically co-reside with your existing pins (
409, with the arithmetic and what's already pinned). That's a backstop, not a substitute for unpinning — checkGET /fleet/limitsandGET /fleet/status(pinnedper node) to see what you're holding.
Pin the model you're benchmarking (use wait: true to drop your own readiness
loop), then send requests serially against it (X-Fleet-Fallback: false
confirms it actually ran) — then unpin it.
Errors: 400 model missing / unknown node_id (the response lists
known_nodes) · 404 model not on disk anywhere · 409 pin would over-commit
the node (pass "force": true to override) · 503 on disk but wouldn't load
(usually insufficient free memory).
Release a pin so the model can be evicted normally. The {model} path segment
accepts / and : (e.g. mlx-community/Foo or qwen3-coder:30b).
| Endpoint | Description |
|---|---|
GET /favicon.svg |
SVG favicon (horse icon in brand purple) |
GET /favicon.ico |
Redirects to SVG favicon |
GET / |
Redirects to /dashboard |