Skip to content

Latest commit

 

History

History
2012 lines (1585 loc) · 71.1 KB

api-reference.md

File metadata and controls

2012 lines (1585 loc) · 71.1 KB

API Reference

Complete endpoint documentation for Ollama Herd.


OpenAI-Compatible Endpoints

POST /v1/chat/completions

OpenAI-compatible chat completions with streaming and non-streaming support.

Request body:

{
  "model": "llama3.3:70b",
  "messages": [
    {"role": "system", "content": "You are helpful."},
    {"role": "user", "content": "Hello!"}
  ],
  "stream": true,
  "temperature": 0.7,
  "max_tokens": 1024,
  "fallback_models": ["qwen2.5:32b", "qwen2.5:7b"],
  "metadata": {"tags": ["my-app", "production"]},
  "user": "alice"
}
Field Type Default Description
model string required Model name (must exist on at least one node)
messages array [] Chat messages in OpenAI format
stream boolean false Enable streaming (SSE)
temperature float 0.7 Sampling temperature
max_tokens integer null Maximum tokens to generate
fallback_models array [] Backup models to try if primary unavailable
metadata.tags array [] Tags for per-app analytics (e.g., ["my-app", "prod"])
user string null User identifier — stored as user:<value> tag

Request headers:

Header Description
X-Herd-Tags Comma-separated tags (alternative to metadata.tags in body)

Tags from all sources (body, header, user field) are merged and deduplicated. See Request Tagging for details.

Streaming response (stream: true):

data: {"id":"chatcmpl-abc123","choices":[{"delta":{"role":"assistant"},"index":0}]}\n\n
data: {"id":"chatcmpl-abc123","choices":[{"delta":{"content":"Hello"},"index":0}]}\n\n
data: {"id":"chatcmpl-abc123","choices":[{"delta":{},"finish_reason":"stop","index":0}]}\n\n
data: [DONE]\n\n

Non-streaming response (stream: false):

{
  "id": "chatcmpl-abc123def456",
  "object": "chat.completion",
  "created": 1710000000,
  "model": "llama3.3:70b",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "Hello! How can I help?"},
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 8,
    "total_tokens": 20
  }
}

Request headers (optional):

Header Effect
X-Fleet-No-Fallback: true Strict mode — serve the exact model requested or return an explicit error; never substitute a loaded model. Without it, a model that isn't loaded is loaded on a node with room, and only when every node would have to evict a resident model is it served by a loaded model of the same category (never another). (Also settable as "fallback": false in the body.)
X-Herd-Tags Comma-separated request tags for per-app analytics.

Response headers — the canonical X-Fleet-* set is emitted on every proxied response (all endpoints, streaming and non-streaming), so a client reads the same three fields anywhere and knows exactly what ran:

Header Always? Description
X-Request-Id ✅ The router's id for this request, the same value stored as the trace's request_id. Also on error responses once the router has given the request an id (a rejection, a failed attempt); a request refused before that, such as one with no model, has none. Pass it to GET /dashboard/api/traces?request_id= for the router's side of the request: node, retries, fallback, queue and route time. Image generation and transcription return it too, but write no trace when they complete
X-Fleet-Node ✅ Node ID that served the request
X-Fleet-Served-Model ✅ The model that actually ran (differs from requested on a fallback)
X-Fleet-Requested-Model ✅ What the client asked for
X-Fleet-Fallback ✅ true / false — whether a substitution happened (a real boolean; not a model name)
X-Fleet-Backend ✅ ollama / mlx / native / vision / image / transcription
X-Fleet-Retries ✅ Retry count (0 when none)
X-Fleet-Score scorer-routed Winning routing score (integer)
X-Fleet-Context-Overflow on overflow estimated_tokens=N; context_length=M when input exceeds the node's context window. Ollama truncates such input and still answers 200; when the backend's count confirms it, the router also raises the input_truncated health check
X-Fleet-Overhead-Ms non-streaming chat Herd's own time for the request: its total time minus Ollama's own total_duration (routing, any wait in herd's queue, translation, the network hop to the node). Each trace's timings splits it.
X-Thinking-Tokens / X-Output-Tokens / X-Budget-Used / X-Done-Reason thinking models, non-streaming Reasoning-budget diagnostics

X-Fleet-Model is retired (0.8.1) — use X-Fleet-Served-Model.

Error responses:

Status Condition
400 Missing model field
404 Model not found on any node (auto-pull attempted first if FLEET_AUTO_PULL=true)
429 Per-client concurrency cap exceeded (FLEET_CLIENT_MAX_IN_FLIGHT); includes Retry-After
503 Model exists but no node can serve it right now

POST /v1/images/generations

OpenAI-compatible image generation. Wraps /api/generate-image in OpenAI's image API format.

Request body:

{
  "model": "z-image-turbo",
  "prompt": "a cat sitting on a laptop",
  "size": "1024x1024",
  "response_format": "b64_json"
}
Field Type Default Description
model string required Image model name
prompt string required Text description of the image
size string 1024x1024 Image dimensions (WIDTHxHEIGHT)
response_format string b64_json b64_json returns base64 PNG; url returns raw PNG bytes
steps integer model default Inference steps
guidance float model default Guidance scale
seed integer random Seed for reproducibility
negative_prompt string "" What to avoid in the image

Response (b64_json):

{
  "created": 1710000000,
  "data": [
    {
      "b64_json": "iVBORw0KGgo...",
      "revised_prompt": "a cat sitting on a laptop"
    }
  ]
}

Response (url): Raw PNG bytes with Content-Type: image/png.

Error responses: 400 (missing fields), 404 (model not available), 502 (generation failed), 503 (disabled).


GET /v1/models

List all models across the fleet — LLM, image, and embedding models (loaded + available on disk).

Response:

{
  "object": "list",
  "data": [
    {
      "id": "llama3.3:70b",
      "object": "model",
      "created": 1710000000,
      "owned_by": "ollama"
    },
    {
      "id": "qwen2.5:7b",
      "object": "model",
      "created": 1710000000,
      "owned_by": "ollama"
    }
  ]
}

POST /v1/embeddings

OpenAI-compatible embeddings, for clients that only speak OpenAI (Chatbox Knowledge Base, LangChain/LlamaIndex OpenAI embedders, generic "OpenAI-compatible" settings). It is a translation layer over the same path as POST /api/embed: nomic-embed-text still goes to the native fastembed server, vision embedding models to the vision server, and everything else is scored, retried and traced exactly as on /api/embed.

Request:

{
  "model": "nomic-embed-text",
  "input": ["first document", "second document"],
  "encoding_format": "float"
}
Field Type Default Description
model string required Embedding model name (any name /api/embed accepts)
input string or array of strings required Text(s) to embed. Empty strings and token-id arrays are rejected with 400.
encoding_format "float" / "base64" "float" base64 is little-endian float32, as OpenAI encodes it. Anything else is 400.
dimensions integer — Forwarded to the backend. If the backend returns a different width (it ignored the parameter), the request fails with 400 rather than returning vectors of the wrong size.

Response:

{
  "object": "list",
  "data": [
    {"object": "embedding", "index": 0, "embedding": [0.0101, -0.0018, ...]},
    {"object": "embedding", "index": 1, "embedding": [0.0207, 0.0114, ...]}
  ],
  "model": "nomic-embed-text",
  "usage": {"prompt_tokens": 6, "total_tokens": 6}
}

usage is the backend's prompt_eval_count (the fastembed server reports a word-count approximation; 0 if the backend reports nothing). The X-Fleet-* headers from the embed path are preserved.

Errors use OpenAI's shape, {"error": {"message", "type", "param", "code"}}: 400 invalid_request_error, 404 with code: "model_not_found", 503/504 model_overloaded, 502 api_error (including a backend that returned a different number of vectors than inputs).


POST /v1/responses

The OpenAI Responses API — what OpenAI Codex speaks. Codex removed Chat Completions support in Feb 2026 (wire_api = "chat" is gone), so this is the only endpoint current Codex can use. Setup: guides/codex-integration.md.

Request:

{
  "model": "gpt-5-codex",
  "input": [
    {"role": "user", "content": [{"type": "input_text", "text": "fix the bug"}]}
  ],
  "tools": [{"type": "function", "name": "bash", "parameters": {}}],
  "stream": true
}

input accepts a plain string or a list of items: message, function_call, function_call_output, custom_tool_call, custom_tool_call_output, reasoning, and additional_tools. Unrecognised item types are dropped and logged once per type.

Model selection — model may be any id. Resolution order: an explicit FLEET_ANTHROPIC_MODEL_MAP entry, then a real local model name, then best-loaded auto-routing, then 404. Requests carrying an image auto-route to a vision-capable model.

Tools — two shapes are accepted and both are translated:

  • a top-level tools array of plain functions, and
  • Codex's additional_tools input item, whose primary tool is a custom-typed exec taking raw JavaScript ("code mode"). Since codex-cli 0.164 its tools arrive inside namespace entries; members of the default functions namespace are exposed as ordinary tools, other namespaces are not exposed.

Herd also rewrites calls models reliably get wrong, because Codex's tool descriptions document an API its tool schema doesn't expose. A direct call to a tool nested inside exec (exec_command, apply_patch, …) becomes a custom_tool_call on exec of the form text(await tools.<name>(…)) — code mode returns only what a script passes to text(), and apply_patch takes the patch as a plain string. The nested tools are declared to the model as stubs so that Ollama can parse such a call at all; they are never returned to the client as tools. When exec_command is offered as a top-level tool (older Codex), an apply_patch tool call becomes an exec_command heredoc instead (apply_patch was a binary on the sandbox PATH there). Successful rewrites are logged at INFO (WARNING through 0.12.0); a call that cannot be rewritten logs WARNING.

Response (non-streaming) is a Responses object whose output is a list of items — an assistant message plus one function_call / custom_tool_call per tool call. Streaming emits the Responses SSE event sequence (response.output_item.added → …arguments.delta → response.completed).

Not supported: previous_response_id (server-side conversation state) — rejected with 400. Codex's default stateless mode resends the full input each turn, which is what Herd implements. mlx: models are skipped for this endpoint.

Response headers: same X-Fleet-* set as every other route, including X-Fleet-Served-Model.


POST /v1/rerank

Reranks documents against a query, for the rerank step of RAG pipelines (Open WebUI, Dify, LangChain, LlamaIndex). Ollama has no rerank endpoint. Herd serves this from the native fastembed server that nodes with the embedding extra already run, so there's no new dependency. Uses the Jina/Cohere request and response shape.

curl http://localhost:11435/v1/rerank -H 'Content-Type: application/json' -d '{
  "query": "how do I stop herd now that launchd is installed?",
  "documents": ["launchctl bootout gui/$UID/...", "pkill is no longer a stop", "MLX setup"],
  "top_n": 2,
  "return_documents": true
}'

Request: query (required) and documents (strings, or {"text": ...} objects). Optional: model, top_n, and return_documents (default false).

Response: {model, results: [{index, relevance_score, document?}], usage: {total_tokens}}. Results are sorted best first, and index points into the original documents. relevance_score is the cross-encoder logit passed through a sigmoid. It's in 0–1 like Cohere's and Jina's, so clients that apply a relevance threshold behave the same way. The serving node is in X-Fleet-Node, not the body.

Models. Weights download on first use, then stay cached.

model Size Notes
ms-marco-minilm-l-6-v2 (default) 80 MB Fastest; English
ms-marco-minilm-l-12-v2 120 MB English
jina-reranker-v1-tiny-en 130 MB English, long inputs
jina-reranker-v1-turbo-en 150 MB English, long inputs
bge-reranker-base 1.04 GB Quality pick; English/Chinese
jina-reranker-v2-base-multilingual 1.11 GB Quality pick; multilingual

Limits: at most 1,000 documents of 32,000 characters each → 400. A cross-encoder runs one forward pass per document.

Errors:

  • 400: invalid request, from the node.
  • 404: unknown model; the error lists the available ones.
  • 503: no online node can serve it right now. With a Retry-After header, the node that runs the reranker is restarting or offline, or the router has just restarted and has not heard from its nodes yet: retry. Without one, no node runs a rerank-capable text server at all (uv sync --extra embedding).
  • 504: timed out, usually because a first request is still downloading weights.

Traces carry the rerank tag.


POST /v1/systemone

Ollama's decision models (Ollama ≥ 0.35: nimble, tev1), which return choices with probabilities instead of text. Use them for triage, classification, and model routing. Herd passes the request through unchanged to a node whose Ollama reports the decision capability for the model. Nodes on older Ollama can't report that capability, so they're skipped.

curl http://localhost:11435/v1/systemone -H 'Content-Type: application/json' -d '{
  "model": "nimble",
  "state": "Our checkout has returned 500 errors since 9am.",
  "questions": {"label": {"type": "choice", "instructions": "Which label fits?",
    "criteria": {"billing": "Payments", "bug": "Software errors", "account": "Login"}}}
}'

Response: Ollama's, unchanged. Example: answers.label.choice == "bug", with probabilities and confidence, plus usage.

Limits: these are Ollama's, enforced at the router before any network hop: the body may not exceed 64 KiB unless images is set, and never 32 MiB → 413.

Errors:

  • 404: no node serves that decision model. Needs Ollama ≥ 0.35 and a local GGUF decision model (ollama pull nimble). Recorded as a rejected trace.
  • 400: passed through from Ollama, for example a :cloud or MLX model, since System One only accepts local GGUF models.

Traces carry the decision tag. Node selection and failover are the same as /api/show: loaded nodes first, and the winning node is in X-Fleet-Node.


Calling the router from a browser

Every endpoint below works unchanged from desktop apps, CLIs and server-side code. Browser-based clients are the exception: the browser sends a preflight OPTIONS and refuses to hand the page any response without Access-Control-* headers. The router emits none by default, so a page-based client gets a CORS error even though the request itself would have succeeded.

Set FLEET_CORS_ORIGINS to an allow-list to enable it — same syntax as OLLAMA_ORIGINS. All X-Fleet-* response headers are exposed to the page, since CORS otherwise hides them from exactly the clients this is for. Full detail, including why nothing is allowed implicitly, is in Configuration Reference § Browser clients and CORS.


Ollama-Compatible Endpoints

HEAD /

Liveness probe, returns 200 with an empty body, like Ollama. OllamaKit (Ollamac, Enchanted) checks reachability with exactly this request and requires a 2xx. GET / is unchanged: it redirects browsers to /dashboard.

POST /api/chat

Ollama-compatible chat endpoint. Streaming is enabled by default (matches Ollama behavior).

Request body:

{
  "model": "llama3.3:70b",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ],
  "stream": true,
  "options": {
    "temperature": 0.7,
    "num_predict": 1024
  },
  "fallback_models": ["qwen2.5:32b"],
  "metadata": {"tags": ["my-app"]}
}
Field Type Default Description
model string required Model name
messages array [] Chat messages
stream boolean true Streaming (NDJSON)
options.temperature float 0.7 Sampling temperature
options.num_predict integer null Max tokens
fallback_models array [] Backup models
metadata.tags array [] Tags for per-app analytics
user string null User identifier — stored as user:<value> tag

The metadata and fallback_models fields are stripped before proxying to Ollama. The X-Herd-Tags header is also supported (same as OpenAI endpoint).

Streaming response (NDJSON, one JSON object per line):

{"message":{"role":"assistant","content":"Hello"},"done":false}
{"message":{"role":"assistant","content":"!"},"done":false}
{"message":{"role":"assistant","content":""},"done":true,"total_duration":1234567890,"prompt_eval_count":12,"eval_count":8}

Non-streaming response:

{
  "message": {"role": "assistant", "content": "Hello! How can I help?"},
  "done": true,
  "total_duration": 1234567890,
  "prompt_eval_count": 12,
  "eval_count": 8
}

POST /api/generate

Ollama-compatible generate endpoint. Uses prompt instead of messages.

Request body:

{
  "model": "llama3.3:70b",
  "prompt": "Why is the sky blue?",
  "stream": true,
  "options": {
    "temperature": 0.7
  }
}

Supports the same metadata.tags, user, and X-Herd-Tags header as /api/chat.

Response format is the same as /api/chat.


GET /api/tags

List all models across the fleet with node information. Includes LLM models, mlx: models, image models (mflux + DiffusionKit) and vision embedding models.

Each entry has Ollama's full field set, plus Herd's details.fleet_nodes. Strict clients depend on this: OllamaKit (the library behind Enchanted and Ollamac) and Reins decode modified_at, digest and every details string as required and non-null, so one missing key fails the whole listing and the model picker comes up empty.

Response:

{
  "models": [
    {
      "name": "gpt-oss:120b",
      "model": "gpt-oss:120b",
      "modified_at": "2026-04-24T05:29:24.844674018-03:00",
      "size": 65369818941,
      "digest": "a951a23b46a1f6093dafee2ea481d634b4e31ac720a8a16f3f91e04f5a40ecd9",
      "details": {
        "parent_model": "",
        "format": "gguf",
        "family": "gptoss",
        "families": ["gptoss"],
        "parameter_size": "116.8B",
        "quantization_level": "MXFP4",
        "fleet_nodes": ["mac-studio-ultra", "macbook-pro-m4"]
      }
    },
    {
      "name": "mlx:Qwen3-Coder-Next-4bit",
      "model": "mlx:Qwen3-Coder-Next-4bit",
      "modified_at": "2026-09-29T17:04:11.482113Z",
      "size": 44800000000,
      "digest": "3f0c…(sha256 of the name, see below)",
      "details": {
        "parent_model": "",
        "format": "mlx",
        "family": "",
        "families": [],
        "parameter_size": "",
        "quantization_level": "",
        "fleet_nodes": ["mac-studio-ultra"]
      }
    },
    {
      "name": "z-image-turbo",
      "model": "z-image-turbo",
      "modified_at": "2026-09-29T17:04:11.482200Z",
      "size": 0,
      "digest": "…",
      "details": {
        "parent_model": "", "format": "mflux", "family": "", "families": [],
        "parameter_size": "", "quantization_level": "",
        "fleet_nodes": ["mac-studio-ultra"],
        "type": "image"
      }
    }
  ]
}

Where the values come from:

  • Ollama models: passed through unchanged from the node's own Ollama /api/tags. The node agent carries them in its heartbeat (ollama.models_available_meta), but only when they change plus once a minute as a refresh, since for a 50-model node they are about 13.6 KB, 15x the rest of the Ollama section. Ollama's own nulls (it sends "families": null for some models) are normalized to [] / "".
  • Models with no Ollama metadata (mlx: models, image and vision-embedding models, or any model on a node running an agent older than this field) get non-null defaults instead of null:
    • details.format: "mlx" for mlx: models, the runtime ("onnx") for vision embedding models, the generator ("mflux", "diffusionkit") for image models, and "" when unknown. Herd does not guess "gguf".
    • family, parameter_size, quantization_level, parent_model: "". families: [].
    • modified_at: the time this router process first listed the model (RFC3339 with fractional seconds, UTC). Stable across calls, so a list sorted by date does not reshuffle on every poll; it resets when the router restarts.
    • digest: sha256("ollama-herd:<name>"). It is not a content digest. It is unique and stable so clients that key their lists on digest don't collapse every such model into one row.
  • size is the on-disk size in bytes, as in Ollama, when any node reports it; it falls back to the resident size for a loaded model with no disk size, and 0 otherwise.
  • fleet_nodes lists every node that has the model (loaded or on disk). Image models add "type": "image"; vision embedding models add "type": "vision-embedding", runtime and dimensions.

GET /api/ps

List all currently loaded (hot) models across the fleet.

Response:

{
  "models": [
    {
      "name": "llama3.3:70b",
      "model": "llama3.3:70b",
      "size": 42949672960,
      "fleet_node": "mac-studio-ultra"
    },
    {
      "name": "qwen2.5:7b",
      "model": "qwen2.5:7b",
      "size": 4294967296,
      "fleet_node": "macbook-air-m2"
    }
  ]
}

POST /api/show

Ollama-compatible model details. Ollama's desktop app calls it before every chat, AnythingLLM reads the context window (model_info["<arch>.context_length"]) and tool support (capabilities) from it, and Cherry Studio's "Check" button probes it.

Request body: {"model": "gpt-oss:120b"}. The legacy {"name": ...} field is also accepted, and a bare name resolves to :latest as in Ollama. Other fields (e.g. verbose) are forwarded.

Behavior:

Model Response
Ollama model Proxied to a node that has it (nodes with it loaded first, then on disk). That node's Ollama response is returned unchanged, with X-Fleet-Node. If a node is unreachable or errors, the next one is tried.
mlx: model Synthesized: details.format: "mlx", capabilities: ["completion"], empty modelfile / parameters / template, and an empty model_info (Herd doesn't know the context length, so it doesn't invent one).
Vision embedding model Synthesized, capabilities: ["embedding"].
Image model Synthesized, capabilities: ["image"].
Unknown 404 {"error": "model '<name>' not found"}, matching Ollama.

Error responses: 400 (missing model / invalid JSON), 404 (unknown model, or every node that should have it answered 404), 502 (no node holding the model was reachable).


POST /api/embed

POST /api/embeddings

Ollama-compatible embeddings endpoint. The router dispatches to the right backend based on model name — clients need no changes:

Model type Backend Port
dinov2-vit-s14, clip, siglip Native vision embedding (ONNX) :11438
nomic-embed-text, nomic-embed-text:latest Native text embedding (fastembed) :11439
All other models Ollama on the selected node :11434

Vision embedding input is base64 image bytes in images (or input), and must be JPEG, PNG or WebP. Any other format, including GIF, TIFF, BMP and EPS, gets 400 {"error": "Image N is not one of: JPEG, PNG, WEBP", "node": "<node>"} before any decoder runs: those decoders carry most of Pillow's security advisories, and the server takes unauthenticated bytes from the LAN.

The native text embedding backend (fastembed / ONNX Runtime) runs entirely outside Ollama's inference queue — nomic-embed-text requests never consume OLLAMA_NUM_PARALLEL slots and can never be stalled by concurrent LLM inference.

Request body:

{
  "model": "nomic-embed-text",
  "input": "The quick brown fox jumps over the lazy dog"
}
Field Type Default Description
model string required Embedding model name
input string/array — Text(s) to embed (string or ["text1", "text2"])
prompt string — Alias for input (legacy compat)
metadata.tags array [] Tags for per-app analytics

Task prefixes for nomic-embed-text: the model supports "search_query: ..." and "search_document: ..." prefixes for asymmetric retrieval quality. These are the caller's responsibility — neither herd nor Ollama adds them automatically.

Response (Ollama-compatible for all backends):

{
  "model": "nomic-embed-text",
  "embeddings": [[0.0101, -0.0018, ...]],
  "total_duration": 4500000,
  "load_duration": 0,
  "prompt_eval_count": 9
}

Error responses:

Status Condition
400 Missing model or input field
404 Model not found on any node
503 No online node can serve it. With Retry-After: the node running the native server is restarting or offline, or the router has just restarted. It is transient, so retry. Without it: no node runs the native server; install fastembed (uv sync --extra embedding). /v1/embeddings passes Retry-After through.
504 Text embedding timed out — first request may trigger 130 MB model download; retry after 30s

POST /api/pull

Pull a model onto the fleet. Ollama-compatible — streams NDJSON progress matching Ollama's wire format. Auto-selects the node with the most available memory, or accepts an explicit node_id.

Request body:

Field Type Default Description
name string — Model to pull (Ollama standard field)
model string — Model to pull (alias for name — many agents send this)
stream bool true Stream NDJSON progress or return on completion
node_id string auto Target node (auto-selects best node if omitted)

Streaming response (stream: true, default):

{"status":"pulling manifest"}
{"status":"pulling abc123...","digest":"sha256:abc123","total":5000000,"completed":2500000}
{"status":"verifying sha256:abc123"}
{"status":"writing manifest"}
{"status":"success"}

Non-streaming response (stream: false):

{"status": "success"}

Response headers:

Header Description
X-Fleet-Node Node that received the pull

Error responses:

Status Condition
400 Missing model name, or model is a non-Ollama model (mflux, DiffusionKit, MLX) — returns install instructions
404 Specified node_id not found or offline
409 Model is already being pulled
503 No node has enough available memory

Non-Ollama models: Image generation models (z-image-turbo, flux-dev, sd3-medium, sd3.5-large) and speech-to-text models (qwen3-asr) are not Ollama models and cannot be pulled via this endpoint. The error response includes the correct install command for each.

Examples:

# Pull a model (streams NDJSON progress)
curl -N http://localhost:11435/api/pull -d '{"name": "codestral"}'

# Pull to a specific node
curl -N http://localhost:11435/api/pull -d '{"name": "llama3.3:70b", "node_id": "mac-studio"}'

# Non-streaming (blocks until complete)
curl http://localhost:11435/api/pull -d '{"name": "phi4", "stream": false}'

GET /api/image-models

List all image models across the fleet (mflux, DiffusionKit, and Ollama native).

Response:

{
  "models": [
    {
      "name": "z-image-turbo",
      "type": "image",
      "backend": "mflux",
      "fleet_nodes": ["mac-studio-ultra"]
    },
    {
      "name": "sd3-medium",
      "type": "image",
      "backend": "mflux",
      "fleet_nodes": ["macbook-pro-m4"]
    },
    {
      "name": "x/z-image-turbo:latest",
      "type": "image",
      "backend": "ollama",
      "fleet_nodes": ["mac-studio-ultra"]
    }
  ]
}

The backend field indicates which system handles the model: mflux (mflux/DiffusionKit via port 11436) or ollama (Ollama native image models with x/ prefix).


Anthropic-Compatible Endpoints

Native Anthropic Messages API surface for Claude Code and other anthropic-SDK clients. See docs/guides/claude-code-integration.md for the full setup walkthrough.

POST /v1/messages

Anthropic Messages endpoint — streaming + non-streaming, full tool use, system prompts, stop sequences.

Request body:

{
  "model": "claude-sonnet-4-5",
  "max_tokens": 4096,
  "messages": [
    {"role": "user", "content": "What's the weather in Paris?"}
  ],
  "system": "You are a helpful assistant.",
  "tools": [{
    "name": "get_weather",
    "description": "Get current weather for a city",
    "input_schema": {
      "type": "object",
      "properties": {"city": {"type": "string"}},
      "required": ["city"]
    }
  }],
  "tool_choice": {"type": "auto"},
  "stream": false,
  "temperature": 0.7,
  "stop_sequences": ["\n\nHuman:"],
  "metadata": {"user_id": "alice"}
}

Headers (recommended):

  • x-api-key: <key> — required when FLEET_ANTHROPIC_REQUIRE_KEY=true
  • anthropic-version: 2023-06-01 — reflected back in response headers

Model resolution: by default claude-* ids auto-route to the best local model you have loaded for that tier — no configuration required. FLEET_ANTHROPIC_MODEL_MAP is optional and pins specific aliases to specific models (its entries win over auto-routing). Real local model names (e.g. qwen3-coder:30b) pass through unchanged. The response's X-Fleet-Served-Model header reports what actually ran. See anthropic-auto-routing.md and configuration-reference.md.

Non-streaming response:

{
  "id": "msg_abc123...",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-4-5",
  "content": [
    {"type": "text", "text": "I'll check that for you."},
    {"type": "tool_use", "id": "toolu_xyz...", "name": "get_weather", "input": {"city": "Paris"}}
  ],
  "stop_reason": "tool_use",
  "stop_sequence": null,
  "usage": {"input_tokens": 25, "output_tokens": 18}
}

Stop reasons: end_turn, max_tokens, stop_sequence, tool_use.

Streaming SSE events: message_start → content_block_start/delta/stop (one set per text or tool_use block) → message_delta → message_stop. Tool calls open a new content_block_start of type tool_use mid-stream and emit args via input_json_delta.

Tool result round-trip: Send a follow-up request with the assistant's tool_use block in messages and a user message containing a tool_result block referencing the same tool_use_id:

{
  "role": "user",
  "content": [{"type": "tool_result", "tool_use_id": "toolu_xyz...", "content": "18C, sunny"}]
}

POST /v1/messages/count_tokens

Token estimate for a Messages payload — used by Claude Code for budget gating before each turn. Best-effort: tiktoken cl100k if installed, otherwise len(text)/4. Not for billing.

Request body: same shape as /v1/messages.

Response:

{"input_tokens": 142}

GET /v1/messages

Friendly probe endpoint — Claude Code may GET this during connection setup.

Response:

{"ok": true, "service": "ollama-herd", "endpoint": "/v1/messages", "ts": 1776857000}

Response headers (all /v1/messages calls)

  • X-Request-Id — the router's id for this request (the trace's request_id)
  • X-Fleet-Node — node id that served the request
  • X-Fleet-Score — routing score (higher = better fit)
  • X-Fleet-Fallback — present when a fallback model was used
  • X-Fleet-Retries — present when the request was retried
  • anthropic-version — echoed from the request

Error responses

The route emits the following non-2xx status codes in addition to the usual 400/401/5xx from upstream. All error bodies follow the Anthropic error shape: {"type": "error", "error": {"type": "...", "message": "..."}}.

  • 413 Payload Too Large — error.type: "prompt_too_large". Returned in two situations:

    • Pre-inference hard cap. After server-side context management (tool-result clearing + LLM compaction with force_all), the prompt still exceeds FLEET_ANTHROPIC_MAX_PROMPT_TOKENS (default 180K). The message tells the caller to run /compact in Claude Code and resubmit.
    • MLX wall-clock timeout. An MLX request exceeded FLEET_MLX_WALL_CLOCK_TIMEOUT_S (default 300s) from admission → last byte. The slot is released; the caller gets the same try /compact hint. Catches wedged-request syndrome where mlx_lm.server emits tokens slowly but never hits a stop condition.

    The server never silently retries either case — correctness of agentic tool-use workflows depends on the client owning the decision to resubmit with altered context.

  • 503 Service Unavailable with Retry-After — MLX admission control rejected the request (queue full). error.type: "overloaded_error". Clients should back off and retry after the header value.


Fleet Management

GET /fleet/status

Full fleet state — nodes, queues, hardware metrics, and health summary.

Response:

{
  "fleet": {
    "nodes_total": 3,
    "nodes_online": 2,
    "nodes_degraded": 0,
    "models_loaded": 4,
    "requests_active": 1
  },
  "nodes": [
    {
      "node_id": "mac-studio-ultra",
      "status": "online",
      "hardware": {
        "memory_total_gb": 192,
        "cores_physical": 24,
        "chip": "Apple M3 Ultra",
        "memory_bandwidth_gbps": 819.0,
        "arch": "apple_silicon"
      },
      "ollama_url": "http://10.0.0.100:11434",
      "cpu": {
        "cores_physical": 24,
        "utilization_pct": 15.2
      },
      "memory": {
        "total_gb": 192.0,
        "used_gb": 45.3,
        "available_gb": 146.7,
        "pressure": "normal",
        "wired_gb": 12.1,
        "compressed_gb": 1.4,
        "swap_used_gb": 0.0,
        "swap_total_gb": 0.0
      },
      "ollama": {
        "models_loaded": [
          {"name": "llama3.3:70b", "size_gb": 40.0}
        ],
        "models_available": ["llama3.3:70b", "qwen2.5:32b"],
        "requests_active": 1
      },
      "mlx_servers": [
        {
          "port": 11440,
          "model": "mlx-community/Qwen3-Coder-Next-4bit",
          "status": "healthy",
          "status_reason": "",
          "kv_bits": 8,
          "model_size_gb": 41.8,
          "last_ok_ts": 1714000000.0
        },
        {
          "port": 11441,
          "model": "mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit",
          "status": "healthy",
          "status_reason": "",
          "kv_bits": 8,
          "model_size_gb": 16.0,
          "last_ok_ts": 1714000000.0
        }
      ],
      "mlx_bind_host": "0.0.0.0",
      "vision_embedding_status": {
        "backend_available": true,
        "cached_model_count": 3
      },
      "text_embedding": {
        "models_available": [
          {"name": "nomic-embed-text", "dimensions": 768, "cached": true}
        ],
        "processing": false
      },
      "text_embedding_port": 11439,
      "text_embedding_status": {
        "backend_available": true,
        "cached_model_count": 1
      }
    }
  ],
  "queues": {
    "mac-studio-ultra:llama3.3:70b": {
      "node_id": "mac-studio-ultra",
      "model": "llama3.3:70b",
      "pending": 0,
      "in_flight": 1,
      "completed": 42,
      "failed": 0,
      "concurrency": 8
    }
  },
  "timestamp": 1710000000.0
}

fleet.nodes_online counts every node requests are routed to: online plus degraded (a late heartbeat, not yet offline). fleet.nodes_degraded is the part of that which is late.

memory field:

Field Type Description
total_gb / used_gb / available_gb float System memory (psutil)
pressure normal | warn | critical On macOS the kernel's own level (kern.memorystatus_vm_pressure_level). Critical withholds cold loads; resident models keep serving
wired_gb float Wired (non-pageable) memory
compressed_gb float RAM the macOS compressor occupies (vm.compressor_bytes_used), not the larger amount it holds logically. 0 off macOS
swap_used_gb / swap_total_gb float Swap in use / allocated. On macOS swap grows on demand, so used/total is ~100% whenever any exists; compare swap_used_gb to total_gb instead

The swap and compressed fields are 0 from agents at 0.10.0 or earlier, meaning "not reported", not "healthy". Pressure drops back to normal once paging settles, even with tens of GB still swapped out, so read both. Related health checks: memory_pressure (WARNING/CRITICAL from pressure), and swap_usage (WARNING when swap_used_gb reaches half of total_gb: memory committed well beyond RAM, so models or other apps are paged out).

mlx_servers field (optional — present only on nodes with MLX configured). Note context_length is null, not 0: mlx_lm.server is OpenAI-native, so herd can neither set a window per request nor ask which one the server launched with. null means "cannot be known"; a 0 would read as a real zero-length window. It is populated only if a future mlx_lm reports one:

One entry per mlx_lm.server subprocess on that node. Lets operators and downstream tooling see per-URL health without polling each port individually.

Field Type Description
port int Port the server listens on (unique per-node)
model string HF repo id or local path (without mlx: prefix)
status string healthy / starting / unhealthy / memory_blocked / quarantined / stopped
status_reason string Human-readable detail (empty when healthy) — e.g. "memory gate: 42.0 GB needed, 16 GB available", "subprocess exited rc=1", or "5 crashes within 300s; next restart in 600s"
kv_bits int 0 / 4 / 8 — KV quantization level (requires setup-mlx.sh patch)
model_size_gb float Estimated weight size from HF cache on disk
last_ok_ts float Epoch seconds of last successful /v1/models poll

mlx_bind_host at the node top level is what the servers bind to (127.0.0.1 local-only; 0.0.0.0 LAN-reachable). Nodes running older agents that predate multi-MLX return an empty mlx_servers list and omit mlx_bind_host.

vision_embedding_status field (optional — present on node agents 0.6.1+):

Carries the asymmetry between "weights cached on disk" and "backend can serve them" so the dashboard can stop falsely advertising chips when onnxruntime isn't installed in the herd-node venv.

Field Type Description
backend_available bool True iff onnxruntime is importable in the herd-node venv at the moment of the last heartbeat
cached_model_count int Number of vision embedding models (DINOv2 / SigLIP / CLIP) with weights present in ~/.cache/huggingface/hub/

The vision_backend_missing health check fires WARNING when backend_available=false AND cached_model_count>0 — i.e., the operator intended to use vision embedding (downloaded the weights) but the runtime isn't installed. Fix: uv sync --extra embedding (or uv sync --all-extras) on the node, then restart herd-node.

text_embedding / text_embedding_port / text_embedding_status fields (optional — present on node agents 0.7.0+):

The native fastembed text embedding server. When running, nomic-embed-text requests are routed to port text_embedding_port (default 11439) instead of Ollama, so embed requests never consume OLLAMA_NUM_PARALLEL inference slots.

text_embedding.models_available lists registered models with name, dimensions, and cached (whether the ONNX weights are on disk). cached=false means the server is running but the model hasn't been requested yet — it will auto-download (130 MB) on the first request.

text_embedding_status carries the same asymmetry signal as vision_embedding_status:

Field Type Description
backend_available bool True iff fastembed is importable in the herd-node venv
cached_model_count int Number of text embedding models with weights on disk in ~/.fleet-manager/models/text-embedding/

Health checks related to text embedding:

  • text_embedding_backend_missing — WARNING: weights cached but fastembed not installed
  • text_embedding_ollama_bypass — WARNING: nomic-embed-text in Ollama but native server not running (embed requests will compete with LLM inference)
  • nomic_loaded_in_ollama — INFO: native server is running but nomic-embed-text still occupies an Ollama slot (cosmetic; self-resolves on KEEP_ALIVE expiry)

Without the native server, nomic-embed-text is served by Ollama's copy until a node in the herd has served it natively; after that it stays native (the two backends' vectors differ, and one index must not mix them).

  • embed_error_rate — WARNING/CRITICAL: embed failures in the last hour (regardless of backend)

Caching note: /dashboard/api/health responses are cached server-side for 30 seconds (configurable in dashboard.py::_HEALTH_CACHE_TTL_S). At the dashboard's 15-second poll cadence this gives a guaranteed cache hit on alternate polls — about a 50% reduction in trace-DB-aggregation work per active dashboard tab. Recommendations are derived from windows of minutes-to-hours, so 30-second staleness is invisible to operators. This endpoint is the only cached one; everything else is computed live.


POST /heartbeat

Receives heartbeats from node agents. Internal endpoint — not intended for external clients.

Regular heartbeat:

{
  "node_id": "mac-studio-ultra",
  "cpu": {"cores_physical": 24, "utilization_pct": 15.2},
  "memory": {"total_gb": 192.0, "used_gb": 45.3, "available_gb": 146.7, "pressure": "normal"},
  "ollama": {
    "models_loaded": [{"name": "llama3.3:70b", "size_gb": 40.0}],
    "models_available": ["llama3.3:70b", "qwen2.5:32b"],
    "requests_active": 1
  },
  "hardware": {
    "memory_total_gb": 192,
    "cores_physical": 24,
    "chip": "Apple M3 Ultra",
    "memory_bandwidth_gbps": 819.0,
    "arch": "apple_silicon"
  }
}

ollama.models_available_meta (optional) maps each model to its Ollama /api/tags metadata (modified_at, digest, format, family, families, parameter_size, quantization_level, parent_model), which the router's GET /api/tags returns. The agent sends it only when it changes, plus once every 60 s; null means "unchanged", and the router keeps its previous copy.

Drain signal:

{
  "node_id": "mac-studio-ultra",
  "draining": true
}

Response:

{"status": "ok", "node_status": "online"}

Image Generation

POST /api/generate-image

Generate an image on the best available node with mflux. Requires FLEET_IMAGE_GENERATION=true.

Request body (JSON):

{
  "model": "z-image-turbo",
  "prompt": "a cat sitting on a laptop",
  "width": 1024,
  "height": 1024,
  "steps": 4,
  "quantize": 8,
  "seed": 42,
  "negative_prompt": ""
}

Response: Raw PNG bytes with Content-Type: image/png.

Response headers:

Header Description
X-Request-Id The router's id for this request (in its log; no trace is written for a completed image)
X-Fleet-Node Node that generated the image
X-Fleet-Served-Model Image model used (with the rest of the canonical X-Fleet-* set)
X-Generation-Time Generation time in ms

Error responses: 400 (missing fields), 404 (model not available), 502 (generation failed), 503 (disabled).


Transcription (Speech-to-Text)

POST /api/transcribe

Transcribe an audio file on the best available node with Qwen3-ASR. Requires FLEET_TRANSCRIPTION=true.

Request: multipart/form-data with audio file field.

curl http://localhost:11435/api/transcribe -F "audio=@recording.wav"

Supported formats: WAV, MP3, M4A, FLAC, MP4, OGG (any format FFmpeg supports).

Response:

{
  "text": "Full transcription text...",
  "language": "English",
  "chunks": [
    {
      "text": "Hello, this is a test.",
      "start": 0.0,
      "end": 2.5,
      "chunk_index": 0,
      "language": "English"
    }
  ]
}

Response headers:

Header Description
X-Request-Id The router's id for this request (in its log; no trace is written for a completed transcription)
X-Fleet-Node Node that transcribed the audio
X-Fleet-Served-Model Transcription model used (with the rest of the canonical X-Fleet-* set)
X-Transcription-Time Processing time in ms

Error responses: 404 (no STT models available), 502 (transcription failed), 503 (disabled).


GET /fleet/queue

Lightweight queue status for client-side backoff decisions. Designed for high-frequency polling.

Note: returns Ollama and MLX queue entries only — these are the backends with real queue depth. Native text embedding (fastembed, port 11439) and vision embedding (port 11438) are instant backends with no queue depth; they appear in the dashboard's Node Models section but not here, since their pending=0 / in_flight=0 entries carry no backoff signal.

Response:

{
  "queue_depth": 5,
  "pending": 2,
  "in_flight": 3,
  "completed": 1250,
  "failed": 3,
  "estimated_wait_ms": 15000,
  "nodes_online": 2,
  "queues": {
    "Studio:gpt-oss:120b": {
      "pending": 2,
      "in_flight": 3,
      "concurrency": 2,
      "model": "gpt-oss:120b",
      "node_id": "Studio"
    },
    "mlx-local:mlx:mlx-community/Qwen3-Coder-Next-4bit": {
      "pending": 0, "in_flight": 1, "concurrency": 1,
      "max_queue_depth": 10,
      "cache_hit_rate": 0.997, "warm_hit_rate": 1.0, "cold_request_pct": 0.02,
      "avg_latency_ms": 5293.6, "avg_prompt_tokens": 201293, "avg_completion_tokens": 95,
      "stats_samples": 1244,
      "tool_repair": {"attempts": 3, "successes": 2, "failures": 1}
    }
  },
  "timestamp": 1712345678.123
}

Use estimated_wait_ms to decide whether to send a request now or back off. queue_depth = pending + in_flight.

MLX-specific fields (present on mlx-local: queue entries):

  • max_queue_depth — admission-control cap; requests over this get 503 (see FLEET_MLX_MAX_QUEUE_DEPTH)
  • cache_hit_rate, warm_hit_rate, cold_request_pct — prompt-cache observability (see docs/plans/mlx-prompt-cache-optimization.md)
  • avg_latency_ms, avg_prompt_tokens, avg_completion_tokens, stats_samples — running averages since process start
  • tool_repair: {attempts, successes, failures} — tool-call JSON repair counters. attempts fires whenever the model emits malformed JSON; successes means repair produced a dict that passed the tool's input_schema; failures means repair couldn't produce valid output and the original was passed through. Sustained rate above ~1% on any model is a signal the model is unreliable. See server/tool_call_repair.py.

Dashboard

HTML Pages

Endpoint Description
GET /dashboard Overview — the fleet's status and health score; live figures with sparklines; an activity chart (tokens/s by model, requests/min, failures); node cards with CPU, memory and tokens/s sparklines; models at work; Fleet Intelligence briefing. Uses the full window width
GET /dashboard/models Models — traffic trends (requests, latency, tokens per hour) and per-model performance, one time range for both
GET /dashboard/benchmarks Benchmarks — fleet throughput under controlled load, run history and charts; linked from Models and Settings (not a nav tab)
GET /dashboard/tags Usage — per-tag analytics with latency, tokens, errors, daily trends
GET /dashboard/health Health — findings by severity (suggestions folded), fleet vitals, model recommendations with one-click pull, briefing history
GET /dashboard/settings Settings — appearance (per browser), runtime toggles, context windows, nodes, configuration
GET /dashboard/color-states Colour reference — every colour token in both appearances, and the utilization bars in each warning state

Pages merged by the dashboard redesign keep their URLs: /dashboard/trends renders Models, /dashboard/recommendations renders Health, and /dashboard/apps renders Usage. Each scrolls to the section it used to be.

SSE Stream

GET /dashboard/events

Server-Sent Events stream for real-time fleet updates. Pushes a JSON snapshot every 2 seconds.

Event data:

{
  "nodes": [...],
  "queues": {...},
  "timestamp": 1710000000.0
}

Each node includes cpu, memory, ollama, and optionally capacity fields (when adaptive capacity learning is enabled).

JSON APIs

GET /dashboard/api/timeseries

Per-bucket request activity for the Overview's live charts, from the trace store.

Param Default Notes
window 3600 Seconds. Snapped to the nearest of 3600, 7200, 10800 (60 s buckets), 21600 (180 s) or 86400 (600 s), so every viewer's buckets align

Returns window_s, step_s, buckets (bucket start times; the last is the current, partial bucket), and arrays aligned to them: requests, failed (any status other than completed or retried, as the error-rate vital counts it), tokens (completion tokens), ttft_p50_ms (null for an empty bucket). Token series are also split as by_model, by_node and by_node_model (keyed node:model, like /fleet/queue), plus requests_by_node_model.

GET /dashboard/api/node-history

CPU %, memory %, queued and running per node, sampled by the router every 5 s and kept in memory for 3 hours (cleared on restart). window (seconds, max 10800). Returns t plus queued/running fleet totals aligned to it, and nodes[id] with cpu, mem, queued, running arrays; null where a node was not online at that sample.

GET /dashboard/api/trends

Hourly aggregated stats for the trends charts.

Parameter Type Default Description
hours integer 72 Number of hours of history to return

Response:

{
  "hours": 72,
  "data": [
    {
      "hour_bucket": "2026-03-08 01:00:00",
      "request_count": 15,
      "avg_latency_ms": 4200.5,
      "avg_prompt_tokens": 125.0,
      "avg_completion_tokens": 350.0
    }
  ]
}

GET /dashboard/api/models

Per-model daily stats and a per-model summary for the Models page. With start_ts and end_ts, the summary covers that range; without them it is all time.

Parameter Type Default Description
days integer 7 Number of days of history

Response:

{
  "days": 7,
  "daily": [
    {
      "model": "llama3.3:70b",
      "day": "2026-03-08",
      "request_count": 42,
      "avg_latency_ms": 4200.5,
      "total_prompt_tokens": 5250,
      "total_completion_tokens": 14700
    }
  ],
  "summary": [
    {
      "model": "llama3.3:70b",
      "total_requests": 312,
      "avg_latency_ms": 4100.2,
      "total_prompt_tokens": 39000,
      "total_completion_tokens": 109200
    }
  ]
}

GET /dashboard/api/overview

Summary totals for the dashboard header cards.

Response:

{
  "total_requests": 847,
  "total_prompt_tokens": 105875,
  "total_completion_tokens": 296450,
  "total_tokens": 402325,
  "models_count": 5
}

GET /dashboard/api/usage

Per-node, per-model, per-day usage stats from request traces.

Parameter Type Default Description
days integer 7 Number of days of history

Response:

{
  "days": 7,
  "data": [
    {
      "node_id": "mac-studio-ultra",
      "model": "llama3.3:70b",
      "day": "2026-03-08",
      "request_count": 25,
      "completed": 24,
      "failed": 1,
      "avg_latency_ms": 4200.5,
      "total_prompt_tokens": 3125,
      "total_completion_tokens": 8750
    }
  ]
}

GET /dashboard/api/traces

Request traces, newest first, for debugging and observability. Filters combine with AND.

Parameter Type Default Description
limit integer 50 Maximum number of traces to return
request_id string Only this request: the X-Request-Id response header. A retried request has one row per attempt
tag string Only requests carrying this tag (one whole entry of tags, not a substring)
since number Only traces at or after this unix timestamp, in seconds

A trace is written as the request finishes (for a stream, after its last chunk), so a lookup made the instant a response ends can come back empty; retry after a few milliseconds.

Response:

{
  "traces": [
    {
      "request_id": "abc12345-def6-7890",
      "model": "llama3.3:70b",
      "original_model": "llama3.3:70b",
      "node_id": "mac-studio-ultra",
      "score": 73.0,
      "scores_breakdown": {
        "model_thermal": 50,
        "memory_fit": 20,
        "queue_depth": 0,
        "wait_time": 0,
        "role_affinity": 15
      },
      "status": "completed",
      "latency_ms": 4200.5,
      "time_to_first_token_ms": 850.2,
      "prompt_tokens": 125,
      "completion_tokens": 350,
      "retry_count": 0,
      "fallback_used": 0,
      "excluded_nodes": [],
      "client_ip": "10.0.0.50",
      "original_format": "openai",
      "error_message": null,
      "timestamp": 1710000000.0,
      "tags": ["my-app", "production"],
      "timings": {
        "route_ms": 1.2,
        "queue_ms": 0.0,
        "upstream_ms": 4180.3,
        "backend_ms": 4172.0,
        "overhead_ms": 8.9
      },
      "finish_reason": "stop",
      "estimated_tokens": 118
    }
  ]
}

GET /dashboard/api/apps

Per-tag aggregated stats for the Apps page.

Parameter Type Default Description
days integer 7 Number of days of history

Response:

{
  "days": 7,
  "by_tag": [
    {
      "tag": "my-app",
      "request_count": 150,
      "avg_latency_ms": 3200.5,
      "avg_ttft_ms": 420.1,
      "total_prompt_tokens": 18750,
      "total_completion_tokens": 52500,
      "error_rate": 0.02
    }
  ]
}

GET /dashboard/api/apps/daily

Per-tag, per-day breakdown for trend charts.

Parameter Type Default Description
days integer 7 Number of days of history

Response:

{
  "days": 7,
  "daily": [
    {
      "tag": "my-app",
      "day": "2026-03-08",
      "request_count": 25,
      "avg_latency_ms": 3100.0,
      "total_tokens": 8500
    }
  ]
}

GET /dashboard/api/benchmarks

List stored benchmark runs.

Parameter Type Default Description
limit integer 50 Maximum number of runs to return

Response:

{
  "data": [
    {
      "run_id": "bench-1772966095",
      "timestamp": 1710000000.0,
      "duration_s": 300.0,
      "total_requests": 150,
      "total_failures": 0,
      "total_prompt_tokens": 18750,
      "total_completion_tokens": 52500,
      "requests_per_sec": 0.5,
      "tokens_per_sec": 175.0,
      "latency_p50_ms": 2100.0,
      "latency_p95_ms": 4500.0,
      "latency_p99_ms": 6200.0,
      "ttft_p50_ms": 450.0,
      "ttft_p95_ms": 1200.0,
      "ttft_p99_ms": 1800.0,
      "fleet_snapshot": "{...}",
      "per_model_results": "[...]",
      "per_node_results": "[...]",
      "peak_utilization": "[...]"
    }
  ]
}

POST /dashboard/api/benchmarks

Save benchmark results from the benchmark script.

Request body:

{
  "run_id": "bench-1772966095",
  "timestamp": 1710000000.0,
  "duration_s": 300.0,
  "total_requests": 150,
  "total_failures": 0,
  "total_prompt_tokens": 18750,
  "total_completion_tokens": 52500,
  "requests_per_sec": 0.5,
  "tokens_per_sec": 175.0,
  "latency_p50_ms": 2100.0,
  "latency_p95_ms": 4500.0,
  "latency_p99_ms": 6200.0,
  "ttft_p50_ms": 450.0,
  "ttft_p95_ms": 1200.0,
  "ttft_p99_ms": 1800.0,
  "fleet_snapshot": "{}",
  "per_model_results": "[]",
  "per_node_results": "[]",
  "peak_utilization": "[]"
}

Response:

{"status": "saved", "run_id": "bench-1772966095"}

POST /dashboard/api/benchmarks/start

Start a benchmark run from the dashboard. Supports default (loaded models only) and smart mode (fill memory with recommended models first).

Request body:

{
  "mode": "smart",
  "duration": 300,
  "model_types": ["llm", "vision", "embed", "image"]
}
Field Type Default Description
mode string "default" "default" (loaded models) or "smart" (fill memory first)
duration float 300 Benchmark duration in seconds
model_types array ["llm"] Types to benchmark: "llm", "vision", "embed", "image"

Response: {"status": "started", "run_id": "bench-...", "mode": "smart", "duration": 300, "model_types": ["llm", "vision", "embed", "image"]}

Error: 409 if a benchmark is already running.


GET /dashboard/api/benchmarks/progress

Get current benchmark status and progress. Poll every 2 seconds during a run.

Response:

{
  "status": "running",
  "phase": "Running 300s benchmark (embed, image, llm)...",
  "elapsed": 45.2,
  "duration": 300,
  "requests_completed": 1250,
  "requests_failed": 0,
  "tok_per_sec": 85.3,
  "models": ["gpt-oss:120b", "nomic-embed-text:latest", "z-image-turbo"],
  "models_pulled": ["codestral:22b", "llama3.2:1b"],
  "pull_progress": {},
  "error": null,
  "run_id": "bench-..."
}

Status values: idle, pulling, warming_up, running, complete, error, cancelled.

During pull phase, pull_progress includes: model, node_id, current, total, ram_gb, on_disk, pct, completed_gb, total_gb.


POST /dashboard/api/benchmarks/cancel

Cancel a running benchmark.

Response: {"status": "cancelled"} or {"status": "not_running"}.


GET /dashboard/api/context-usage

Per-model context usage analysis — actual vs allocated context sizes.

Query params: days (default: 7) — lookback window for trace analysis.

Response:

{
  "days": 7,
  "models": [
    {
      "model": "gpt-oss:120b",
      "allocated_ctx": 131072,
      "override_ctx": 16384,
      "request_count": 67234,
      "prompt_tokens": {"avg": 1288, "p50": 833, "p75": 1322, "p95": 3706, "p99": 4797, "max": 10938},
      "total_tokens": {"p95": 4120, "p99": 5409, "max": 34721, "max_24h": 8675},
      "utilization_pct": 4.1,
      "recommended_ctx": 16384,
      "savings_pct": 87.5
    }
  ]
}

GET /dashboard/api/recommendations

Model mix recommendations for the fleet based on hardware, usage patterns, and curated benchmark data. Results are cached for 5 minutes.

Query parameters:

Parameter Type Default Description
refresh int 0 Pass 1 to force re-analysis (bypasses cache)

Response:

{
  "nodes": [
    {
      "node_id": "mac-studio",
      "total_ram_gb": 512.0,
      "usable_ram_gb": 506.0,
      "current_models": ["gpt-oss:120b", "qwen3:235b-a22b"],
      "recommendations": [
        {
          "model": "gpt-oss:120b",
          "display_name": "GPT-OSS 120B",
          "category": "reasoning",
          "ram_gb": 72.0,
          "quality_score": 89.0,
          "reason": "Currently your most-used reasoning model (4303 requests in 24h)",
          "priority": "high",
          "already_available": true
        }
      ],
      "total_recommended_ram_gb": 313.0,
      "ram_headroom_gb": 193.0
    }
  ],
  "usage": {
    "total_requests_24h": 4484,
    "category_breakdown": {"reasoning": 4390, "general": 52, "coding": 42},
    "top_models": [{"model": "gpt-oss:120b", "requests": 4390, "category": "reasoning"}],
    "category_coverage": {"general": true, "coding": true, "reasoning": true, "creative": true, "fast-chat": true}
  },
  "fleet_summary": "1 node(s), 512GB total RAM, 6 model(s) recommended using 313GB",
  "uncovered_categories": [],
  "generated_at": 1741700000.0
}

The recommender uses a curated catalog of 30+ models with benchmark data (MMLU, HumanEval, MT-Bench) and considers: available RAM per node (with 6GB OS overhead), last 24h request patterns, models already downloaded, cross-fleet distribution (avoids redundant large models), and a 50% RAM cap per model to ensure variety.


POST /dashboard/api/pull

Pull a model onto a specific node via Ollama's /api/pull API. Blocks until the pull completes (may take minutes for large models).

Request body:

{
  "node_id": "mac-studio",
  "model": "codestral:22b"
}

Response:

{"ok": true, "node_id": "mac-studio", "model": "codestral:22b"}

Returns {"ok": false, ...} if the pull fails. The router proxies the pull request to the target node's Ollama instance using the same HTTP client used for inference (600s read timeout).

POST /dashboard/api/delete

Delete a model from a specific node via Ollama's DELETE /api/delete API.

Request body:

{
  "node_id": "mac-studio",
  "model": "qwen3-coder-480b:latest"
}

Response:

{"ok": true, "node_id": "mac-studio", "model": "qwen3-coder-480b:latest"}

Returns {"ok": false, ...} if the delete fails. This permanently removes the model from the node's disk — it must be re-downloaded to use again.

GET /dashboard/api/pinned-models

Return the pin state that feeds the Ollama preloader. Env-level pins (from FLEET_PINNED_MODELS) are fleet-wide and read-only here; per-node pins live in <data_dir>/pinned_models.json and are managed via POST /dashboard/api/pinned-models.

Response:

{
  "env_pins": ["gpt-oss:120b"],
  "per_node": {
    "Neons-Mac-Studio": ["gemma3:27b", "gpt-oss:120b"]
  },
  "max_count": 3
}

max_count is FLEET_MODEL_PRELOAD_MAX_COUNT; if the sum of pins on any node exceeds it, the preloader truncates and logs a warning — raise the env var only if your backend can actually hold more concurrent models (macOS Ollama 0.20.4 caps at 3 regardless).

POST /dashboard/api/pinned-models

Toggle a per-node pin. Writes atomically to <data_dir>/pinned_models.json. The model preloader re-reads the file every refresh cycle (10 min), so dashboard toggles take effect within that window.

Request body:

{"node_id": "Neons-Mac-Studio", "model": "gemma3:27b", "pinned": true}

Response:

{"ok": true, "per_node": {"Neons-Mac-Studio": ["gemma3:27b", "gpt-oss:120b"]}}

Validation:

  • node_id and model required — 400 if either is empty.
  • vision-embedding category models (DINOv2, SigLIP, CLIP) are rejected when pinned=true — those run on the embedding service, not Ollama, so a preloader pin would be meaningless. Unpin (pinned=false) always succeeds so stale entries can be cleaned up.

GET /dashboard/api/model-management

Per-node model details with disk sizes, usage statistics, and last-used timestamps for the model management UI.

Response:

{
  "nodes": [
    {
      "node_id": "mac-studio",
      "models": [
        {
          "name": "gpt-oss:120b",
          "display_name": "GPT-OSS 120B",
          "category": "reasoning",
          "size_gb": 60.9,
          "parameter_size": "120B",
          "quantization": "Q4_K_M",
          "last_used": 1741700000.0,
          "days_unused": 0.1,
          "total_requests": 18065,
          "loaded_in_vram": true,
          "unused": false
        }
      ],
      "total_size_gb": 959.0,
      "disk_available_gb": 2994.0,
      "disk_total_gb": 3722.0
    }
  ]
}

Models are sorted by: loaded in VRAM first, then by last-used descending, then alphabetically. The unused flag is true if the model has never been used through the router or hasn't been used in 7+ days.

GET /dashboard/api/settings

Current router configuration, toggleable settings, and registered node list with versions.

Response:

{
  "router_version": "0.3.0",
  "router_hostname": "Neons-Mac-Studio",
  "config": {
    "toggles": { "auto_pull": true, "image_generation": false, "transcription": false, "dynamic_num_ctx": false },
    "server": { "host": "Neons-Mac-Studio", "port": 11435, "data_dir": "~/.fleet-manager", "max_retries": 2 },
    "heartbeat": { "heartbeat_interval": 5.0, "heartbeat_timeout": 15.0, "heartbeat_offline": 30.0 },
    "scoring": { "score_model_hot": 50.0, "score_model_warm": 30.0, "..." : "..." },
    "rebalancer": { "rebalance_interval": 5.0, "rebalance_threshold": 4, "rebalance_max_per_cycle": 3 },
    "pre_warm": { "pre_warm_threshold": 3, "pre_warm_min_availability": 0.6 },
    "auto_pull_config": { "auto_pull_timeout": 300.0 },
    "context_protection": { "context_protection": "strip" }
  },
  "nodes": [
    {
      "node_id": "Neons-Mac-Studio",
      "status": "online",
      "agent_version": "0.3.0",
      "ip": "http://localhost:11434",
      "models_loaded_count": 2,
      "is_router": true
    }
  ]
}

POST /dashboard/api/settings

Toggle runtime-mutable boolean settings: auto_pull, image_generation, transcription, dynamic_num_ctx, num_ctx_auto_calculate, fleet_intelligence; all other fields are ignored ("status": "no_change"). Changes take effect immediately and are written to ~/.fleet-manager/env, so they survive a restart.

Request:

{"auto_pull": false}

Response:

{"status": "updated", "updated": {"auto_pull": false}}

GET /dashboard/settings

HTML page for the Settings dashboard tab. Shows router info, toggle switches, read-only configuration tables grouped by category, and a node list with version tracking.


Fleet Management Endpoints

Read + control endpoints for clients that need to reason about capacity or pin models (e.g. a benchmark that must hold hardware constant). See docs/plans/client-ergonomics-from-agent-feedback.md.

GET /fleet/status

Full fleet state — every node's hardware/cpu/memory/ollama/mlx status plus derived models_loaded_count and free_slots per node, and all queue info.

GET /fleet/queue

Lightweight queue depths + estimated wait, for high-frequency client backoff.

GET /fleet/limits

Effective serving constraints so a client can auto-serialize instead of self-DoSing: hot_model_cap, per-node free_slots, router max_retries, and the MLX in-flight caps. Firing more concurrent requests than the box can absorb just produces 429s (or Ollama queue-full 503s) — read this first.

POST /fleet/pin

Pre-warm a model resident and persist the pin (reloaded if evicted). Body: {"model": "<name>", "node_id": "<optional>", "wait": <bool>, "timeout_s": <float>}. One call replaces the manual ollama … keep_alive dance. Returns 404 if the model isn't on disk on any online node.

wait (default false) — when true, the call blocks until the router's routing view confirms the model resident before returning, so a caller that pins-then-scans doesn't race the heartbeat and get fallback-substituted. The pre-warm itself already blocks through the Ollama load; wait additionally waits out the heartbeat-reflection lag (~5s, the window where the model is loaded but routing hasn't seen it yet). timeout_s (default 30) caps that wait. Response adds "ready": true|false and "ready_after_ms": <int>. On timeout, ready: false (the model may still become resident shortly — the client learns the truth instead of racing).

// POST /fleet/pin  {"model": "qwen3-coder:30b", "wait": true}
{
  "ok": true,
  "model": "qwen3-coder:30b",
  "pinned_node": "mac-studio",
  "ready": true,
  "ready_after_ms": 2140,
  "per_node": { … }
}

mlx: models are always-resident subprocesses (configured via FLEET_NODE_MLX_SERVERS), so pinning is a no-op: the call skips warming and reports ready from the MLX server's health, with an explanatory note.

⚠️ Unpin when you're done — pins are forever

A pin persists until you delete it. It is not scoped to your session, and nothing expires it. The preloader reloads pinned models unconditionally — no recency check, no budget — because you asked for them.

Pinned models must all stay resident simultaneously. Pin more than the box can hold and they evict each other, and the preloader reloads them in a loop, forever. A benchmark that pinned each model it tested and never unpinned reached 307 GB of pins (including a 290 GB model) on a 512 GB box and produced hours of ~300 GB disk→memory churn (2026-07-17).

If you're benchmarking, the pattern is:

POST /fleet/pin   {"model": "X", "wait": true}   # pin + wait until resident
…run your requests against X…
DELETE /fleet/pin/X                              # ← DO NOT SKIP THIS

Since 0.8.2 the herd refuses a pin that can't physically co-reside with your existing pins (409, with the arithmetic and what's already pinned). That's a backstop, not a substitute for unpinning — check GET /fleet/limits and GET /fleet/status (pinned per node) to see what you're holding.

Pin the model you're benchmarking (use wait: true to drop your own readiness loop), then send requests serially against it (X-Fleet-Fallback: false confirms it actually ran) — then unpin it.

Errors: 400 model missing / unknown node_id (the response lists known_nodes) · 404 model not on disk anywhere · 409 pin would over-commit the node (pass "force": true to override) · 503 on disk but wouldn't load (usually insufficient free memory).

DELETE /fleet/pin/{model}

Release a pin so the model can be evicted normally. The {model} path segment accepts / and : (e.g. mlx-community/Foo or qwen3-coder:30b).


Static Assets

Endpoint Description
GET /favicon.svg SVG favicon (horse icon in brand purple)
GET /favicon.ico Redirects to SVG favicon
GET / Redirects to /dashboard