NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
Go modules
Go modules, #2984 by repository stars
Latest stable
v0.1.0
Last release 21 days ago
01 Nov 2026
Ships fairly regularly
a new release about every 5 weeks
Nearly every release is documented
notes for 25 of 25 stable releases
Nothing withdrawn
no release was ever pulled
1 years old
39 releases · first in 2025
Drag across the ruler to compare, or tap a release to open it.
Olla is a high-performance proxy and load balancer for LLM infrastructure.
Olla is a high-performance proxy and load balancer for LLM infrastructure.
# Docker
docker pull ghcr.io/thushan/olla:v0.1.0
# Binary (see assets below)
./olla --config config.yamlIt's a big release and acknowledges the work we did to get here too with everyone's contributions to get us to this milestone! We'd have called this v0.0.30 but really, it's the first of the month and Olla has passed a lot of the stability checkpoints now.
There are lots of things to be aware of, documented in our Olla v0.1.0 Release Article or see the Full Release Writeup.
5xx as a success. total_requests, per-endpoint requests, success_rate, total_failures and the matching olla_* metrics now count each request once per backend attempt, so the status verdict, the low success rate issue and any alert on these may change (#267)./olla/anthropic/v1/messages, /messages/count_tokens and /models take the same limits as the proxy routes (100 requests per minute per IP, burst 50, 1000 globally) and share each client's per-IP bucket. Many sessions behind one IP (NAT, CI, parallel agents) may see 429s, so raise server.rate_limits.per_ip_requests_per_minute and burst_size (#257).per_ip_requests_per_minute: 0 no longer turns rate limiting off. The global limit (default 1000/min, burst 50) still applies. Set global_requests_per_minute: 0 as well to disable both (#256).tools are still translated. For older builds, override the profile with zz-sglang.yaml or zz-lmdeploy.yaml and set api.anthropic_support.enabled: false (#242)./v1/responses, /v1/messages, /v1/rerank and Ollama's /api/ps and /api/version. A custom profile serving these paths needs the same entries (#240)./engines/v1/chat/completions and Lemonade /api/v1/chat/completions. If no backend serves a chat path, the request gets a 404 (#241).vlm with the vision capability and tools maps to function_calling. ?type=llm no longer lists vision models; ?capability=chat still does (#250)./olla/ollama/api/ps merges every Ollama host. Entries carry olla_endpoint, a partial answer sets X-Olla-Partial, and the route is rate limited. The /olla/ollama/api/list stub is gone and /api/blobs returns 501 (#243).http_status, and a backend 5xx logs at WARN instead of INFO. Check any log-based alerting (#253).retry_attempts and retry_backoff are now honoured. A negative retry_backoff fails startup (#245).models.yaml with its own name_patterns needs the two new entries (#248).X-RateLimit-* headers are omitted when no rate limit applies (#262).olla.type in /olla/models and a top-level type on /olla/models/{id} (#249)./info, and /olla/tei/ provider prefixes. Thanks to @MartinEmrich for the contribution (#231).503 when the only capable backend is down (#236)./api/embed and embedding models now route correctly in a fleet that includes TEI (#247).proxy.retry controls failover (#260).HTTP_PROXY or across hosts (#223, #234).config.yaml are reported at startup (#258), listed once each (#263).max_concurrent_requests in a profile (#255).Content-Type header (#238), and Content-Encoding: identity bodies are read (#238)./olla/models/{id} works for IDs containing slashes (#244).:latest (#235), and Ollama vision and tool capabilities are read from the backend (#250).make smoke-build-local checks build-local from clean and dirty build dirs (#239, #252)./api/embed, /api/ps and capabilities in /api/tags (#251).Full detail, with the links for each change, is in the v0.1.0 release notes.
Full Changelog: v0.0.29...v0.1.0
Your coding agent can read these notes before it upgrades. Set up the MCP server →
This was going to be v0.0.30. It's v0.1.0 because it's a big release: TEI embeddings and rerank support, the reworked herd dashboard, rate-limit and concurrency hardening, retry controls and corrected request stats. Some of these change what existing setups see, so it gets a minor version bump. Check the changes below before you upgrade. Fixes from here ship as v0.1.x.
These v0.1.0 changes alter what existing setups see. Each links to its entry below.
/olla/anthropic/v1/messages now forwards to them natively
(SGLang v0.5.9+, LMDeploy v0.13.0+) instead of translating. LMDeploy requests carrying tools are still translated.
On older builds, copy the profile to config/profiles/zz-sglang.yaml or zz-lmdeploy.yaml (the zz- prefix is needed
because profiles load in lexical order and the last one with a given name wins) and set api.anthropic_support.enabled: false.
See Profile Loading and Changed: SGLang and LMDeploy./v1/responses, /v1/messages, /v1/rerank and
Ollama's /api/ps and /api/version, and a request on a declared path only goes to backends whose profile declares it.
A custom profile that serves these paths needs the same entries. See profile paths./engines/v1/chat/completions and
Lemonade /api/v1/chat/completions. If no backend in the fleet serves a chat path, the request gets a 404.
See chat path.vlm with the vision capability,
and tools maps to function_calling. ?type=llm no longer lists vision models; ?capability=chat still does.
Model-less requests with images or tools go to hosts with a model that declares them. See Ollama vision./olla/ollama/api/ps merges every Ollama host. Entries carry olla_endpoint, a partial answer sets X-Olla-Partial, and the route is rate limited.
The /olla/ollama/api/list stub is gone, so those requests now reach the backend, and /api/blobs returns 501.
See /api/ps.http_status, and a backend 5xx logs at WARN instead of INFO. Check any log-based alerting or
level filters. See the log change.5xx as a success. total_requests, per-endpoint requests, traffic,
success_rate, total_failures and the matching olla_* metrics now count each request once per backend attempt and a 5xx as failed, so the
status verdict, the low success rate issue and any alert on these may change. See stats counting.retry_attempts and retry_backoff are now honoured. A fetch that failed once may now be retried, and a negative
retry_backoff fails startup. See discovery retries.text-generation and chat.
A custom models.yaml with its own name_patterns needs the two new entries. See name patterns.per_ip_requests_per_minute: 0 no longer turns rate limiting off. If you set per_ip_requests_per_minute: 0 (or OLLA_SERVER_PER_IP_RATE_LIMIT=0) to turn rate limiting off,
the global limit (default 1000/min, burst 50) now applies; set global_requests_per_minute: 0 as well to disable both.
See rate limit tiers./olla/anthropic/v1/messages, /messages/count_tokens and
/models take the same limits as the proxy routes (default 100 requests per minute per IP, burst 50, 1000 globally),
and share each client's per-IP bucket with them. Claude Code stays well inside these, including a few parallel sessions,
but many sessions behind one IP (a team behind NAT, CI runners, parallel agents) may now see 429s: raise
server.rate_limits.per_ip_requests_per_minute and burst_size. See Anthropic rate limits.X-RateLimit-* headers are omitted when no rate limit applies. With both limits at 0 (and for an unlimited health probe) responses used to carry X-RateLimit-Limit: 0 and X-RateLimit-Remaining: 0; they now carry none. See rate limit headers.olla.type in /olla/models and a top-level type on /olla/models/{id}. See model kind.proxy.retry controls failoverFailover on a connection failure was hardcoded: every eligible endpoint once, and any method replayed. A POST whose backend read the
body and then dropped the connection was therefore sent again to the next endpoint, so a prompt could run twice. The new proxy.retry
block (enabled, max_attempts, non_idempotent) makes this configurable, with OLLA_PROXY_RETRY_* environment overrides. The
defaults are today's behaviour, so nothing changes unless you set it: non_idempotent: false stops POST, PATCH and DELETE failing over after
a connection failure (GET still does), max_attempts: N caps attempts, enabled: false makes every request a single attempt. A failed
connection still marks the endpoint unhealthy in every case, a negative max_attempts fails startup, and both engines and the Anthropic routes follow the policy.
Timeouts and HTTP error statuses do not fail over. The deprecated proxy.max_retries and
proxy.retry_backoff stay accepted, do nothing, and log one warning at startup when set to a non-zero value. They are not mapped onto the new block.
A request that fails after part of its response was sent is no longer retried (see the Upgrading note above). See Retry Behaviour. Reported by @matthewjhunter.
/olla/anthropic/v1/messages, /messages/count_tokens and /models were registered as plain routes, so the rate limit
never reached them, although the API reference said the standard limits applied. They are the main entry point for
Claude Code. Translator routes now register through RegisterTranslatorRoute and sit behind the same rate limit as the proxy
routes, and a translator added later is covered without extra wiring. A limited request gets 429 with Retry-After and
Anthropic's rate_limit_error body, and an oversized one gets 413 with request_too_large, both carrying a request-id,
so Claude Code retries and reports them properly. The size limit already applied to these routes; it now answers in the same
Anthropic shape. Measured against a real Ollama, one Claude Code session peaked at 2 requests a minute and three parallel
sessions at 9 a minute, with a burst of 4 when four sessions started together, so the existing defaults (100 per minute per IP, burst 50) need no change; many sessions
behind one IP should raise per_ip_requests_per_minute. See Rate Limit Best Practices.
Reported by @matthewjhunter.
A chunked request (no Content-Length) to /olla/anthropic/v1/* was read into memory up to server.request_limits.max_body_size
(100MB by default) before the translator's own max_message_size (10MB) rejected it, so a few concurrent oversized chunked
requests could hold gigabytes. The read now stops at the smaller of the two limits and answers 413 with request_too_large.
For /messages that is the same status as before; the rejection now happens before the handler reads the body, so it is counted
as a size-limit violation rather than in the translator metrics, and the message text is "Request body too large". A chunked body
between the two limits on /messages/count_tokens (previously a 400) or /models (previously served) now also gets the 413.
Requests with a Content-Length, and all proxy routes, behave as before.
X-RateLimit-* headers are omitted when rate limiting is offEvery response through a rate-limited route carried X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, even with both limits at 0, so a 413 or a plain 200 showed X-RateLimit-Limit: 0, X-RateLimit-Remaining: 0, which a client reads as fully throttled. The headers are now sent only while a limit applies.
The Overview herd table now shows In flight, Requests, Type and Models next to the status, response rate and latency columns. A Columns menu adds Priority, Probe latency (the last health-check probe, not proxied requests), Traffic and Failures (estimated from requests and response rate); the choice is saved in the browser. Hover or focus a status pill to see why an endpoint is amber or red. A search box and an All/Unhealthy toggle filter the herd table and the Endpoints panel. Under 600px wide only Endpoint, Status, Resp. rate and Latency show. Frontend only; no API or config change. See Admin Dashboard.
The Overview also gets Sticky sessions (active, hit rate, evictions, cap), Translator (passthrough share, streaming, fallbacks, latency) and Process (heap, memory pressure, goroutines, GC pause) cards, polled every 15s. Sticky hides while sticky sessions are off; Translator appears once the first request has gone through a translator.
config.yaml are reportedA key in the main config that Olla does not read (a typo, or one that was documented but never existed) used to be dropped silently. Startup now logs one WARN line listing every such key by its full path and line,
for example proxy.engnie (line 12), and olla --validate-config lists them under "Ignored keys" without changing the
result or exit code. Loading still never fails on them, and models.yaml and profiles are not checked. Free-form maps such
as model_aliases and endpoint headers are never reported. Each key is listed once, and both the WARN and the report show at most 100, ending with and N more; the count covers every key found. Reported by @matthewjhunter.
In translation mode Olla sent every backend /v1/chat/completions. Docker Model Runner serves the OpenAI API under
/engines/v1/ (the root /v1/ alias only exists in recent builds) and Lemonade under /api/v1/, so on those hosts
translated requests 404ed, and because a 404 is not retried the client saw the error even when another healthy
backend could have answered. Olla now resolves the chat path from each endpoint's profile and applies it per attempt,
so a fleet mixing DMR, Lemonade, vLLM and Ollama works, and failover between them uses the right path each time.
Backends that speak Anthropic natively are still passed through to their anthropic_support.messages_path. Endpoints
whose profile has no chat path (Text Embeddings Inference) are never sent translated requests; if nothing selected can
serve chat the request is refused with a 404 that is logged as a sampled "Request rejected" line.
/olla/models/{id} for IDs containing slashesGET /olla/models/BAAI/bge-small-en-v1.5 returned 404 because only the first path
segment was read as the ID. The whole remainder of the path is now the ID, so slash IDs
resolve both raw and percent-encoded (BAAI%2Fbge-small-en-v1.5), and aliases still
resolve to them. An empty ID or one ending in a slash returns 400.
discovery.model_discovery.retry_attempts and retry_backoff were accepted but never used, so one dropped
connection failed discovery for that endpoint until the next interval. Discovery now retries connection errors,
timeouts and 5xx responses, doubling retry_backoff (capped at 30s) between attempts. All attempts share the
existing timeout, so a cycle never runs longer than it did before. Parse errors, 4xx responses and listings over
the 10MB cap are not retried, and a cycle that still fails on a retryable error counts as one failure towards the five-failure cooldown.
A negative retry_backoff is now rejected at startup.
HTTP_PROXY or across hostsModel discovery still honoured HTTP_PROXY/HTTPS_PROXY, so with a proxy set in the environment, a plain-HTTP
endpoint's auth: token or API key went to that proxy on every discovery cycle. Discovery and the health checker also
followed redirects to other hosts, and Go only strips Authorization on those, so an api_key header or custom
headers: entry could be replayed at the redirect target. Both now connect directly and stop at a redirect that
leaves the configured host and port (a same-host http to https upgrade is still followed), so a cross-host redirect
counts as a failed discovery or health check. Loopback endpoints were never proxied, so only setups that relied on HTTP_PROXY
to let discovery reach a non-loopback backend are affected: that backend now needs to be reachable directly from the Olla host,
as chat traffic already required.
SGLang (v0.5.9+) and LMDeploy (v0.13.0+) serve the Anthropic Messages API themselves, so their shipped
profiles now carry anthropic_support and /olla/anthropic/v1/messages forwards to them as-is
(X-Olla-Mode: passthrough) rather than translating through /v1/chat/completions. Older builds answer
404 for /v1/messages, and min_version is informational: Olla does not enforce it. If any SGLang or
LMDeploy in your fleet predates those versions, copy the profile to config/profiles/zz-sglang.yaml or
zz-lmdeploy.yaml and set api.anthropic_support.enabled: false in the copy (steps on the
SGLang and
LMDeploy pages), or
set translators.anthropic.passthrough_enabled: false to translate for every backend. LMDeploy v0.13.0 to
v0.17.x reject Anthropic tools, so its profile declares no_tool_use and requests carrying tools are still
translated, including on v0.18.0 and later.
/olla/models now reports each model's kind (llm, vlm, embeddings, reranker or classifier) as
olla.type, and /olla/models/{id} as a top-level type. ?type= matches that kind exactly, so an
embedder whose name says instruct is listed under embeddings and no longer also under llm, and a
model that reports vlm or carries the vision capability is vlm only. The legacy ?capability=chat still returns every model that can chat, vision
models included. vLLM, SGLang, Lemonade, LMDeploy, llama.cpp, vLLM-MLX and Docker Model Runner models no
longer carry the provider name as their type; /internal/status/models now omits it unless the backend
reported a kind. TEI rerankers and classifiers were already typed in the LM Studio-format listing; what is
new is that rerankers and classifiers recognised by name on backends that report no kind are typed there
too instead of falling back to llm. That type can therefore be a value outside LM Studio's own
vocabulary (reranker, classifier).
vlm and tool support is recordedNewer Ollama releases report capabilities in /api/tags, but only embedding was read, so models such as
qwen3-vl:8b listed as llm and ?type=vlm and ?capability=vision returned nothing. A model that lists
vision is now vlm with the vision capability (it keeps text-generation), and tools becomes the
function_calling capability. Model-less requests that carry images or tools now prefer hosts with a
capable model, so hosts whose models carry no such tag are left out of those requests while an Ollama host
has a match; name the model to reach them. If the capable hosts are all down the request gets a 503
capability_unavailable. Older Ollama versions report no capabilities and are unchanged.
Olla now routes to Hugging Face Text Embeddings Inference
(TEI) servers. Set type: "text-embeddings-inference" on an endpoint; the provider routes are
/olla/tei/ and /olla/text-embeddings-inference/, and requests are forwarded to TEI's own paths
(/embed, /embed_all, /embed_sparse, /v1/embeddings, /rerank, /predict, /similarity,
/tokenize, /decode, /info, /health, /metrics). TEI has no /v1/models, so the shipped
text-embeddings-inference.yaml profile discovers the single model each instance serves from GET /info
(served_model_name, falling back to model_id) and reads its kind from model_type, giving
embedding, reranker and classifier models their own types and capabilities. Mixed fleets work: a TEI
host is never sent chat requests, and model-less native requests are steered to a TEI instance serving
the matching kind. See the TEI integration guide.
TEI reranker and classifier models get their own types (reranker, classifier) with
rerank and classification capabilities (rerankers carry both, as TEI's /predict serves them too), and are never tagged text-generation. They show
up under /olla/models?type=reranker and ?type=classifier. Model-less /embed, /embed_all,
/embed_sparse, /similarity, /rerank and /predict requests, whose bodies carry no model
field, are routed to an instance serving a model of the matching kind. A custom models.yaml
doesn't need reranker or classifier entries under type_capabilities: any type_capabilities
or context_thresholds key your file omits falls back to the built-in default.
vLLM and other OpenAI-style servers don't say what a model is, so a reranker or embedder looked like
a chat model. Two new capabilities.name_patterns entries tag rerank for names containing rerank
(bge-reranker-v2-m3, Qwen3-Reranker) and embeddings, similarity and vector-search for the common
embedders (bge-m3, e5-large, nomic-embed, snowflake-arctic-embed2, all-MiniLM, multi-qa-*,
msmarco-*, stella-*). Those models drop text-generation, and chat too, so an embedder named
gte-Qwen2-7B-instruct is no longer a chat model. Model-less rerank and embeddings requests are steered to
them. A kind the backend does report, and capabilities it declares, are never overridden. A custom
models.yaml with its own name_patterns list needs the two entries added to benefit.
When a request names a model, endpoint selection now depends only on which endpoints host that model.
Capability tags are heuristic, so a request for llama3 could previously be narrowed away from its own
hosts because some other model carried (or lacked) a tag. Capability filtering now applies only to
requests that name no model (for example a model-less vision or /rerank request). When it does apply,
the capabilities must all be on a single model: an endpoint hosting one vision model and a different
tools model no longer satisfies a request needing both. A request needing function_calling no longer also
requires the tools tag, which many backends never set. If you depended on a named request being
steered by capability tags, name the model you want instead.
A request that names no model but needs a capability (for example /embed, /rerank, /v1/embeddings or a request with images) is steered to hosts with a model of that kind. When none of those hosts was healthy, Olla fell back to every compatible endpoint, so with the only embedder stopped, /embed went to the reranker and classifier instances and TEI answered 424. Olla now answers 503 with X-Olla-Routing-Reason: capability_unavailable and X-Olla-Routing-Strategy: capability when models with the capability exist but none is on a healthy endpoint. Nothing to change on your side: if no model with the capability has ever been discovered, the unfiltered fallback still applies, and requests that name a model are unaffected.
/olla/proxy/v1/responses and /olla/proxy/v1/messages were narrowed to the profiles that
declared them, evicting backends that serve them without saying so. The shipped profiles
now declare /v1/responses (vLLM, SGLang, LMDeploy, llama.cpp, LM Studio, vLLM-MLX, LiteLLM,
Ollama, OpenAI-compatible, Lemonade, Docker Model Runner),
/v1/messages (vLLM, SGLang, LMDeploy, llama.cpp, LM Studio, vLLM-MLX, LiteLLM, Lemonade, Ollama),
/v1/messages/count_tokens (vLLM, SGLang, LMDeploy, llama.cpp, vLLM-MLX, LiteLLM) and /v1/rerank
(SGLang, llama.cpp, vLLM-MLX). These paths only exist in newer backend releases (for example Ollama 0.14.0 for /v1/messages, vLLM 0.11.1, LM Studio 0.4.1), and older ones answer 404; /v1/rerank on llama.cpp also needs --reranking. Ollama also declares /api/ps and /api/version, so a proxy call to
them goes to Ollama hosts only. Lemonade serves /v1/responses at /api/v1/responses and
/v1/responses, and Docker Model Runner registers it under /v1 and /engines/v1; both are declared
at the unprefixed path so a standard /olla/proxy/v1/responses call keeps them as candidates.
Docker Model Runner also declares /anthropic/v1/messages and /anthropic/v1/messages/count_tokens,
which a direct proxy call used to miss in a mixed fleet.
The generic OpenAI-compatible profile stands in for servers Olla has no profile for, such as a remote
vLLM or SGLang typed openai, so it declares /v1/responses and /v1/messages: it stays a candidate
for each of them in a fleet where other profiles declare the path. It does not declare
/v1/messages/count_tokens or /v1/rerank, because the paths of this profile also admit Ollama and
LM Studio, which answer 404 on both; a remote server that serves them needs its own profile. Declaring a path is not Anthropic passthrough
(anthropic_support stays disabled), and a request that names a model is still routed by that
model. A custom profile in config/profiles/ that serves these paths needs the same entries.
TEI declares single-segment paths such as /embed, /rerank, /health, /metrics and /tokenize.
Request path inspection matched paths by suffix, so those would also have claimed deeper paths on
other backends (for example /api/embed or a .../v1/rerank). One-segment profile paths now match
only a request path of that single segment.
Olla wrote a WARN for every request it rejected, so a client hammering a rate limit produced millions of identical lines in minutes and pushed useful history out of the rotated log. Rate limit (429), request size (413/431), dashboard access (403) and proxy routing rejections are now sampled: the first in each 10 second window logs in full, and the rest become one summary such as WARN Rate limit exceeded, repeats suppressed layer=rate_limit client_ip=203.0.113.7 suppressed=19994 window=10s. The Model routing rejected request and Large request blocked WARNs are gone, since the sampled Request rejected line carries the same detail. Responses, and the rate limit and size limit counters, are unchanged, and a backend's own 4xx and 5xx responses are still logged per request.
per_ip_requests_per_minute: 0 returned early and skipped the global limiter as well, while global_requests_per_minute: 0 only disabled itself. The two limits are now independent switches, and 0 disables only its own tier. Two further faults went with it. A request already over its per-IP limit still took a global token, so one noisy client could starve everyone else, even on the default config; the per-IP limit is now checked first and a request rejected by either limit is charged to neither. A global rejection also reported the per-IP value in X-RateLimit-Limit and a fixed Retry-After: 60; it now reports the global limit in X-RateLimit-Limit, logs the reason Global rate limit exceeded and sets Retry-After from the global bucket's refill. Rejections of either kind also report X-RateLimit-Remaining: 0. Reported by @matthewjhunter.
Content-Encoding: identity bodies and unreadable /v1/embeddings bodiesA client that sent Content-Encoding: identity had its body skipped, so the model name was never read and the
request was routed as if it named none. identity means no encoding and is now read like any plain body; gzip,
br and lists such as gzip, identity are still skipped. Separately, /v1/embeddings is now an embeddings path
for capability routing, so a model-less request whose body Olla can't read is still kept off chat-only hosts, and
is refused with 503 (capability_unavailable) when embedders exist but none is healthy. Requests that name a model are unaffected.
The per-request Request completed line did not say what status the client got, so a backend returning 502 or 503
read the same as a success. It now carries http_status on every proxy route (/olla/proxy/, provider routes and the
Anthropic route, passthrough and translation, streaming included), and a backend 5xx is logged at WARN. A backend 4xx
stays at INFO. Olla's own rejections are unchanged and still go through the sampled Request rejected line, and a
rejected request no longer also writes an INFO Request received line.
Two counting bugs made the status figures read better than the fleet was doing. A request that named a model was
counted twice (since v0.0.29), doubling total_requests, per-endpoint requests and traffic in /internal/status,
/internal/status/endpoints and /internal/metrics. And a backend 5xx was counted as a success whenever the
connection itself worked, so a backend answering 503 to everything could still show 100%. A backend 5xx now counts
as a failure in success_rate, total_failures, olla_failures_total, the per-endpoint and per-model stats and
/internal/stats/translators (passthrough and translation). A backend 4xx, 429 included, still counts as a success,
the same line the WARN log level draws. Only the counters change: the client still gets the backend's response as-is,
a 5xx does not fail over to another backend and the circuit breaker treats it as before. Counting is per backend attempt: a transport
failure on endpoint A then a success on B records one failure and one success. The dashboard's
response-rate hover text now says this instead of "regardless of HTTP status".
A request that named a model but didn't send Content-Type: application/json was routed as if it named
none, so it could land on any compatible host, including ones without the model (TEI answered 424 for
embeddings sent this way). curl -d sends form-urlencoded, fetch() with a string body sends
text/plain, and plenty of scripts send nothing. Olla now decides from the body: a body that opens with
{ is read for its model and capabilities whatever the header says, and the request is forwarded
unchanged. Multipart, form-encoded and compressed bodies are still not read. Nothing to change on your side.
/api/embed and embedding models in a fleet with TEIThe built-in Ollama profile now lists /api/embed, so Ollama's native embeddings route is not
captured by TEI's /embed. Ollama embedding-only models are typed embeddings from the
capabilities that /api/tags reports (newer Ollama releases; older ones report none and stay
untyped), so /v1/embeddings still reaches them when a TEI instance is in the fleet.
Because /api/embed is now a recognised Ollama path, /olla/ollama/api/embed checks the requested
model like Ollama's other routes. A model pulled
without a tag lists as all-minilm:latest; a bare all-minilm resolves to it here, as it now does
on every other route (see the untagged-names fix below).
LM Studio models carry "object": "model" beside their real type (llm, vlm,
embeddings), and the generic value used to mask the real one. LM Studio embedding
models were never tagged embeddings and vlm models never got vision. The real type
now wins, so those models gain the capabilities their type implies: vision,
multimodal and image-understanding for vlm, embeddings, similarity and
vector-search (without text-generation) for embeddings, and chat and completion
for llm. Model-less vision and embeddings requests are steered to those models.
Olla could exit with fatal error: concurrent map iteration and map write (or concurrent map write) when a model listing, alias lookup or status request ran while discovery was merging the same model from a second endpoint, re-registering it, or removing an endpoint. The unifier was editing model records that request handlers were still reading. It now edits a private copy and publishes it, so readers always see a complete record. The unified-stats counters behind the status endpoints are also guarded now. Nothing needs changing in your config.
A model pulled with ollama pull all-minilm is listed as all-minilm:latest, and Ollama accepts
all-minilm for it, but Olla returned a 404 model_not_found for the bare name on /olla/proxy/,
the provider prefixes, /olla/anthropic/v1/messages and GET /olla/models/{id}. When no endpoint
serves the exact name, Olla now retries once as <name>:latest against Ollama endpoints only, so a
vLLM foo:latest still never answers foo. The request goes to Ollama as you wrote it, and foo
and foo:latest share one stats row, and one sticky key when it is built from the model name (prefix_hash hashes the raw body, so it still differs). Nothing to change on your side.
/olla/ollama/api/ps covers every Ollama endpoint; /api/list removed/olla/ollama/api/ps was forwarded to one arbitrary instance, so with several Ollama
endpoints it reported a fraction of what was loaded. It now queries every healthy Ollama
endpoint and merges the results, one entry per loaded model per host with an added
olla_endpoint field; endpoints that fail are skipped and flagged with X-Olla-Partial.
Each endpoint is queried with its own credentials only, ignoring HTTP_PROXY.
POST /olla/ollama/api/blobs/{digest} now returns 501 like the other model-management
routes, which answer 501 to every method. /olla/ollama/api/list returned 501 but is not
an Ollama endpoint, so the route is gone: requests to it now reach the backend and get
Ollama's own 404.
A failed discovery on one endpoint no longer cancels discovery for the others. After five consecutive failures an endpoint's discovery pauses for at least five minutes (or the discovery interval, if longer), and 4xx responses pause it immediately. Olla retries on the first periodic scan after the pause and resumes by itself once the backend recovers, with no restart. The pause does not affect health checks. Discovery responses over the size cap are rejected with a clear error. llama.cpp metadata Olla doesn't use, such as a boolean meta.vocab_type or a string meta.ftype, no longer fails the listing and leaves it empty (issue 222). See Model Discovery and llama.cpp troubleshooting.
If a service failed to start, the ones already running were rolled back in start order, so a service could be stopped after something it depends on. They now unwind last-started first, and rollback still runs if the startup context was cancelled. Thanks to @SDTBU-FXL for the contribution.
When the same model (matched by digest) was registered from more than one backend, the merged alias list was rebuilt in random order, and /olla/models and the provider listings emit the first alias as the model id. The id could change between refreshes. Merging now keeps a stable order and leads with the model's own id.
The shipped config.yaml, the examples and the docs carried keys the config schema has no field for, which Olla silently drops: server.rate_limits.per_endpoint, the whole proxy.retry block, discovery.health_check, unification.enabled and endpoint tags. They are gone from the shipped configs and doc snippets. Failover to another healthy endpoint on a connection failure is automatic, and the deprecation comment that pointed at a non-existent retry.max_attempts now says so (proxy.retry has since become a real block, see above). Unification is controlled by model_registry.enable_unifier.
characteristics.max_concurrent_requests and resources.concurrency_limits in a profile are read as metadata only and do not limit concurrency. The profile and backend docs now say so, and that the backend's own concurrency settings are what limit parallelism. The rate-limit docs also stop claiming health and system endpoints have their own limits: the reference now lists which routes are rate limited and which aren't. A test now decodes the shipped configs against the full schema so a dead key fails the build. Behaviour is unchanged. Reported by @matthewjhunter.
Nothing published for this version
Nothing published for this version
Olla is a high-performance proxy and load balancer for LLM infrastructure.
Olla is a high-performance proxy and load balancer for LLM infrastructure.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.29
# Binary (see assets below)
./olla --config config.yaml
/internal/ui/ - Overview (fleet status, success rate, latency, live requests-per-second sparkline), Endpoints (per-endpoint health, priority, latency, model count) and Models (inventory grouped by family, with hosting endpoints). Built in Svelte 5 + TailwindCSS, polled (not pushed) against the existing /internal/status* JSON, and served from the same listener as the proxy - no second port (#205).dashboard.access_policy config block (allowed_cidrs + allowed_hosts), loopback-only by default. The published Docker image ships with allowed_cidrs pre-widened to the RFC1918 ranges so docker run -p 40114:40114 ghcr.io/thushan/olla:latest gives you a working dashboard with no config mount - which also means the container is reachable from anyone else on the same LAN through that published port (#213). See the Admin Dashboard docs for the full security model.make build-web (e.g. plain go build or go install) now logs a clear startup warning and serves 503 at /internal/ui/ instead of a silent placeholder (#213).GET /internal/metrics in Prometheus text format, built from the same data as /internal/status and /internal/stats/models - no external exporter needed for core proxy monitoring. Thanks to @Puupuls for the contribution (#188)./internal/status, /internal/status/endpoints and /internal/status/models now emit a weak ETag, so polling clients (including the new dashboard) can use If-None-Match and get 304s instead of re-fetching the full payload (#205, #213).sticky_outcome, routing strategy/action/reason, provider_model, translator fallback reason), not just response headers. Addresses issue #178 (#182).Info again in the access log; a prior regression had silently demoted them to Debug (#211).model_aliases entry silently proxy to the wrong backend: a request for an alias whose target model exists on no endpoint now fails fast with 404/503 and routing_action: rejected, matching how unknown models are already handled. Reported by @skaravos (#197, issue #191).config/models.yaml now actually parses. It used \d inside double-quoted YAML scalars, which is invalid YAML, so the file has never loaded on any install - the failure was swallowed and Olla silently fell back to embedded defaults, discarding any customisation with no diagnostic. Reported by @billford (#206, issue #204).--validate-config flag checks configuration and provider profiles without starting the server, with a pass/warn/fail report and exit codes (#210).logging.level in config is now actually applied to the runtime logger, instead of being parsed and ignored (#209).model_extraction.family_aliases and special_rules.preserve_family into the family-extraction pipeline - both were parsed but previously had no effect. Fixes Kimi-K2 being misclassified under deepseek (shared GGUF architecture lineage) (#208).user:pass@host credentials now fails startup instead of loading silently and leaking the credentials into every status/dashboard JSON response. The boot error rewrites the URL into a ready-to-paste auth: block with placeholders, so the fix doesn't require re-typing the real credentials (#213).id and url fields on /internal/status and /internal/status/endpoints are now derived from a sanitised URL (userinfo, query and fragment stripped) rather than echoing the raw configured URL (#213)./internal/status no longer reports "status": "critical" on a healthy fresh boot with no traffic yet. A new system.has_traffic boolean lets clients branch on the no-traffic state directly instead of parsing the success_rate string (#213)..dockerignore for a smaller, more predictable build context (#213).- url: "http://user:pass@host:8080" now fails startup. Move credentials into an auth: block:
- url: "http://host:8080"
auth:
type: basic
username: user
password: passid/url fields: now derived from a sanitised URL rather than the raw config value. Bookmarked dashboard deep-links to a specific endpoint will change once after upgrading. Clients comparing url for identity should switch to id."critical". Check system.has_traffic if you branch on traffic state./olla/proxy/ with zero healthy endpoints now returns 503 instead of 502. Update any monitor keyed on that status code.owned_by in converted model listings: the shared organisation-extraction logic used by the vLLM, vLLM-MLX, SGLang, llama.cpp, LMDeploy and Lemonade converters now also matches a hyphenated leading segment (e.g. Qwen2.5-7B → owned_by: "Qwen2.5") rather than falling through to a generic default. Treat owned_by as a best-effort label, not a canonical identifier.Thanks to everyone who contributed to this release:
GET /internal/metrics Prometheus endpoint (#188).config/models.yaml parse failure (issue #204).Full Changelog: v0.0.28...v0.0.29
Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
An endpoint URL with embedded user:pass@host is rejected at config load. Previously
such a URL loaded silently and the credentials flowed into every status/dashboard JSON
surface as the literal URL string. The endorsed credential path is the auth: block,
which is held as json:"-" fields on the internal endpoint record and never reaches the
status layer.
Before:
- url: "http://user:pass@host:8080"
name: "protected-backend"
type: "openai-compatible"
After:
- url: "http://host:8080"
name: "protected-backend"
type: "openai-compatible"
auth:
type: basic
username: user
password: pass
For secrets kept out of config, use the username_file / password_file variants (see
Endpoint Authentication).
The boot error rewrites the rejected URL into an auth: block with the credentials
replaced by placeholders, so the fix is a copy-paste away without the error ever echoing
your real credentials. For a full user:pass@host URL this is an exactly equivalent
basic block; for a username-only user@host URL there is no equivalent basic
configuration (basic auth requires a password), so the error offers migration
alternatives instead - a basic block with a password placeholder, or a bearer block if
the username was really a token.
/internal/status no longer reports "status": "critical" on a fresh boot with all
endpoints healthy. With no proxy traffic, the system status derives from endpoint health
alone, success_rate reports "N/A", and a new always-present boolean
system.has_traffic lets clients branch on the no-traffic state without parsing the
success-rate string. Previously a healthy fresh boot fell through a < 90.0 success-rate
threshold and reported critical, which coupled with the dashboard produced a misleading
red status on first start.
id derivation changedThe id values surfaced on /internal/status, /internal/status/endpoints, and the
per-model endpoint_ids on /internal/status/models are now derived from the sanitised
URL (scheme+host+port+path) with positional disambiguation for siblings that share a
sanitised form. Credential rotation no longer changes the ID, because userinfo, query,
and fragment are stripped before hashing. Bookmarked dashboard deep-links to a specific
endpoint row will change once after upgrading, then stay stable for endpoints with a
distinct name or a distinct sanitised URL. IDs may change again if a sibling sharing the
same sanitised URL is later added or removed, because the shared -N suffix is positional
and gets renumbered (and the degenerate case of two endpoints sharing both name and
sanitised URL is not immune either). The IDs are base36 FNV-1a, identical across all three
status payloads for the same endpoint from the same repository snapshot.
url in status responses is now sanitisedThe url field on /internal/status and /internal/status/endpoints now has userinfo,
query string, and fragment stripped before it is surfaced. (/internal/status/models
exposes no URL at all - only endpoint display names and endpoint_ids.) Previously this
field echoed the raw configured URL, which could include embedded credentials. Clients
that compared this field against the raw config value for identity should switch to the
id field, which is designed for that purpose.
The three status JSON routes now emit a weak ETag of the form W/"<base36>" over
their stable fields. If-None-Match uses weak comparison, so clients that echo the
ETag verbatim are unaffected. Clients doing strong-only comparison should allow weak
matches. See Conditional requests and compression
for the full contract.
Successful proxy requests log at Info again in the access log. A regression had
demoted them to Debug, which silently broke the "log all requests" expectation the
security practices document promises. Only /internal/ GET/HEAD polling that returns
2xx or 304 stays at Debug so an open dashboard tab does not flood the log; 4xx,
5xx, and any non-GET/HEAD method under /internal/ continue to log at Info.
A binary built without make build-web (e.g. via go install or a plain go build)
now logs a clear startup line and serves 503 at /internal/ui/ with a body naming the
fix. Previously such a binary served a silent placeholder. See
Development Setup: building the dashboard.
/internal/ui/A read-only, single-page dashboard is now embedded in the binary and served from the
same listener as the proxy: Overview (fleet status, success rate, latency, a live
requests-per-second sparkline), Endpoints (per-endpoint health, priority, latency,
model count), and Models (inventory grouped by family, with hosting endpoints). It
polls the existing /internal/status* JSON every 5-15 seconds with ETag/304
caching - no WebSocket, no push, and nothing in it changes state.
The Overview herd table stays readable at phone width (the status column and figures fit without sideways scrolling), response rate and latency show a single dash where an endpoint has no traffic, the sparkline says "no recent traffic" instead of drawing an empty axis, and the commit hash is shortened to 8 characters with the full hash on hover.
There is no authentication. Access is controlled entirely by the new dashboard.access_policy
config block (allowed_cidrs + allowed_hosts), which defaults to loopback-only. The
published Docker image ships with allowed_cidrs pre-widened to the RFC1918 ranges and
the listener bound to 0.0.0.0, so docker run -p 40114:40114 ghcr.io/thushan/olla:latest
followed by opening /internal/ui/ works with no config mount - and also means the
dashboard, and the container, are reachable from anyone else on the same LAN through that
published port. The access policy gates /internal/ui/ only: the /internal/status*,
/internal/health, /internal/metrics and /version JSON it reads stay as reachable as
they already were, allowed_cidrs or not. Unifying that gap under one policy is tracked
in issue #214.
See Admin Dashboard for the full security model and configuration reference.
GET /internal/metricsPrometheus-format metrics are now exposed directly, built from the same data as
/internal/status and /internal/stats/models - system status, per-endpoint health,
security counters, model usage and routing. No external exporter is required for core
proxy monitoring. Thanks to @Puupuls for the contribution. See
System API for the metric series.
A request that lands on an endpoint whose circuit breaker is open now fails over to the next available endpoint instead of the request failing outright. The endpoint is removed from that request's candidate list only; its persisted health is left untouched, because an open breaker already reflects accumulated failure state and demoting health on top of it is the proxy engine's job for genuine connectivity failures, not the retry path's. Exhaustion errors now distinguish circuit-breaker-open counts from connection-failure counts instead of conflating the two under one message.
A request for a model_aliases entry whose target model exists on no endpoint used to be
logged as rejected but still proxied to a compatible backend anyway, returning 200 from
the wrong model. It now returns 404/503 with routing_action: rejected, matching how
unknown models are already handled. One side effect: /olla/proxy/ with zero healthy
endpoints now returns 503 instead of 502 - update any monitor keyed on that specific
status code.
owned_by in converted model listings may return the raw matched segmentThe organisation-extraction logic used by the vLLM, vLLM-MLX, SGLang, llama.cpp, LMDeploy
and Lemonade converters is now shared. Alongside the existing org/model slash split, it
also recognises a hyphen-separated leading segment against a known-organisation list and
returns that segment verbatim - so a model ID like Qwen2.5-7B now yields owned_by: "Qwen2.5" rather than falling through to the converter's generic default. Anything
parsing owned_by for exact organisation matching should treat it as a best-effort label,
not a canonical identifier.
config/models.yaml customisations are no longer silently discardedThe shipped config/models.yaml used \d inside double-quoted YAML scalars, which is not
a valid YAML escape - the file has never parsed, on any install. The failure was swallowed
and Olla silently fell back to embedded defaults, so any customisation (e.g. capability
tagging via name_patterns) had no effect and no diagnostic. The regex patterns are fixed
to single-quoted scalars, config loading now happens eagerly at boot rather than lazily on
first request, and a found-but-unparseable candidate logs a WARN with the path and YAML
error instead of failing silently.
--validate-config flagRun olla --validate-config to check the configuration and provider profiles without
starting the server - a clear pass/warn/fail report with exit codes, useful in CI or
before a restart.
logging.level in config is now honouredThe logging.level config field was parsed but never applied to the runtime logger.
Precedence is OLLA_LOGGING_LEVEL env var, then config file, then default; an invalid
value now warns and falls back instead of being silently ignored.
Olla is a high-performance proxy and load balancer for LLM infrastructure.
Olla is a high-performance proxy and load balancer for LLM infrastructure.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.28
# Binary (see assets below)
./olla --config config.yamlThis is a huge release and contains a lot of new exciting changes and bugfixes!
anthropic_support.messages_path), fixing 404s against DMR's /anthropic/v1/messages (#171).downloaded flag is mapped to available state, so downloaded models route correctly instead of being treated as unhealthy. Thanks to @matthewjhunter for the report and fix (#161, issue #160).type: "openai" now resolves model listings correctly via /olla/openai/v1/models; removed the duplicate openai.yaml profile that caused non-deterministic route registration. openai-compatible is the canonical value,openai remains an accepted alias. Thanks to @petersimmons1972 for the report (#151, issue #148).Highly requested feature finally implemented!
bearer, api_key and basic auth, with credentials from inline strings, ${ENV_VAR}, or _file siblings for Docker/k8s. Works with vLLM/llama.cpp/LiteLLM --api-key or any bearer-auth reverseconfig_error instead of dead; 429 honours Retry-After without tripping the circuit breaker; POST retries are skipped once response bytes have flushed (no double billing on mid-stream resets).max_body_size on non-proxy and translator routes (including chunked bodies) (#173).X-Olla-* headers from upstream responses (anti-spoofing), sanitise X-Request-ID to prevent log injection, and stop leaking the auth scheme in status JSON (#173).rs/cors) for browser-based clients like OpenWebUI and dashboards. Off by default; auto-exposes the X-Olla-* header set. Thanks to @ccsmart for the request (#159, issue #156).reasoning/reasoning_content now maps to Anthropic thinking blocks in both streaming and non-streaming responses (#176).usage chunks, synthesises fallback output_tokens, keeps same-chunk reasoning/content/tool-call deltas intact, and fixes repeated/interleaved tool-call handling (#176).Flush/Unwrap on wrapped writers (#176).context_management) are now ignored during translation and forwarded unchanged in passthrough mode. Thanks to @RenxuLogan for the report (#157, issue #154).OLLA_-overridable (#164, #165):server.read_header_timeout (default 10s) - guards against Slowloris-style slow-header attacks.proxy.response_header_timeout (default 30s) - raise for on-demand model loaders (e.g. Lemonade) that would otherwise abort cold starts at 30s. Now honoured by both Olla and Sherpa. Thanks to @matthewjhunter (#164, issue #163).proxy.connection_keep_alive (default 30s) and proxy.tls_handshake_timeout (default 10s).model_registry.unification.cleanup_interval / stale_threshold are now honoured (previously silently ignored, hard-coded to 5m) (#171).atomic.Pointer, snapshotted once per request/stream, so a hot-reload can't hand an in-flight request a torn config (#173, #176).LoadOrCompute, eliminating racing transport rebuilds on first use (#173).litepool.Get returns errors instead of panicking (handles nil and typed-nil factory results) (#176).runtime.GC(), an unread request counter) (#173)./internal/status/models now returns endpoint names for every entry (was inconsistently URL-then-names). Consumers reading endpoints[0] as a URL must switch to names (#173).X-Olla-* response headers are dropped, so a chained Olla-in-front-of-Olla setup loses the inner instance's headers (#173).Olla and the default load balancer is least-connections (#172)./olla-validate --quick / --nightly) with a local multi-protocol mock backend and fault injection - no Docker, CI-safe (#174)./version self-path reporting (previously reported a 404'ing /internal/version) (#171).Thanks to everyone who reported issues and contributed to this release:
response_header_timeout (#164, issue #163).type: "openai" endpoints (#148).Full Changelog: v0.0.27...v0.0.28
Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
Nothing published for this version
We've added native support for LMDeploy after so long as well as bugfixes for sticky sessions thanks to @lbatalha .
We've added native support for LMDeploy after so long as well as bugfixes for sticky sessions thanks to @lbatalha.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.27
# Binary (see assets below)
./olla --config config.yamlDocumentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
We've added native support for LMDeploy after so long as well as bugfixes for sticky sessions thanks to @lbatalha.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.27
# Binary (see assets below)
./olla --config config.yaml
Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
This is a bugfix release that addresses SSL connection issues due to mis-handling of the Host header.
This is a bugfix release that addresses SSL connection issues due to mis-handling of the Host header.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.26
# Binary (see assets below)
./olla --config config.yamlDocumentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
This is a bugfix release that addresses SSL connection issues due to mis-handling of the Host header.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.26
# Binary (see assets below)
./olla --config config.yaml
Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
Nothing published for this version
Olla is a high-performance proxy and load balancer for LLM infrastructure.
Olla is a high-performance proxy and load balancer for LLM infrastructure.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.25
# Binary (see assets below)
./olla --config config.yamlThanks to @dnnspaul for contributing the Model Aliasing feature to Olla to alias models easily via the configuration.
We've now got a way of having sticky sessions in Olla to help keep requests aligned to KV Caches across multiple endpoints, taken from the working implementation in TensorFoundry's FoundryOS.
Lots of bugfixes and chores from March & April.
Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
Olla is a high-performance proxy and load balancer for LLM infrastructure.
# Docker
docker pull ghcr.io/thushan/olla:v0.0.25
# Binary (see assets below)
./olla --config config.yaml
Thanks to @dnnspaul for contributing the Model Aliasing feature to Olla to alias models easily via the configuration.
We've now got a way of having sticky sessions in Olla to help keep requests aligned to KV Caches across multiple endpoints, taken from the working implementation in TensorFoundry's FoundryOS.
Lots of bugfixes and chores from March & April.
Documentation: thushan.github.io/olla | Issues: github.com/thushan/olla/issues
Nothing published for this version
This is a bugfix release to fix some agentic workloads for translator mode, agent tooling and improved logging.
This is a bugfix release to fix some agentic workloads for translator mode, agent tooling and improved logging.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.23...v0.0.24
Nothing published for this version
This is a major release bringing in some exciting features:
This is a major release bringing in some exciting features:
/olla/proxy) resolution issues when certain mixes of backends were presentFull Changelog: https://github.com/thushan/olla/compare/v0.0.22...v0.0.23
Nothing published for this version
This release was largely for fixing model_url resolution but also contains some maintenance fixes.
This release was largely for fixing model_url resolution but also contains some maintenance fixes.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.21...v0.0.22
Nothing published for this version
This release is to help address path translation issues (see #80) with tools like Docker Model Runner and Olla's default way of handling paths and not
This release is to help address path translation issues (see #80) with tools like Docker Model Runner and Olla's default way of handling paths and not preserving paths.
We added a new setting for endpoints to instruct Olla to preserve paths when proxying requests.
- url: "http://localhost:12434/engines/llama.cpp/"
name: "local-docker"
type: "openai-compatible"
priority: 100
preserve_path: true # this way, /v1/completions will forward properly to Docker Model Runner
model_url: "/models"
health_check_url: "/"
check_interval: 2s
check_timeout: 1s
There's also a bugfix for the missing OpenAI routing (for type: openai-compatible) with refreshed profiles for OpenAI.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.20...v0.0.21
This release brings back llamacpp integration and adds experimental Anthropic message support (disabled by default) at /olla/anthropic so you can poin
This release brings back llamacpp integration and adds experimental Anthropic message support (disabled by default) at /olla/anthropic so you can point Claude Code and other tools easily.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.19...v0.0.20
Nothing published for this version
837dc42f6a54ddc5507920ae871642817b34fe5b use map rather than MapOf (deprecated)
This release has several performance fixes (noticeably uplift for ARM), critical fixes for all archs and adds support for sglang and LemonadeSDK.
Encourage all to upgrade to this release.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.18...v0.0.19
This is mostly a maintenance release and includes consolidation of configuration of the Sherpa and Olla Proxies internally.
This is mostly a maintenance release and includes consolidation of configuration of the Sherpa and Olla Proxies internally.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.17...v0.0.18
Nothing published for this version
This release brings support for litellm and also the ability to filter (generically) within the config - with include/exclude globs. For now, this all
This release brings support for litellm and also the ability to filter (generically) within the config - with include/exclude globs. For now, this allows you to exclude profiles you don't want loaded and exclude models from an endpoint.
Learn more about filters.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.16...v0.0.17
Nothing published for this version
This release has two big features.
This release has two big features.
Health is monitored during every request and if a request routes to a just failed endpoint (before the healthcheck has run) it will transparently move to another healthy endpoint that contains the model. Transparent to the caller, but logs in the CLI and headers are available to know what happened.
We think this makes olla pretty awesome.
We also capture the last packet from the stream/payload and pull out metrics for common things from endpoints (TPS etc) and track other metrics that are (currently) shown in the logs.
Later these will be used for a new robusta balancer.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.15...v0.0.16
_tldr;_ _This release adds proper cross platform docker images thanks to @ghostdevv and adds support for vLLM natively, proxy profiles so you can targ
tldr; This release adds proper cross platform docker images thanks to @ghostdevv and adds support for vLLM natively, proxy profiles so you can target streaming or buffering proxies, it finally also adds documentation (via mkdocs) and a few fixes and improvements
Full Changelog: https://github.com/thushan/olla/compare/v0.0.14...v0.0.15
_tldr;_ _This release addresses an issue with streaming tokens from LLM backends (it was buffering instead of streaming) and adds proper OpenWebUI sup
tldr; This release addresses an issue with streaming tokens from LLM backends (it was buffering instead of streaming) and adds proper OpenWebUI support to Olla
Full Changelog: https://github.com/thushan/olla/compare/v0.0.13...v0.0.14
Documentation updates and includes examples for using OpenWebUI with Olla.
Documentation updates and includes examples for using OpenWebUI with Olla.
ed93e9a75d72c691cf69abb7d5d8bb0d93afab4a deprecate old proxy_olla & proxy_sherpa
This is our first official release that's released publicly.
Amongst the highlights:
We've beefed up the performance across the board, Olla now can sip < 40Mb process size on Linux serving hundreds of endpoints.
Full Changelog: https://github.com/thushan/olla/compare/v0.0.11...v0.0.12
15411f15f0bdf8e57ab4fa25669f7a86dbde7efa First pass at implementing model normalisation
301707c18816743f9df18e5730bef2b627d4d4e3 security fixes for avoiding copying bad headers.
be39dd5f5cdd9fb67888e9aff9561b089dd01d2d Reapply "change to use json-iterator and have each profile manage it's parsing of responses."
c7d19ce186c388ba7e823bd9bd4df1f8510724b6 update banner to have blue + docker bits
75fec547c2f09b5bf7322587959f0bf1666a70c2 - goreleaser includes docker.yaml
Nothing published for this version
Nothing published for this version
Nothing published for this version
First internal release for wider testing.
First internal release for wider testing.
One column per month.